Researchers have developed a neural network that can identify lung pathologies in X-ray images and generate concise verbal descriptions to accompany them. Currently, physicians spend several minutes compiling these captions, but the artificial intelligence (AI) solution reduces this time to around 30 seconds when significant text revision is not needed. In most cases, the radiologist simply needs to confirm the suggested diagnosis—such as fibrosis, an enlarged heart, or a suspected malignant tumor—or lack thereof.

The AI solution developed by researchers from Skoltech (Moscow, Russia) employs state-of-the-art machine vision and computational linguistics models, including GPT-3 small, a precursor to the widely popular GPT-3.5 and GPT-4 models accessible through the ChatGPT bot. The neural network is trained on data consisting of image-text pairs. Potential enhancements to the system include applying it to MRI and CT scans, integrating active learning, and combining it with another neural network to visually emphasize the areas of interest mentioned in the caption. Active learning refers to models that refine their predictions by considering the adjustments made by human reviewers.

“Regular models merely classify, but our neural network leverages advanced machine vision and computer linguistics models to automatically describe X-ray images in words,” said Skoltech research scientist Oleg Rogov. “We compiled our own radiological dictionary to make the model more accurate, specifically where radiological terms and their usage in texts are concerned. Naturally, we also put together a large integrated database of X-ray images for use as training data,” emphasizing that the neural network is only “aware” of those diagnoses that can actually manifest themselves on lung X-rays. The training set was balanced in terms of which diseases are represented. 

http://m.allmycbs.com/en/index.asp