Medical prognoses using artificial intelligence - How important is interpretability?

If artificial intelligence is to be involved in disease prognoses, the transparency of the insights gained must not be lost.

In recent years, soventec has built up and applied expertise in the field of neural networks/artificial intelligence. This knowledge is centred in the soventec AI platform AIcebird^®

AIcebird^® is to be understood as a modular construction kit. The requirements for the development and use of AI are efficiently adapted to the specific application. An implemented use case is the minimally invasive early diagnosis of cancer. AIcebird^® offers software modules for this purpose that train the system in such a way that it is able to evaluate hundreds of biomarkers on the basis of artificial intelligence. In this way, medical personnel can be provided with information for differentiated early cancer detection.

The use of artificial intelligence thus creates an evaluation system that quickly discovers conspicuous features, patterns and correlations that would take humans a great deal of time to notice, if at all.

Regulatory consideration of the interpretability of artificial intelligence in the medical environment

Even if AI-supported evaluation systems are generally not used for diagnosis but as an assistance system for trained medical staff, we are moving in the medical field, which is regulated to protect patient safety. A look at medical device legislation is therefore essential for any conception of software that aims to be used in a clinical environment.

Medical device legislation obliges medical device manufacturers to prioritise the safety of products, to use state-of-the-art technology, to achieve the best possible risk-benefit ratio and to ensure the repeatability of results.

Although this does not result in any direct requirements for the nature of artificial intelligence in medical devices, indirect requirements can be derived.

Since there are now tools and methods in the area of the "interpretability" of artificial intelligence that increase the transparency and explainability of artificial intelligence, attention must be paid to this according to the "state of the art". Ideally, this will lead to an improvement in the remaining aspects mentioned: safety, risk-benefit ratio and repeatability.

However, it also means that medical devices that ignore the methods for increasing the interpretability of AI may not be state of the art.

What does interpretability mean in the field of "artificial intelligence"?

In the field of "machine learning" in particular, computers learn rules independently from data. "Deep learning" is often used for this. This is a method of machine learning that uses artificial neural networks with numerous intermediate layers between the input layer and the output layer, thereby forming an extensive internal structure.

However, the claim that machine learning generates "black box" models that provide valuable output but cannot be understood by humans at all is only partially true. Humans do not have to understand the complexity of the models. Due to the extensive internal structure, in many cases they cannot do so at all. The algorithms are not only mathematically difficult to understand, but also hardly understandable due to the high dimensionality and abstraction.

When creating machine learning models, however, attention can certainly be paid to increasing explainability and traceability. This is particularly important in a highly regulated industry such as medical technology.

To increase the interpretability of artificial intelligence, one can look at "explainability" and "transparency".

Explainability describes the degree to which a system can provide clarity about the reasons that lead to certain outcomes.

Transparency describes the degree to which a system reveals information about its inner life, i.e. its inner structure and training data. Transparency, in contrast to explainability, thus presupposes the "opening of the black box".

What consequences would a lack of interpretability have for the use of artificial intelligence in the clinical setting?

Non-transparent models can lead to incorrect conclusions being drawn from certain data, but this is not noticed when the models are trained. The trained model is based on the training data. These can contain errors or be unbalanced and one-sidedly weighted. This can lead to incorrect statements when the models are applied, for example an indication of an incorrect disease prognosis.

However, if one recognises that a model, based on previous input data, overweights certain information, for example, this incorrect weighting can be better recognised and corrected.

Models that cannot be explained may result in an AI that reliably produces valuable output not being accepted and used. For example, if the model provides medical staff with reliable clues but no explanation of where certain insights come from, it is unlikely that there will be acceptance of the system or fundamental trust in the clues provided.

However, if the mode of action of a model can be explained, this can increase acceptance and also significantly simplify the reconciliation of indications from the AI with expert knowledge.

Why not work with fully transparent, explainable models?

The more complex a model is, the more difficult it is to understand its decisions. A linear model depicts simple interrelationships and is thus easy to explain. A deep neural network (deep learning) can model very complex interrelationships with a high degree of abstraction, but is also multidimensional in structure.

Since the soventec platform AIcebird^® is used, for example, to recognise complex interrelationships, we move in the area of conflict between desired interpretability and necessary abstraction, which the human brain cannot process at all. We can therefore only pursue the goal of increasing interpretability with carefully selected tools, without claiming that all processes taking place are understandable.

It is therefore necessary to improve interpretability by increasing transparency and explainability for a given level of complexity.

In addition, the incomplete transparency and explainability must be made clear to the users and secured with sufficient verification. A residual risk remains. However, this residual risk also exists in a different form with non-AI-based forecasts.

AIcebird^® based systems can become a helpful advisor in the sense of assistance systems. However, a treating physician will continue to evaluate given advice with his medical expertise and reject it if necessary. Even according to the prevailing legal opinion, the doctor must still be the final decision-maker. In the future, however, this could change so that AI-based systems could also make diagnostic and therapy-relevant decisions "autonomously".

Exemplary presentation of methods to increase transparency and explainability

Various concepts exist to make the structure of neural networks more transparent:

With simple models, such as this one, it is still possible to visualise it, although it already becomes confusing here

However, as soon as networks with feedback mechanisms or intended random variation of parameters are implemented in order to represent system-inherent errors, such a representation is no longer helpful.

The doctor works in a similar way, drawing conclusions about a diagnosis on the basis of a blood count, for example. Here, too, there are normal ranges and conspicuous ranges.

This method is therefore more suitable for humans who are used to working in this way. Humans can cognitively grasp the greatly reduced, albeit all-inclusive, complexity. This increases confidence in a supposed "black box".

Despite all the scepticism about AI applications in the medical field, one must always keep in mind: Even a doctor is not infallible in his diagnosis. And even decisions made by an experienced doctor based on his experience and intuition are often not fully explainable and yet still correct. It is the same with AI-based applications in the medical environment.

AI holds great potential in the field of diagnostics and can bring great benefits to patients at an early stage. The development of intelligent systems must keep people in mind. They don't just want to benefit, they want to understand and be involved.

Let's face this responsibility together!