Explainability: An Important Facet of ML Models

Recently, I have been looking into the area of model interpretability, which is the characteristic of the model to be understandable by humans. This falls under the broader area of research called Explainable AI (XAI), which I believe is very important aspect of model development. Most of the time we focus on model accuracy, but do we know if the model is giving significance to input that intuitively doesn't make sense or the model has learned any bias from the input data, which might not be apparent. That's where explanations become important for model evaluators to judge model robustness, fairness and for the users to build trust in model predictions.

Few weeks ago, I came across a documentary "Coded Bias", where a MIT media lab researcher Joy Buolamwini discovered that most facial recognition software does not accurately identify dark-skinned faces. Such all softwares are driven by machine learning algorithms behind the scene. Often times, in a rush to deliver outcomes, businesses neglect the bias that could creep in due to the type of data being used. I think all of us have a responsibility to make sure that AI is free from bias and fair as it is becoming ubiquitous in the product and services we use in our daily lives.

It is no surprise that in the last few years deep learning has been adopted in several AI application. But as many of us familiar with this space, deep learning models are considered to be black boxes, where the layers and hidden units could be fairly complex, making it extremely difficult to interpret the model behavior and predictions. The good part of deep learning models is that they are usually much more accurate, so naturally businesses wants to adopt it more, but at the same time these businesses especially in regulated industries such as Finance and Healthcare are very cautious as they need good explainability for model outputs.

Let's first start with understanding the personas that needs explanations and for what purpose:

Model Developers - Personas such as Data Scientist, Machine Learning Engineer want to make sure the model behaves as expected, so explanations are important to improve performance . For example, let's say as per explanations, certain predictions are getting attributed to inputs that do not make sense intuitively. In such a case the model developer could make changes to the input using transformations so that the explanations make sense.

Investigators/Regulators - In regulated industries like Finance and Healthcare, the model output needs to be investigated due to possibility of severe consequences. For example, Federal Reserve's SR 11-7 guidance which applies in scenarios such as credit approval, suspicious activity reports for Anti-Money Laundering(AML)

End users - This is the user who interacts with an application driven by ML model. For example, banks loan application portal, disease diagnosis application. It would much more informative if the user is provided clear explanation of why the loan was denied or why the model thinks a patient has a chronic disease. From regulatory standpoint also, as per GDPR the user has the right to meaningful information about the logic involved where automated decision-making is used.

Next, let's take a look at the types of explanations used today:

Model-Based - In such an approach, the model is selected with explainability in mind. For example, certain models such as decision tree are much more easy to interpret. But the challenge with such approach is that the models might not properly fit decreasing model's accuracy

Post-hoc - These techniques are especially used with Black Box models, for example deep neural networks, which are usually very accurate, but not easy to explain and done after the model has been trained. The main objective in such explanations is to show "feature importance" for given predictions and a common method used is Shapley values, however there are other methods also such as LIME, LRP, and gradient based. These explanations could be at each prediction level (local) or generally at the model as a whole (global).

MLOps has done a great job at bringing the best practices of software development to model development, seamless deployment and monitoring. Next to ensure that these models are free from biases and easy to interpret, I am interested in learning how people in the MLOps community have incorporated Explainability feature in their platform. What kind of tooling and experiences have you built to serve the needs of the different types of personas mentioned earlier