How can you ensure your AI models make causal inferences, not just correlations?

To understand the causal effect of data and not just the correlations, we need to go beyond observing statistical relationships and consider the underlying mechanisms that drive these relationships. This involves designing experiments or using observational data in a way that allows us to isolate the effect of a specific variable or treatment on the outcome of interest, while controlling for other factors that might influence the outcome. By carefully designing studies and analyzing data using causal inference methods, we can make more informed decisions and draw conclusions about the causal relationships in the data.

Best ways to analyze data effectively We should look for patterns and trends We should compare current data against historical trends We should look for any data which is against our expectations We should pull data from various sources and then Determine the next steps

Understand your data #Data Collection Process: Gain insights into how the data was collected, including the methodologies, instruments, and protocols used. Understanding the data collection process helps identify potential sources of bias or error and informs decisions regarding data preprocessing and model training. #Data Quality and Consistency: Assess the quality and consistency of your data across different sources, time periods, or data collection methods. Inconsistencies or discrepancies in data quality can introduce noise and bias into your analysis, affecting the reliability of model predictions.

This is the most important step. In my experience modelling events as Markov Chains has even revealed enough causal relationships that we did not really need to deploy an ML model in production. We just learnt the causal relationship and directly applied that knowledge as a rule in our production. One specific example is the solution we deployed to preventing collusion in Poker games at GetMega. Our EDA (exploratory data analysis) discovered 2 patterns responsible for most of the collusion. We ended up directly coding up detection of those 2 patterns in the games in real-time and that led to savings of over ₹20 lakhs worth of collusion each month.

To ensure AI models discern causality, not just correlations, deeply understand your data. Know its origins, characteristics, and potential biases. It's crucial to identify and account for variables that might influence your results, like confounders or mediators. This thorough analysis ensures your AI's findings are robust and grounded in actual cause-effect relationships, providing insights that are actionable and reliable.

To ensure AI models make causal inferences, selecting appropriate models like Structural Equation Models, Instrumental Variable Regression, or Propensity Score Matching can help. These models align with causal assumptions, capturing complex relationships beyond mere correlations. Consider factors such as endogeneity and hidden variables. Granger Causality and Bayesian Networks offer alternative approaches. Validate results rigorously, testing assumptions and communicating findings transparently. Collaboration with domain experts enhances model selection and interpretation, ensuring robust causal inference.

Use multiple models and employ automated GenAI powered processes that use agents. With "chain of thought" concepts you can achieve incredible results, that you hardly can achieve manually.

As an example, you can improve Gen AI's predictions in healthcare with casual inference modeling technique blended with expert in the loop feedback. Utilize data from studies where treatments are randomly assigned to help show cause and effect. Add a Structural Causal Model to map out how patient traits, treatments, and outcomes are linked. Collect feedback from doctors to ensure the AI's predictions match medical understanding. Finally, merge the AI with the SCM, so it makes predictions informed by causality, not just correlations. This approach boosts the AI's ability to make reliable, insightful forecasts.

Choose your model wisely #Identifying Causal Mechanisms: Consider the underlying causal mechanisms in your data-generating process. If your research question aims to understand not just associations but also the causal relationships between variables, causal inference models are essential. These models explicitly model causal relationships, allowing for the estimation of causal effects and the identification of underlying mechanisms driving observed patterns. #Assumptions and Conditions: Understand the assumptions and conditions required for valid causal inference with your chosen model. Causal inference models often rely on assumptions such as causal sufficiency, no unmeasured confounding, and consistency.

Testing your assumptions means challenging them with facts, evidence, or opinions that can either confirm or invalidate them. For example we can test our assumptions by conducting surveys, interviews, focus groups, or observations with our experts or customers

💡In our AI world, I think of assumptions as the secret ingredients for a magic potion. Some AI models, like chefs, assume their kitchen is ghost-free (no hidden factors messing with cause and effect) and time flows normally (effects don’t time-travel to tweak their causes). To ensure the potion won’t turn you into a frog, methods like sensitivity checks, magic wands (instrumental variables), or stirring the potion under controlled experiments are used. The key? Be like a wise wizard: openly share your recipe's secrets and its potential to turn hats into cats! 🌞

Test your assumptions #Sensitivity Analysis: Conduct sensitivity analyses to assess the robustness of causal inference results to violations of key assumptions. Sensitivity analysis involves systematically varying assumptions or model specifications to evaluate the impact on causal estimates. This helps quantify the degree of uncertainty and potential bias introduced by violations of assumptions, providing insights into the robustness of causal conclusions. #Instrumental Variables: In situations where unmeasured confounding is suspected, instrumental variable (IV) analysis can be employed to estimate causal effects while addressing confounding bias.

Testing assumptions in causal inference hinges on using tools like DAGs and counterfactual logic to validate underlying causal structures. Techniques such as propensity score matching and instrumental variable analysis play a crucial role, simulating randomized trials from observational data. This stage is vital for distinguishing actual causality from mere correlation, ensuring models capture true causal relationships. It blends statistical rigor with domain expertise, marking a pivotal step toward robust causal inference.

Valid causal inference in AI modeling depends on fulfilling specific assumptions, such as no unmeasured confounding or reverse causality. Testing these assumptions through sensitivity analysis, instrumental variables, or randomized experiments is essential. Transparency about these assumptions and their limitations is crucial. For instance, when analyzing the effect of a training program on employee productivity, it's important to consider and test for potential confounding factors. This approach ensures the accuracy of causal relationships and maintains trust in the model's findings.

Validation can be done by checking the quality of our data, rerunning the analysis, and analyzing a sample of comments compared to the AI's results.

To ensure AI models make causal inferences rather than just correlations, validation methods such as cross-validation, bootstrapping, and out-of-sample testing can be employed to assess accuracy. Robustness can be tested through variations in data sets, models, or parameters. Generalizability can be evaluated using external evidence, domain knowledge, and counterfactual analysis. These measures help verify the reliability and validity of causal estimates, distinguishing them from mere correlations.

In the validation and verification, we use a dataset that measures understanding of concepts, instead of just text representations. We do this by using different sentence types, questions that evaluate understanding and use perturbation to the input.

Validate your results #Cross-Validation and Bootstrapping: Employ cross-validation techniques, such as k-fold cross-validation or leave-one-out cross-validation, to assess the stability and generalizability of causal estimates. Bootstrapping, which involves resampling from the dataset with replacement, can also provide robust estimates of uncertainty and evaluate the variability of causal effects across different samples. #Out-of-Sample Testing: Validate the performance of your model by testing it on out-of-sample data that was not used during model training or estimation. Out-of-sample testing helps assess how well the model generalizes to new, unseen data and provides insights into its predictive accuracy and robustness.

Very much needed indeed, and we did that by measuring the price performance of AI prices we recommend. It is easier to price high or low considering the demand, but what actually mattere is the booking momentum. There shouldn't be too few bookings as it would result in low utilisation and wasted resources, so reduce the prices. And if bookings are coming too fast, raise the prices early enough so to have inventory available to sell at profit. No point in raising the prices when inventory is already sold out.

Communicate your findings #Tailoring to Audience: Understand the background, expertise, and needs of your audience, and tailor your communication accordingly. Adapt your language, level of technical detail, and presentation style to ensure that your message resonates with diverse stakeholders, from technical experts to non-specialist decision-makers. #Narrative Storytelling: Frame your findings within a coherent narrative that guides the audience through the rationale, methodology, results, and implications of your causal inference analysis. Use storytelling techniques to engage and captivate the audience, emphasizing the significance of the problem, the novelty of the approach, and the potential impact of the findings.

É meu dever comunicar as descobertas do meu trabalho de forma clara e eficaz ao meu público. Isso inclui detalhar os dados, modelo, suposições, resultados e suas implicações de maneira acessível e prática, utilizando visualizações como gráficos e tabelas para ilustrar pontos-chave. Também é fundamental reconhecer as incertezas, limitações e considerações éticas do meu estudo, além de sugerir caminhos para futuras pesquisas ou ações baseadas nos resultados. Essa comunicação cuidadosa é crucial para assegurar que o impacto e o valor do meu trabalho sejam compreendidos e aplicados de maneira efetiva.

💡❗️Imagine you're a storyteller, with a tale of data and discovery. Your audience could be anyone, from fellow data adventurers to curious villagers (peers, clients, or stakeholders). Your story should really weave through the huge abundance of data landscapes, model mysteries, assumption archives, result revelations, and the implications island 🏝 Make sure you light the path with visual aids like graphs, tables, and diagrams, illuminating the journey. As you near the end, acknowledge the fog of uncertainty, the cliffs of limitations, and the ethical beasts lurking. Leave breadcrumbs for future explorers with recommendations. Your goal? To make the complex, captivatingly clear 🌞💡

It is critical that AI developments and findings need to be shared open-source. Too much these days is happening behind closed doors and not shared broadly. Even layperson development of AI or development in non traditional areas (like AI in diplomacy, per a FB group I moderate) can bring new ideas and applications across sectors if shared. Hoarding info and development is unfortunately too frequent.

- Clarity and Concision: Present your findings in a clear and concise manner, avoiding technical jargon when possible. - Transparency: Be transparent about your data sources, model limitations, and any assumptions made. - Focus on the "Why": Highlight the causal relationships your model uncovered, not just the correlations. Explain the cause-and-effect mechanisms at play. - Target Audience: Tailor your communication style to your audience, whether it's technical experts or business stakeholders. - Effective communication ensures your causal inferences aren't siloed knowledge but drive informed decision-making and advance your field of study.

One way to think about any ML solution is to consider which of the following three functions it primarily serves and then tailor your model and pipeline for it: (1) generate; (2) predict; (3) search. For example, a generative pre-trained transformer can be thought of as a lossy compression engine that takes many terabytes of scrapped Internet data and compresses it into gigabytes of weights in a neural network. The compression is lossy and results in hallucinations when the transformer fills in the gaps inaccurately. Depending on whether your solution is primarily targeting #1, #2, or #3, your ML engineer may supplement with retrieval augmented generation (RAG), fine-tuned models (SLMs) with reduced dimensionality, few shot learning, etc

Imagine you have an AI model that predicts ice cream sales based on temperature. To ensure the model makes causal inferences, be sure to: - Understand Causality: Know why people buy ice cream in heat. - Model Confounders: Include factors beyond temperature like ads or holidays. - Prioritize Interpretability: Ensure AI can explain high sales on hot days. - Practice Responsible AI: Check for biases in flavor recommendations. - Utilize Counterfactual Analysis: Imagine sales on a sudden cold day to understand temperature's true impact.

External Validation and Replication: After obtaining causal estimates, it's vital to validate and replicate your findings using external data or independent studies. External validation involves testing the robustness and generalizability of your causal inferences by applying your model to different datasets or real-world scenarios. Replication entails recreating the analysis using different methodologies or by independent researchers to ensure the reliability and reproducibility of the results. By subjecting your causal inferences to external validation and replication, you strengthen the credibility and confidence in the conclusions drawn from your AI models, fostering greater trust among stakeholders and the broader scientific community.

Causality is an illusion, a market conspiracy that zealots in the church of zero sum games ascribe and aspire to. Show me a grain of surety in a world riddled by the plague of epistemic uncertainty - where the knowledge of a thing at best is a consensus, a sort of parasitic interval that all parties for a time agree - and I will gladly forfeit half my kingdom to experience the ambrosia of certainty. I wager boldly because certainty, certainly doesn't exist and in this paroxysm of self inconsistency maybe it does. Or in the least we can agree that it might exist and leave the certainty of our beliefs to some axiom, built upon another axiom, all the way down - like giant turtles heaving upon each other and marking terra firma.

Ensuring AI models infer causality involves: 1. Using causal inference methods like instrumental variables. 2. Crafting features that reflect causal relationships based on domain knowledge. 3. Conducting randomized control trials when possible. 4. Incorporating counterfactual analysis. 5. Deep domain understanding to distinguish causation from correlation. 6. Validating beyond predictive accuracy. 7. Employing interpretable models. 8. Performing robustness checks to confirm causal relationships are not artifacts. By integrating these approaches, you can enhance your AI models' ability to make causal inferences rather than mere correlations.