AI Models: You Break it, You Buy It

Lawmakers across the globe have taken steps to ensure that AI models are secure, safe, trustworthy, fair, and explainable; and that they respect intellectual property laws and user privacy. In 2024, the EU AI Actwent into effect, and the United States, along with the European Union, United Kingdom, and 7 other countries signed the Council of Europe Framework Convention on Artificial Intelligence and Human Rights, Democracy, and the Rule of Law (International AI Convention). US states also enacted AI laws, including no less than 18 AI laws in California, in addition to laws in Colorado and Utah.

Setting aside the extraordinarily fast-moving pace of AI regulation, and its potential consequences for innovation and investment, AI model developers have intrinsic motives to ensure that models and applications conform to responsible AI principles so that customers trust the technology enough to pay for it, as well as to avoid liability and to protect their reputations. What responsible AI means for users of hosted models or the AI-powered services of model developers, is another matter.

Warranty Void if Seal is Broken

Organizations merely using models may be unaware of the potential risks and liabilities they might incur by fine-tuning models or building retrieval augmented generation (RAG) applications. It may seem reasonable to assume that model developers will ensure compliance with applicable laws and regulations, and that existing cybersecurity and governance, risk management, and regulatory compliance (GRC) functions cover responsible AI. In fact, users of hosted models or services may be regulated in their own right, and security is only one responsible AI component. New AI frameworks and standards might not be covered by existing GRC programs.

Somewhat different to the shared responsibility model in cloud security, when an organization modifies a model or deploys a RAG application, they logically become responsible for many aspects of responsible AI. If an organization fine-tunes a model or uses internal documents for RAG, it should take steps similar to the original model developer to ensure compliance with responsible AI principles and with applicable laws and regulations. Organizations in regulated industries must take industry-specific regulations, as well as supply chain security and vendor due diligence, into account.

Of course, specific requirements vary by sector, industry, and use case. Every model and application is different and the exact technical requirements for responsible AI depend on its details and on each organization’s technology stack, use case, model, and application. As a result, every organization should conduct a thorough analysis of responsible AI technical requirements.

Model developers in the United States have made significant efforts to, in effect, regulate themselves ahead of any federal AI laws or regulations, and organizations using hosted models and AI-based services are moving ahead with a wide range of AI applications. A cursory review of European General Data Protection Regulation (GDPR) enforcement actions, ePrivacy Directive, and the EU AI should make clear that US model developers are taking responsible AI seriously. In comparison, regulatory compliance is a mixed picture for organizations using hosted models and AI-based services. 

Need for Ongoing Testing

Organizations seeking assurance that models or applications conform to responsible AI should test them (e.g., using open source AI benchmarks like DecodingTrust), but many organizations lack the technical expertise and tooling to appropriately address responsible AI. 

From a GRC perspective, organizations can leverage AI frameworks and standards like the US National Institute of Standards and Technology (NIST) Artificial Intelligence Risk Management Framework (NIST AI 600-1), and the ISO/IEC Artificial Intelligence Management System (AIMS) standard (ISO/IEC 42001:2023), along with open source AI test tools. 

Components of responsible AI (e.g., security, safety, privacy, fairness, accuracy and trustworthiness, human-control, human-centric design, explainability, transparency, etc.) are different to one another, require different tools to test, and may be addressed at different stages of the MLOps pipeline and model or application software development lifecycle. To demonstrate due care, organizations should map relevant aspects of responsible AI to specific, repeatable tests, establish metrics and baselines, and be in a position to monitor models or applications and compare test results, e.g., each time a model or a RAG document set is updated.