As AI/ML Models Become More Prevalent, So Too Must Risk Management

Emerging Opportunity: AI/ML Model Risk Management

As the AI/ML ecosystem continues to mature across applied AI and the machine learning infrastructure, one area that we believe is an emerging opportunity for investment is AI/ML model risk management. AI/ML models are becoming more prevalent in our day-to-day lives and are increasingly relied upon to make high-impact or high-risk decisions on everything from military/national defense to credit underwriting to determining who is best qualified for a job. Over the last year, we have witnessed increased attention from both the private and public sector on improving the industry’s standards for managing the potential risks posed by AI/ML models. For example, recently multiple U.S. and European government entities released requests for information, comments, or draft legislation that point toward impending regulation over the use of A/ML models, particularly for high-risk use cases, as shown in the list below.

Recent Examples of Regulatory Commentary & Guidance on AI/ML Risk Management

• July 2021: The Department of Commerce’s National Institute of Standards and Technology issued a request for information (RFI) from the public that will inform the development of its AI risk management guidance.
  
  
• June 2021: The U.S. Government Accountability Office released a framework to help federal agencies and others use AI responsibly.
  
  
• April 2021: Although not formal guidance or regulation, the Federal Trade Commission wrote a detailed blog post outlining expectation about truth, fairness, and equity in the use of AI.
  
  
• April 2021: The European Commission published the Artificial Intelligence Act, first-of-its-kind draft legislation, to manage the risks stemming from AI systems.
  
  
• March 2021: The U.S. Office of the Comptroller of the Currency issued an RFI for the public on financial institutions’ use of AI. It sought respondents’ views on appropriate governance, risk management, and controls over AI’s use.

We expect AI/ML model risk management will become a critical need for any company with a high-risk application of AI/ML, and it will become a best practice for the rest of the AI/ML industry. These models, especially those in high-risk use-cases, will need to be rigorously tested, validated (in many cases independently), and continuously monitored to ensure that they are secure, unbiased, and perform as expected. Historically, AI/ML has occurred in a “black box” with an inability to explain why a model made a given prediction and little to link real-time results to how the model was initially trained.

This presents an opportunity for a new genre of machine learning infrastructure companies to emerge, bringing a technology-driven approach to model risk management. This next-generation of model risk management solutions will allow AI/ML companies to appropriately document their model assumptions and training data; quantitatively test models for bias, fairness, and accuracy in a reproducible manner; and support enterprisewide model governance and approval workflows to ensure that all stakeholders involved in the model development process sign-off on a given AI/ML algorithm.

Over the next decade, companies will need to significantly invest in their people, processes, and technology to comply with the multiple AI/ML-related model risk management standards and regulations in the works across the United States and Europe. We believe that companies that specifically address the AI/ML risk management market with an innovative technological approach can improve both the efficiency and cost for companies to appropriately manage their models. This represents a potential multibillion-dollar market opportunity.