Innovations in artificial intelligence (AI) and machine learning (ML) are leading organizations to reevaluate the possibilities that these technologies can offer. However, deploying these models at an enterprise scale presents challenges, especially in terms of meeting security and governance requirements. In this context, MLOps is presented as a critical solution by automating governance processes, reducing the time needed to bring proof of concepts to production environments, and ensuring the quality of deployed models.
ML models in production are not static artifacts; they reflect the environment in which they are deployed and therefore require exhaustive monitoring mechanisms to ensure model quality, absence of biases, and feature importance. Organizations often want to implement additional checks to ensure that the model meets their organizational standards before deployment. Automating these checks allows for regular and consistent repetitions, rather than relying on sporadic manual checks.
This article illustrates how to use common architectural principles to transition from a manual monitoring process to an automated one using AWS services such as Amazon SageMaker Model Registry and Amazon SageMaker Pipelines. This enables organizations to deliver innovative solutions to their clients while maintaining compliance in their ML workloads.
As AI becomes ubiquitous, its use in sensitive contexts, such as interacting with users through chatbots in tax agencies, raises the need for these systems to align with organizational guidelines. In this scenario, organizations may have dozens or even hundreds of models in production, and require robust mechanisms to ensure that each model is properly reviewed before deployment.
Traditionally, organizations have established manual review processes to prevent outdated code from reaching the public, using committees or review boards. With the rise of continuous integration and continuous delivery (CI/CD), MLOps can reduce the need for manual processes, increasing the frequency and depth of quality checks. Through automation, in-demand skills, such as data and model analysis, are scaled across various product teams.
In this article, we use SageMaker Pipelines to define the necessary compliance checks as code. This allows for the introduction of arbitrarily complex analyses without being limited by the tight schedules of highly technical individuals. Automation takes care of repetitive analysis tasks, allowing technical resources to focus on improving the quality and depth of the MLOps pipeline to ensure that checks function properly.
Deploying an ML model in production typically requires approval of at least two artifacts: the model and the endpoint. In our example, the organization approves a model if it passes quality, bias, and feature importance checks. Additionally, the endpoint is approved if it functions as expected in a simulated production environment. In future posts, guidance will be provided on deploying a model and implementing compliance checks. In this article, the extension of this process to large language models (LLMs) is discussed, which produce a variety of outputs and present complexities in terms of automatic quality assurance checks.
The solution described is deployed across multiple accounts within an AWS organization, with centralized components such as a model registry in SageMaker, ML project templates in SageMaker Projects, model cards in Amazon SageMaker, and container images in Amazon ECR. We use an isolated environment to deploy and promote across multiple environments. The strategy can be adjusted based on the organization’s posture, providing a centralized, adjustable model tailored to strict compliance requirements through verification.
The article provides a detailed view of how to build the pipeline using SageMaker Pipelines, integrating multiple AWS services to automate model approval. It explains how to create and register models in SageMaker, define the pipeline with evaluation steps and model state updates, and execute the pipeline based on events through Lambda and EventBridge, optimizing the manual review process.
Finally, this approach is applied to generative AI models, considering complexities and metrics of interest such as memorization, disinformation, bias, and toxicity using different reference datasets. This allows for effectively handling the complex aspects of generative model behavior.
In conclusion, this article has discussed a solution to automate compliance checks for AI/ML models, increasing the speed and quality of deployment through CI/CD techniques applied to the ML modeling lifecycle, enabling organizations to scale in the era of generative AI. If you have comments or questions, please leave them in the comments section.
Source: MiMub in Spanish
