Amazon has announced the general availability of Amazon Bedrock Intelligent Prompt Routing, an innovative tool that promises to optimize efficiency in interacting with language models. This functionality, which had been in testing since December, allows for intelligent routing of requests between different models within the same family, improving both costs and the quality of the responses provided.
The technology behind Amazon Bedrock Intelligent Prompt Routing is based on dynamically predicting the quality of response from different models to a specific request. This allows for directing each request to the most suitable model, optimizing the relationship between cost and quality. This advancement represents a significant milestone in the implementation of generative artificial intelligence applications, making it easier for users to automatically and optimally route large language models.
With the official launch, the tool has incorporated important improvements driven by user feedback and extensive internal testing. Users have the option to use default prompt routers by Amazon Bedrock or create customized configurations that adjust performance to their specific needs. Default routers simplify implementation, offering ready-to-use solutions that require minimal configuration.
Furthermore, Amazon has expanded the variety of model families available, incorporating options from Nova, Anthropic, and Meta, with standout models like Claude and Llama. In this new phase, users also have the ability to create custom routers, choosing the models they want to use and their routing configuration.
An important aspect of this functionality is the improvement in latency, as it has been able to reduce component overhead time by over 20%, achieving a performance of approximately 85 milliseconds at the 90th percentile. This translates into tangible benefits in both latency and costs, prioritizing the use of less expensive models without compromising task accuracy.
Internal testing has shown that using Amazon Bedrock Intelligent Prompt Routing can generate average savings of 60% compared to using more expensive models. However, users are recommended to conduct tests in their own use cases to better understand the benefits, as effectiveness may vary depending on the type of task and models selected.
To facilitate the adoption of this tool, Amazon has provided a series of resources and guides accessible through the AWS management console, command line interface, or API. This aims to encourage developers and companies to make the most of this innovative tool in the field of generative artificial intelligence.
via: MiMub in Spanish