Optimizing cost and latency with Amazon Bedrock prompt caching | Amazon Web Services
Prompt caching in Amazon Bedrock can reduce your input token costs by up to 90 percent when you repeatedly send the same context to foundation models, based on Amazon Bedrock prompt caching pricing. Without caching, a 10,000-token contract sent alongside 50 user questions means 500,000 input tokens billed at full price for content the model has already processed.
You can mitigate this issue by shortening prompts, reducing context windows, or implementing application-level caching. Each option involves a trade-off:
- Shortened prompts reduce token count but might also reduce context quality.
- Smaller context windows lower cost at the expense of the model’s ability to reason over complete information.
- Response caching handles identical queries well, yet provides no benefit when the same context is paired with different questions.
Prompt caching in Amazon Bedrock helps reduce this challenge at the infrastructure level. When you cache parts of your conversation context (system prompts, documents,...
Copyright of this story solely belongs to aws.amazon.com. To see the full text click HERE
