Retrieval quality depends on which embedding model produced the vectors, and the right model is not the same for every corpus in an organisation. Pinning one model for the whole installation forces a compromise on all of them. This release makes the embedding model a per-request choice, served by a dedicated inference service, and separately reduces the cost of the conversational path by caching the parts of a prompt that do not change.
A dedicated embedding service
- ONNX inference over gRPC. Dense vector generation runs in a microservice executing ONNX models, reached over a unified gRPC service definition rather than embedded in the application runtime.
- Model chosen per request. The request carries the model to use, so one corpus is embedded with a multilingual model and another with a domain-specific one against the same service.
- Configured model carried through. The configured model name travels from the platform's AI configuration through the execution to the service, so what an administrator selected is what runs.
- Context window declared. An embedding model's specification carries its context token limit, so text is chunked against the model's actual capacity rather than a global assumption.
Prompt caching on the conversation path
- Cached where the provider supports it. The conversational assistant marks the stable portion of a prompt for provider-side caching, so a long system context is not re-billed on every turn.
- Cache usage reported. Cache read and cache write token counts are tracked and surfaced alongside ordinary usage, so the saving is measured rather than assumed.
- Model configuration on demand. A specific model's configuration is fetched directly, so a caller resolves the parameters of the model it is about to use without loading the whole catalogue.
Installations that do not deploy the embedding service are unaffected, and prompt caching engages only against providers that offer it — where they do not, the conversation path behaves exactly as before.