A JavaScript Native PyTorch-aligned Machine Learning Framework, built from scratch on WebGPU.
Open source
24 alternativesSemantic Cache Router
Semantic Cache Router presents a distributed semantic cache and stateful routing system.
About Semantic Cache Router
Semantic Cache Router is a distributed system for reducing repeated LLM calls. A Go gateway routes requests by system_prompt using consistent hashing to Python workers, which check a Redis Stack vector index for semantically similar past responses and skip the LLM call on a match.
It matches queries with 384-dim embeddings at similarity 0.8 or higher and stores misses with a TTL. It requires Docker with Compose v2, 4 GB RAM recommended, and downloads about 1 GB of images and model weights on first run.
Key features
- Semantic vector cache returns stored responses for similar queries
- ANN search on 384-dim embeddings at similarity threshold 0.8
- Consistent hashing pins same system_prompt to same worker
- System prompt included in cache key for namespace isolation
- Embedding model pre-loaded at container startup
- Swappable _call_llm method keeps routing provider-agnostic
Best for: Developers operating multi-worker LLM services with repeated queries in the same context.
Good for
- Measured ~25ms cache hits versus ~180ms LLM calls in local testing
- Worker affinity without infrastructure-level session pinning via consistent hashing
- Adding a worker migrates only about 1/N keys
Watch out for
- Requires Docker with Compose v2 and 4 GB RAM recommended
- First run downloads about 1 GB of images and model weights
- EvictionPolicy ships as SphereLFU stub with TTL storage
Semantic Cache Router at a glance
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Not stated
- Language
- English
- Last checked
- Oct 6, 2026 · Online when checked
- Listed
- Oct 5, 2026
Semantic Cache Router alternatives
Tools and sites you could use instead, ranked by how closely they match.
This repository contains utilities intended for use with Llama models.
PromptSite provides prompt version management for tracking, experimentation, and debugging LLM prompts.
Open source
24 alternatives
DeepEval is an LLM evaluation framework, according to its repository description.
Open source
24 alternatives
OneCompression is a Python package for compressing large language models.
Open source
24 alternatives
Notte provides cloud browser infrastructure and web automation for AI and coding agents.
Popular in AI Infrastructure & MLOps
Well-known listings from the same topic.
Guild AIguild.aiFlow AIflow-ai.comsystemprompt.iosystemprompt.ioCoreWeavecoreweave.comKili Technologykili-technology.comAPI docsdocs.x.ai
Semantic Cache Router FAQ
How does request routing work?
The system_prompt field is hashed with FNV-1a on a consistent hash ring with 150 virtual nodes per worker, so requests with the same system prompt go to the same worker.
How does cache matching work?
The worker creates an embedding with sentence-transformers/all-MiniLM-L6-v2 and searches Redis Stack for past responses within cosine distance 0.2, equivalent to similarity 0.8. Hits skip the LLM; misses call the LLM and store the result with a TTL.
What is needed to run it?
Docker with Compose v2 and 4 GB RAM recommended, then run docker compose up --build -d. The first run downloads about 1 GB of images and model weights.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.