LangWatch provides evaluations for language models and tests for AI agent applications.
Open source
16 alternativesLangKit
LangKit monitors language model prompts and responses and extracts signals related to safety and security.
About LangKit
LangKit is an open-source Python text metrics toolkit for monitoring language models. It extracts signals from input and/or output text, with out-of-the-box metrics for text quality (readability, complexity and grade scores), relevance (similarity between prompts and responses, and against user-defined themes), security and privacy (regex pattern matching, jailbreak and prompt-injection similarity, hallucination consistency checks, and refusal detection), and sentiment and toxicity.
The metrics are packaged as modules whose UDFs wire into the String feature UDFs that whylogs provides by default, so importing a module and instantiating a schema is enough to produce a profile. That profile can be visualized and monitored in the WhyLabs platform or analyzed by the user directly. Installation is done through PyPI with pip install langkit[all]. Some metrics depend on user-supplied inputs such as regex pattern groups and themes, and published benchmarks show throughput dropping sharply as more metric modules are enabled.
Key features
- Text quality metrics: readability score plus complexity and grade scores
- Relevance similarity scores between prompts, responses and user-defined themes
- Security checks for jailbreaks, prompt injection, refusals and regex pattern matches
- Hallucination detection through consistency checks between responses
- Sentiment analysis and toxicity analysis on text
- Python UDF modules that wire into whylogs String feature schemas
Best for: Teams using Python and whylogs to monitor LLM prompts and responses who want safety, relevance, quality and toxicity signals in one toolkit.
Good for
- Open source under the Apache-2.0 license
- Metrics plug into the existing whylogs String feature UDF collection
- Covers text quality, relevance, security, sentiment and toxicity in a single toolkit
Watch out for
- Published AWS benchmarks show much lower throughput when all metric modules are enabled
- Some metrics require user-defined inputs such as regex pattern groups and themes
- Several security signals are similarity scores against known jailbreak, prompt-injection and refusal examples
LangKit at a glance
- Website
- github.com/whylabs/langkit
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Yes · Source code
- Language
- English
- Last checked
- Oct 6, 2026 · Online when checked
- Listed
- Oct 5, 2026
LangKit alternatives
Tools and sites you could use instead, ranked by how closely they match.
promptfoo tests prompts, agents, and retrieval systems with evaluations and security testing.
Open source
16 alternatives
OpenCodeReview combines deterministic checks and language model agents for code review with line-level comments.
Open source
17 alternatives
rehydra scans LLM prompts for accidental personal information before they reach a model.
Open source
17 alternativesPopular in AI Code Review & Testing
Well-known listings from the same topic.
Prediction Guardpredictionguard.comGreptilegreptile.comGitAutogitauto.aipromptfoopromptfoo.devRefractionrefraction.devKatalonkatalon.com
LangKit FAQ
How is LangKit installed?
From the Python Package Index, using pip install langkit[all].
How does LangKit work with whylogs?
Its modules contain UDFs that automatically wire into whylogs' default String feature UDFs; after importing the modules and initializing a custom schema, calling why.log on prompt and response text produces default whylogs text metrics plus the metrics from the imported modules.
Where can the resulting profile be viewed?
It can be visualized and monitored in the WhyLabs platform, or further analyzed by the user directly.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.