evalship

evalship is a quality control tool for LLM evals. At setup, a coding agent reviews scorers, judges and prompts to find evals that pass when they shouldn't and rules no test case checks. It then watches pull requests, comments with which evals regressed and the likely cause, flags untested AI calls, can block merges that break evals, and tests prompt and model variants to show which changes beat chance. It also offers a weekly email eval report. The page states it works with any eval framework and mentions Langfuse and Braintrust. Pricing is listed with Free, Team, Pro and Business plans and a launch offer of 60% off for the next 9 accounts.

Key features

• Finds flaws in existing LLM evals during setup, such as evals that pass when they shouldn't and rules no test case checks
• Comments on every pull request with which evals broke, the likely cause, and a report comparing main against the PR
• Flags AI calls changed in a pull request that no eval exercises
• Separates flaky evals from regressions by checking how often evals fail by chance on main
• Blocks merges that break evals when enabled as a required check
• Optimizes prompts and models by testing variants and reporting whether gains exceed chance
• Provides fix instructions that a coding agent can use to address eval flaws
• Sends a weekly email eval report
• Works with any eval framework, including Langfuse and Braintrust per the page
• Eval history retention of 30 days to 2 years depending on plan

Best suited for

• Teams maintaining LLM evals in CI for AI applications
• Developers reviewing pull requests that change prompts or models
• Teams wanting to compare prompt or model changes on cost and pass rate

Pricing

Free ($0), Team ($7.60/mo or $76/yr), Pro ($19.60/mo or $196/yr), Business ($79.60/mo or $796/yr) listed with a launch offer of 60% off forever for the next 9 accounts; prices shown are the discounted figures on the page