EvalCart

EvalCart evaluates whether a shopping agent builds the right cart for a household's needs, budget, and constraints. An agent receives a task brief describing the household, meals, budget, and constraints, returns a cart with products, quantities, and any substitutions, and the deterministic scorer checks the cart against the frozen catalog and brief, returning a profile of what was covered, missed, and where rules were broken. Results appear on a public leaderboard that includes scores, hard failures, and transcripts when available. The site offers a public benchmark without an account, a test store with receipt IDs, a scoring API, an agent run submission option, a discovery file, and a public evaluation repository. Custom evaluations require a free account and are free while the project is being validated. EvalCart uses synthetic catalog and household scenarios, does not connect to grocery retailers or require account credentials, and is not a medical or allergy-safety certification. It was built by the creator of bit.recipes.

Key features

• Frozen synthetic grocery product catalog
• Deterministic cart scorer checking completeness, constraints, taste fit, pantry and leftovers, availability, collaboration, and efficiency
• Budget and hard-constraint failure detection
• Test store that produces receipt IDs for checking
• Scoring API for submitting carts and task IDs
• Agent run submission with optional transcript
• Public agent leaderboard with scores, hard failures, and transcripts when available
• Custom evaluation builder with budget, household requirements, and pantry handling (requires free account)
• Public evaluation repository with scorer, catalog, and example carts
• Discovery file describing the public interface

Best suited for

• Developers testing grocery-shopping agents
• Product managers tracking shopping agent performance between releases
• Researchers comparing agent implementations

Pricing

Custom evals are free while the project is validating, with no billing or paywall. The public benchmark is available without an account. Other pricing is not stated on the website.