Simon Willison’s Weblog

smevals—a small eval suite for evaluating models, prompts, and harnesses. I've been working with Jesse Vincent's Prime Radiant applied AI research lab building out this evals framework to help answer questions about the capabilities of different models.

The result is smevals, a new tool for running small eval suites across different model configurations and grading the results.

This blog entry describes the tool in detail. Here's the 10 second version:

1. Tell your coding agent to run uvx smevals docs to learn the tool (this outputs the README)
2. Then tell it to build you an eval suite

Once you've created an eval - which takes the form of a directory with some YAML files - you can run it against models like this:

uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6

Runs are treated separately from grading operations - you can grade your runs (against your defined set of checks) using:

uvx smevals grade path-to-eval/

Then you can run a localhost web server to explore the results:

uvx smevals serve path-to-eval/

Or run the smevals build command to build that report as static HTML, which you can then host anywhere. Here's an example showing an eval suite I built to evaluate how well models can write haikus.

The most time-consuming part of this project was figuring out the vocabulary for it! Here's what I settled on, quoted from the announcement:

• An eval is a collection of challenges designed to answer a question about a model, for example, how good is that model at generating SVGs?
• Each eval is a collection of tasks. A task is a specific challenge, for example "Generate an SVG of a pelican riding a bicycle".
• When you run the eval you do so against one or more configs. Each config specifies a model to be evaluated, but may also include other parameters to test, such as different system prompts, model parameters, or agent harnesses.
• A run records what happened when a specific config was used to execute a specific task. A runner is the script that executes a run.
• Once you have collected one or more runs, you need to evaluate the results to see how well the model (or config) did. This is done by a grader, which produces a grade.
• Each grader runs a sequence of checks. These can be simple operations, like checking for a specific string in the output, or confirming that the output is valid XML. They can also be more complicated custom operations (implemented as scripts called checkers), including using other models to answer questions about the run.

I've been trying to figure out an approach I like for evals for several years now. smevals is my third iteration on the idea and it feels right to me. I'm looking forward to expanding this more in the future, as well as pointing it at some of my own projects.

# 9:15 pm / projects, ai, generative-ai, llms, llm, evals, jesse-vincent

Oxide and Friends: The Open Weight Revolution with Simon Willison. On Monday Bryan Cantrill and Adam Leventhal invited me to join their podcast to talk about the wild week we've had - with Kimi K3 showing open weight models can stand toe-to-toe with proprietary frontier ones, accidental cybersecurity attacks, and public letters about Open Weights and American AI Leadership signed by almost every big name in AI (with one notable exception).

It was a great conversation, even though it's already out-of-date! DeepSeek V4 Flash 0731 and Anthropic's own embarrassing cyber incident would absolutely have made the cut if we had recorded just a few days later.

We also talk about Golden Gate Claude, the Zizians, Alameda wild turkey attacks, Soviet Marburg virus research, the Lead-crime hypothesis, and a bunch of other worthy digressions.

Finally, we revisited some of our predictions from January, and we added a new Pope prediction:

Prediction by the end of this year: the Pope says something about open models.

# 9:33 pm / predictions, ai, generative-ai, local-llms, llms, oxide, bryan-cantrill, podcast-appearances, ai-in-china, ai-security-research

deepseek-ai/DeepSeek-V4-Flash-0731 (via) The latest release in DeepSeek's V4 family, "with substantially enhanced agentic capabilities". It's 304 billion parameters - 167GB on Hugging Face - but it appears to punch well above its weight.

Artificial Analysis rank it ahead of MiniMax M3 - a 428B model. It's $0.14/million input and $0.27/million output pricing means this may currently be the best value-per-intelligence model out there. It's looking very good on the Intelligence Index vs. Cost per Intelligence Index Task chart:

I got a disappointing pelican from it using the default reasoning level via OpenRouter:

But when I bumped reasoning level up to high I got something much better:

llm -m openrouter/deepseek/deepseek-v4-flash-0731 -t pelican -o reasoning_effort high

# 11:59 pm / ai, generative-ai, llms, pelican-riding-a-bicycle, deepseek, llm-release, openrouter, ai-in-china, artificial-analysis