Thursday, 24th April 2025
Exploring Promptfoo via Dave Guarino’s SNAP evals
I used part three (here’s parts one and two) of Dave Guarino’s series on evaluating how well LLMs can answer questions about SNAP (aka food stamps) as an excuse to explore Promptfoo, an LLM eval tool.
[... 692 words]OpenAI: Introducing our latest image generation model in the API. The astonishing native image generation capability of GPT-4o - a feature which continues to not have an obvious name - is now available via OpenAI's API.
It's quite expensive. OpenAI's estimates are:
Image outputs cost approximately $0.01 (low), $0.04 (medium), and $0.17 (high) for square images
Since this is a true multi-modal model capability - the images are created using a GPT-4o variant, which can now output text, audio and images - I had expected this to come as part of their chat completions or responses API. Instead, they've chosen to add it to the existing /v1/images/generations API, previously used for DALL-E.
They gave it the terrible name gpt-image-1 - no hint of the underlying GPT-4o in that name at all.
I'm contemplating adding support for it as a custom LLM subcommand via my llm-openai plugin, see issue #18 in that repo.
Introducing Datasette for Newsrooms. We're introducing a new product suite today called Datasette for Newsrooms - a bundled collection of Datasette Cloud features built specifically for investigative journalists and data teams. We're describing it as an all-in-one data store, search engine, and collaboration platform designed to make working with data in a newsroom easier, faster, and more transparent.
If your newsroom could benefit from a managed version of Datasette we would love to hear from you. We're offering it to nonprofit newsrooms for free for the first year (they can pay us in feedback), and we have a two month trial for everyone else.
Get in touch at hello@datasette.cloud if you'd like to try it out.
One crucial detail: we will help you get started - we'll load data into your instance for you (you get some free data engineering!) and walk you through how to use it, and we will eagerly consume any feedback you have for us and prioritize shipping anything that helps you use the tool. Our unofficial goal: we want someone to win a Pulitzer for investigative reporting where our tool played a tiny part in their reporting process.
Here's an animated GIF demo (taken from our new Newsrooms landing page) of my favorite recent feature: the ability to extract structured data into a table starting with an unstructured PDF, using the latest version of the datasette-extract plugin.
