Simon Willison’s Weblog

Releasing the largest multilingual open pretraining dataset (via) Common Corpus is a new "open and permissible licensed text dataset, comprising over 2 trillion tokens (2,003,039,184,047 tokens)" released by French AI Lab PleIAs.

This appears to be the largest available corpus of openly licensed training data:

• 926,541,096,243 tokens of public domain books, newspapers, and Wikisource content
• 387,965,738,992 tokens of government financial and legal documents
• 334,658,896,533 tokens of open source code from GitHub
• 221,798,136,564 tokens of academic content from open science repositories
• 132,075,315,715 tokens from Wikipedia, YouTube Commons, StackExchange and other permissively licensed web sources

It's majority English but has significant portions in French and German, and some representation for Latin, Dutch, Italian, Polish, Greek and Portuguese.

I can't wait to try some LLMs trained exclusively on this data. Maybe we will finally get a GPT-4 class model that isn't trained on unlicensed copyrighted data.