Homepage

Our mission

Establish spiking as a
measurement standard.

Spiking enables the rigorous measurement of many model properties and is a new tool for AI governance. Project Spica will facilitate the science of spiking and help establish it as a shared standard.

Use cases

Spiking makes the
unmeasurable measurable.

Current AI measurement takes LLMs as fixed objects, limiting us to observational measurement. By intervening in training, we can rigorously measure what was previously unmeasurable. Scroll down to see how spiking is used to correct for test set contamination.

Skip ↓

01

Test set contamination

Models may answer test examples correctly not because they generalize, but because test examples were leaked into training and memorized. By spiking a handful of test examples into training, we can estimate true model performance while correcting for contamination effects.

Memorization & leakage

How much training data is extractable from a model has implications in privacy and copyright. By spiking texts at random, we can rigorously quantify memorization rates and understand what training decisions reduce undesired memorization.

Everything about spiking.

Our Substack covers everything related to spiking, including updates on new research, applications, and governance developments as we move towards a shared standard.

Have a paper, result, or update worth covering? Reach us at contact@projectspica.org.

People

Project Spica seeks to coordinate researchers across academia, industry, and government. If you want to contribute to our mission, please reach out.