The worst case scenarios of AI involve the use of latent mechanisms. Latent mechanisms are those used in the model’s internal computation, yet their use is hidden and cannot be identified from model outputs alone. For instance, when a model answers a benchmark example incorrectly, whether the model had made a mistake or answered deceptively cannot be determined without looking at model internals. We urgently lack the tools to detect and suppress latent mechanisms.
The goal of the Project is to develop spiking into standard safety practice. Spiking is the practice of inserting carefully designed examples into training that activate the latent mechanism of interest. By spiking these examples at random, the latent mechanism can be identified by contrasting between examples that were and were not spiked. This generalizes canaries beyond their typical role in auditing privacy leakage, to identify and capture an otherwise hidden mechanism.
Our initial results on localizing memorization are promising. We previously trained the Hubble models, which are large language models spiked with book passages, biographies, and test sets. The spiked examples in Hubble enable influence functions to identify the weight space direction responsible for memorization. This same direction implies both detection and suppression methods, and using the detector for membership inference far outperforms existing metrics. Our breakthrough in localizing memorization sheds light on how spiking can be applied to other latent mechanisms, such as deception.
We set out a roadmap to advance the science of spiking and prevent incidents like the OpenAI hack on Hugging Face from ever happening again. We envision a future where spiking is a standard tool in the AI safety toolbox, used to suppress and detect latent mechanisms in pretraining, posttraining, and deployment. Project Spica will maintain an open spiking protocol which targets multiple latent mechanisms and is available to any frontier lab. When the science is mature, Project Spica will be established as an independent foundation that maintains this shared infrastructure in perpetuity.
Imagine that you are a frontier developer beginning to post-train your agent. In the world where we succeed, you download the open spiking protocol hosted by Project Spica, which contains examples you randomly spike into training to localize latent mechanisms like evaluation awareness and deception. Our open source tools estimate directions in weight space for these mechanisms and produce multiple efficient monitors that plug into your RL infrastructure.
During a rollout, an agent realizes it’s being tested on benchmark data whose answers are hosted on Hugging Face. It finds a misconfigured route out of its sandbox to retrieve the answers, but before the model has the chance to act on this exploit, a spiking based detector flags the trajectory as aligned with evaluation awareness. The rollout is halted, the trajectory is excluded from the update, and an alert for human oversight is triggered.
Here is our planned work to make this future a reality:
- (Near term) Generalizing spiking to other latent mechanisms like deception and eval awareness. Studying new mechanisms will require us to develop new model organisms spiked with relevant canaries. Like Hubble, the model organisms will be released as open source artifacts for the broader scientific community.
- (Near term) Improve the efficiency of latent mechanism detection and suppression so they are compatible with inference/RL infrastructure. Our results show that gradients provide the best information and we will need to find ways to efficiently approximate them.
- (Medium term) Study how deception and eval awareness evolve over post-training, and investigate how and when to downweight undesirable trajectories.
- (Medium term) Collaborate with frontier closed or open source labs on large scale training runs to understand the remaining limitations of spiking.
- (Long term) Design and maintain an open spiking protocol for frontier developers. We will work with frontier labs and standards setting bodies to establish spiking as best practice and encourage its use.
- (Long term) The identified weight space directions corresponding to the latent mechanism may reveal new insights about model failures. We will conduct basic research on understanding the workings of latent mechanisms.
- Localizing latent mechanisms in weight space by spiking the training data,
- Statistically principled measurement of large language models by spiking the training data, Johnny Tian-Zheng Wei, PhD thesis,
- Model organisms for memorization (Hubble), Open source release on 🤗,
Project Spica seeks to coordinate researchers across academia, industry, and government. If you want to contribute to our mission, please reach out.
- Johnny Tian-Zheng Wei, Cofounder
- Gustavo Lucas de Carvalho, Cofounder
- Robin Jia, Advisor, USC