1. ← Home
Login
A language model that only knows elementary school curriculum
~ai.llms~papers
littlelearner-ll.github.io 3 days ago

Summary

From the article:

Modern LMs are trained on everything at once, so it is hard to tell whether a new skill was learned or merely elicited. We constrain the training distribution itself: an 88B-token corpus filtered to the U.S. elementary-school curriculum, with models trained from scratch on it and matched unfiltered controls.

[...]

Can standard interventions push a model past what its pretraining data taught it? With the boundary under experimental control, we can ask cleanly. In our experiments, each intervention amplifies in-scope ability; none of them meaningfully improves out-of-scope performance.