FromZero

We build stuff and share it with the community.

That's pretty much it. And yes, "stuff" was the right word to use.

Why "FromZero"?

FromZero is inspired by Re:Zero - Starting Life in Another World. Just like the main character, all of our models begin from zero.

Thankfully, they don't spend their time dying repeatedly or having mental breakdowns.

I mean, training is already painful enough.

Our goal

Well, there isn't actually a definitive goal we're chasing. I mean, yeah, we want to be the greatest. But so does everyone else.

We're just here to build weird models, share them with the community, and grow side-by-side with everyone.

Current Models

• Syn-2.6M: A 2.6M-parameter Tiny Language Model (TLM) trained only on synthetic data (tiny-textbooks, tiny-stories-v2, longpage, and finephrase).
• Er-Large: A 31M-parameter model trained on 34B tokens. It achieves almost state-of-the-art performance on many benchmarks.

Future Releases

• Orez - Tiny (1.3M), Small (5M), Medium (12.4M), Large (30M): A family of Small Language Models ranging from 1.2M to 30M parameters. Each of them will be massively overtrained on a minium of 1000+ tokens per parameter.

Discord

Join our Discord community: FromZero.

Website

Our website: AskJASP

If You Want to Join Us

We are always looking for more members. If you'd like to join, open a discussion under this README and tell us why. We'll review your request and either approve or reject it.