ModernBERT pre-trained from scratch on various obfuscated representations of the same 10B tokens.
🔄 In a Training Loop
Daniel Gallagher
DanielGallagherIRE
AI & ML interests
I like encoders because they don't talk back.
Recent Activity
liked a dataset about 6 hours ago
nvidia/Nemotron-Personas-Vietnam liked a model about 9 hours ago
allenai/OlmoEarth-v1_3-Base liked a model about 19 hours ago
allenai/OlmoEarth-v1-BaseOrganizations
A collection of obfuscated version of a 20B-token sample of the FineWeb Edu dataset.
-
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation
Viewer • Updated • 19.4M • 923 -
DanielGallagherIRE/FineWeb-Edu-10B-Nouns-Only
Viewer • Updated • 9.71M • 663 -
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled
Viewer • Updated • 9.68M • 931 -
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered
Viewer • Updated • 9.71M • 364
ModernBERT pre-trained from scratch on various obfuscated representations of the same 10B tokens.
A collection of obfuscated version of a 20B-token sample of the FineWeb Edu dataset.
-
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation
Viewer • Updated • 19.4M • 923 -
DanielGallagherIRE/FineWeb-Edu-10B-Nouns-Only
Viewer • Updated • 9.71M • 663 -
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled
Viewer • Updated • 9.68M • 931 -
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered
Viewer • Updated • 9.71M • 364
Models pre-trained on various obfuscated versions of a sample of 10B tokens to analyse whether such obfuscated models are still performant.