google/paligemma-3b-pt-224
Image-Text-to-Text โข 3B โข Updated โข 256k โข 615
๐ค Open to Collab
Aliasgar Khimani
NovusEdge
AI & ML interests
None yet
Recent Activity
liked a model 6 days ago
google/paligemma-3b-pt-224 liked a dataset 11 days ago
NovusEdge/revealed updated a dataset 11 days ago
NovusEdge/revealedOrganizations
liked a
model 6 days ago
liked a
dataset 11 days ago
updated a
dataset 11 days ago
published a
dataset 11 days ago
updated a
model 11 days ago
published a
model 11 days ago
replied to ginigen-ai's post 3 months ago
@RiverRider Just checked out the repo. The semiotic divergence angle is kinda interesting ngl, tracking where meaning splits across communities rather than pure confidence calibration. How are you seeing it perform on domain-specific jargon vs everyday terms?
reacted to ginigen-ai's post with ๐ฅ 3 months ago
Post
10538
๐ง Does your LLM know when it's about to be wrong?
Most leaderboards measure accuracy. We measure metacognition โ whether a model catches its own errors. Benchmark + leaderboard + adapters, all open. ๐
The surprise: even a K-AI #1 model (JGOS-31B-Citizen) is the strongest on multiple-choice traps (trap_rate 0.005 โ ~2 misses in 400) yet blind to its own free-form mistakes (self-confidence AUROC = 0.5, pure random). A tiny base-frozen adapter recovers that signal.
Two independent axes (never compared across a row): โ trap_rate โ does it fall for tempting trap options? (lower = stronger) โก adapter gain ฮ โ how much a lightweight adapter catches errors the model itself misses. (higher = more adapter value)
What's open: ๐ 300+100 trap problems (each with a hidden trap + TICOS type) ๐ 24-model leaderboard ๐งฉ 11 per-model adapters โ adapters, NOT fine-tunes (base stays frozen; the adapter just reads the hidden state โ P(wrong))
Submit any HF model โ auto-scored daily at 09:00 KST and added to the board.
๐ Leaderboard โ ginigen-ai/Metacognition-Leaderboard-Space
๐ Benchmark โ ginigen-ai/Metacognition-Bench
๐งฉ Adapters โ https://huggingface.co/collections/FINAL-Bench/metacognition-adapters-6a42c032e6beb803dd032961
๐ Article โ https://huggingface.co/blog/ginigen-ai/metacognition
Benchmark by ginigen-ai ยท Adapters by FINAL-Bench (Darwin/Chimera platform + AETHER metacognition tech).
Most leaderboards measure accuracy. We measure metacognition โ whether a model catches its own errors. Benchmark + leaderboard + adapters, all open. ๐
The surprise: even a K-AI #1 model (JGOS-31B-Citizen) is the strongest on multiple-choice traps (trap_rate 0.005 โ ~2 misses in 400) yet blind to its own free-form mistakes (self-confidence AUROC = 0.5, pure random). A tiny base-frozen adapter recovers that signal.
Two independent axes (never compared across a row): โ trap_rate โ does it fall for tempting trap options? (lower = stronger) โก adapter gain ฮ โ how much a lightweight adapter catches errors the model itself misses. (higher = more adapter value)
What's open: ๐ 300+100 trap problems (each with a hidden trap + TICOS type) ๐ 24-model leaderboard ๐งฉ 11 per-model adapters โ adapters, NOT fine-tunes (base stays frozen; the adapter just reads the hidden state โ P(wrong))
Submit any HF model โ auto-scored daily at 09:00 KST and added to the board.
๐ Leaderboard โ ginigen-ai/Metacognition-Leaderboard-Space
๐ Benchmark โ ginigen-ai/Metacognition-Bench
๐งฉ Adapters โ https://huggingface.co/collections/FINAL-Bench/metacognition-adapters-6a42c032e6beb803dd032961
๐ Article โ https://huggingface.co/blog/ginigen-ai/metacognition
Benchmark by ginigen-ai ยท Adapters by FINAL-Bench (Darwin/Chimera platform + AETHER metacognition tech).
- 11 replies
replied to ginigen-ai's post 3 months ago
sweet! how about we take it over on twitter: https://x.com/0kaliasgar :))
replied to ginigen-ai's post 3 months ago
We're building something complementing to this @ https://engrammic.ai
You down to have a chat sometime about metacognition and externalized epistemics? ๐
