BERT-Tiny AI-vs-Human Text Detector (demo)

A very small model (about 4.4M parameters, 17.5 MB) that labels an English answer or paragraph as HUMAN (written by a person) or AI (written by ChatGPT). It was made as a teaching demo. Read the limitations before using it.

  • Base model: google/bert_uncased_L-2_H-128_A-2 (BERT-Tiny)
  • Data: Hello-SimpleAI/HC3. These are human answers and ChatGPT (early 2023) answers to the same questions, from the finance, medicine, open_qa and wiki_csai subsets.
  • Test accuracy: 96.9% (F1 0.968). It was 68.5% before fine-tuning.

How to use

from transformers import pipeline

detector = pipeline("text-classification", model="spatialgeography/bert-tiny-ai-human-text-detector")
print(detector("There are several factors to consider. Ultimately, it depends on your situation.", truncation=True))
# [{'label': 'AI', 'score': ...}]

Training data preparation

  • Balanced: each question keeps the same number of human and AI answers, giving 6,439 + 6,439 for training and 716 + 716 for testing.
  • Split by question: 90% of questions are for training and 10% for testing, so no question appears in both.
  • Cleaned: whitespace is collapsed and spaces before punctuation are removed, so the model can't use formatting shortcuts such as the line breaks found only in AI answers.
  • Answers shorter than 10 words were dropped. Inputs are truncated to 256 tokens.

Training

Setting Value
Epochs 3 (1,209 steps)
Batch size 32
Learning rate 1e-4, linear decay, 100 warm-up steps
Max length 256 tokens
Hardware Laptop CPU (4 threads), 18 min 41 s
Epoch Accuracy F1 Precision (AI) Recall (AI)
0 (before) 68.5% 0.641 0.746 0.561
1 94.7% 0.945 0.988 0.905
2 96.3% 0.962 0.980 0.946
3 96.9% 0.968 0.977 0.959
Test subset Texts Accuracy
medicine 266 98.5%
finance 756 98.0%
open_qa 262 94.3%
wiki_csai 148 92.6%

Limitations (important)

  • Not a reliable AI detector. The 96.9% applies only to HC3-style questions and answers. On other topics it fails. For example, it labelled a ChatGPT-style paragraph about remote sensing as HUMAN with 0.99 confidence.
  • It learned the style of ChatGPT from early 2023. Newer models (GPT-4/5, Claude, Gemini, Llama) and edited or paraphrased AI text will often be labelled HUMAN.
  • Short texts, non-native English and formal human writing can be wrongly labelled AI.
  • Do not use it to judge students' or anyone's work. AI-text detection, even with much larger models, has high error rates and can be unfair to some writers.

License

The model weights are released under CC-BY-SA-4.0 to match the HC3 dataset license.

Downloads last month
4
Safetensors
Model size
4.39M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for spatialgeography/bert-tiny-ai-human-text-detector

Finetuned
(142)
this model

Dataset used to train spatialgeography/bert-tiny-ai-human-text-detector

Evaluation results

  • accuracy on HC3 (finance, medicine, open_qa, wiki_csai), 10% held-out questions
    self-reported
    0.969
  • f1 on HC3 (finance, medicine, open_qa, wiki_csai), 10% held-out questions
    self-reported
    0.968