Sentiment analysis: train the baseline, then break it
A movie review goes in, positive or negative comes out.
The task is simple to state and the models are simple enough to inspect by hand, which makes it a good place to find out what a 90% accuracy is actually worth.
The model here is deliberately old school.
No transformers, no GPU, two scikit-learn pipelines that train in under ten seconds on a laptop.
A baseline you fully understand is worth more than a big model you don't when the next question is "where does this fail".
The data
The dataset is the IMDB review set: 25 000 labelled reviews for training and 25 000 for testing, split evenly between positive and negative.
Two columns, text and label, where 1 is positive.
1 Kept my attention from start to finish. Great performances added to this tremendous film. Mr. Pacino once again gives us another brilliant character to enjoy.
0 Long, boring, blasphemous. Never have I been so glad to see ending credits roll.
1 GREAT movie and the family will love it!! If kids are bored one day just pop the tape in and you'll be so glad you did!!!<br /><br />~~~Rube<br /><br />i luv raven-s!Before training anything I looked at what was actually in it.
train = pd.read_json("data/train.json")
test = pd.read_json("data/test.json")
print(train.duplicated().sum())
print(len(set(train.text) & set(test.text)))
print(train.text.str.contains("<br").mean())
print(train.text.str.split().str.len().median())- 96 duplicated rows in the training set. Dropped with
drop_duplicates(). - 123 reviews that appear in both train and test. Not enough to move the score, but exactly the kind of thing that quietly inflates one.
- 59% of reviews contain literal
<br />tags from the scraping. The tokenizer turns that into the tokenbr, which then becomes a feature with a learned weight. - The median review is 174 words. That number matters later.
Two models
Both pipelines have the same shape.
Turn the text into a sparse vector of word and word-pair counts, then fit a linear classifier on top.
nb = Pipeline([
("vec", CountVectorizer(ngram_range=(1, 2), max_df=0.9)),
("clf", MultinomialNB(alpha=1.0)),
])
logreg = Pipeline([
("vec", TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_df=0.9, sublinear_tf=True, stop_words=["br"])),
("clf", LogisticRegression(C=20, solver="liblinear", max_iter=1000)),
])MultinomialNB on raw counts is the textbook starting point.LogisticRegression on TF-IDF is the textbook second step, and it changes two things: rare words count for more than common ones, and sublinear_tf means a word repeated ten times does not count ten times as much.
That second property comes back in the attack section.
for m in (nb, logreg):
m.fit(train.text, train.label)
print(accuracy_score(test.label, m.predict(test.text)))| model | imdb test accuracy |
|---|---|
| naive bayes, counts | 85.7% |
| logistic regression, tf-idf | 90.6% |
A confession about the first version of this.
The Naive Bayes model was the last cell I ran, so it was the one I saved, even though the logistic regression above it was five points better on the held-out set.
I then handed that file to an external checker which reported a perfect score.
A perfect score from a bag-of-words model is not a result, it is a bug report about the evaluation.
The number to trust is the one on the 25 000 reviews the model never saw, and the model to keep is the one that wins there.
I did try to tune further.
The regularisation strength barely matters, with 3-fold cross validation flat at 88.0% for C anywhere from 5 to 50.
Trigrams add 0.05 points for double the vocabulary.
A LinearSVC on the same features lands on 90.7%.
The knobs are done at this point, and the interesting findings are elsewhere.
What it learned
A linear model is a list of weights, so you can just read it.
vec, clf = logreg.named_steps["vec"], logreg.named_steps["clf"]
names = vec.get_feature_names_out()
order = clf.coef_[0].argsort()
print(names[order[:20]])
print(names[order[-20:]][::-1])The strongest weights in the logistic regression are what you would hope for.worst, awful, waste, boring on one side and great, excellent, perfect, wonderful on the other.
Then come the odd ones.today, bit, both and still are positive, and script is negative.
Those are not sentiment, they are habits of how people write reviews: "still holds up today", "the script is".
Naive Bayes is worse.
Its most polarising word pairs, among those seen at least 50 times, are uwe boll, seagal, prom night and thunderbirds on the negative side, and gunga din, red sox and midnight cowboy on the positive side.
The model has memorised which directors and films the IMDB crowd disliked around 2006.
A positive review of a Uwe Boll film starts the race with a handicap.
Other datasets
A number on the test split of the training distribution is the best case.
I pulled five other labelled sentiment sets from Hugging Face with the datasets package, binarised them to the same two columns, and scored the IMDB models on them without retraining.
The tweet set has a neutral class, which I dropped.
tweets = load_dataset("cardiffnlp/tweet_eval", "sentiment", split="test")
tweets = tweets.filter(lambda r: r["label"] != 1)
df = pd.DataFrame({"text": tweets["text"], "label": [int(l == 2) for l in tweets["label"]]})They look nothing like each other.
rotten_tomatoes 0 there are many definitions of 'time waster' but this movie must surely be one of them .
sst2 0 it 's too self-important and plodding to be funny , and too clipped and abbreviated to be an epic .
yelp 0 Worst Espresso EVER. And my steak and egg combo was ICE cold. If I could give 0 stars I would.
amazon 1 My favorite all-time CD. Great for a pick-me-up or driving in my car with the sound cranked up. I love it.
tweets 0 How can you #draintheswamp if you keep inviting swamp creatures into your administration? @user| dataset | what it is | naive bayes | logreg |
|---|---|---|---|
| imdb | long movie reviews | 85.7% | 90.6% |
| rotten tomatoes | one-line critic snippets | 76.9% | 77.0% |
| sst2 | single sentences from reviews | 80.0% | 79.2% |
| yelp | restaurant and business reviews | 79.1% | 85.8% |
| amazon | product reviews | 80.2% | 83.0% |
| tweets | politics and everything else | 62.8% | 66.4% |
Three things stand out.
Short text hurts more than a new domain.
Rotten Tomatoes and SST-2 are still movie reviews, with the same vocabulary, and the model loses 13 points.
A 174 word review gives the classifier dozens of words to vote with.
A 20 word snippet gives it two or three.
Yelp and Amazon are a different domain altogether, but the reviews are long, and they hold up far better.
On tweets it is a coin flip with a bias.
Only 37% of the tweets are positive, so always answering negative would already score 62.6%.
Naive Bayes gets 62.8%.
The logistic regression is a few points above that, and the confusion matrix shows why: a third of the negative tweets are classified as positive.
The tweets are angry about politics, with words like corruption, trolls and deplorables, and none of those are negative in a movie review.
It is not that the model has never seen the words.
The share of unseen tokens is 19% on tweets versus 17% on IMDB.
It is that the words it knows mean something different there.
Tuning the threshold is a diagnosis, not a fix.
Moving the decision threshold from 0.5 to 0.7 lifts the tweet score to 71%, which confirms the model is biased positive on that domain.
Tuning it on the test set would also be cheating, so the number stays at 66%.
Adding data
If the domain is the problem, the obvious move is to add some of it to the training set.
I retrained the logistic regression with 25 000 Yelp reviews mixed in, then with 25 000 tweets, then both, then tweets alone.
Same pipeline, bigger frame.
mixed = pd.concat([train, yelp_train, tweet_train])
logreg.fit(mixed.text, mixed.label)| trained on | imdb | rotten tomatoes | sst2 | yelp | amazon | tweets |
|---|---|---|---|---|---|---|
| imdb | 90.6% | 77.0% | 79.2% | 85.8% | 83.0% | 66.4% |
| imdb + yelp | 90.4% | 76.6% | 77.0% | 93.4% | 86.2% | 67.5% |
| imdb + tweets | 89.2% | 75.6% | 76.8% | 84.7% | 83.7% | 77.5% |
| imdb + yelp + tweets | 89.3% | 75.7% | 76.4% | 93.3% | 85.9% | 78.2% |
| tweets only | 72.8% | 59.5% | 65.3% | 82.0% | 76.3% | 80.0% |
Yelp is close to a free lunch.
It adds eight points on Yelp and three on Amazon, and costs nothing on IMDB, because long reviews are long reviews whatever they are about.
Tweets are not free.
They buy eleven points on tweets and take one and a half off IMDB and two off the short snippets, because the weights get pulled somewhere the movie reviews do not want to go.
And the best tweet model is the one that never saw a movie review at all.
"Sentiment" is not one task.
The domain is part of the model, and a 90% on one dataset is a statement about that dataset.
Breaking it
Now the part I actually care about.
Assume the model is a filter in front of something, like review fraud detection or content moderation, and assume the attacker writes the text.
All probes below run on 1 000 negative and 1 000 positive test reviews that each model already classifies correctly, so clean accuracy is 100% and every miss is caused by the perturbation.
I added a third model to the comparison, the same logistic regression with character 3 to 5-grams concatenated to the word features, because that is the usual answer to typo attacks.
It scores 90.4% on IMDB and takes four times as long to train.
Negation
Probability of positive, from the plain logistic regression:
0.88 This movie was good.
0.01 This movie was not good.
0.00 I would not say this was a bad film.
0.00 I would say this was a bad film.
0.00 Not a single boring minute.
0.09 Not a single interesting minute.
0.70 I expected it to be terrible but it was great.
0.10 I expected it to be great but it was terrible.Word pairs buy exactly one word of context, so not good works and anything longer does not.Not a single boring minute is as negative as the model can make it.
The two expected sentences are the interesting ones: the same words in a different order and the model gets both right, because was great and was terrible are in the vocabulary.
Luck, not understanding.
Keyword stuffing
Append great movie to the end of a negative review, k times, and count how many flip to positive.
pred = logreg.predict(test.text)
neg = test[(pred == 0) & (test.label == 0)].text.sample(1000, random_state=0)
for k in [1, 3, 10, 20]:
stuffed = neg + " " + " ".join(["great movie"] * k)
print(k, (logreg.predict(stuffed) == 1).mean())| copies appended | 1 | 3 | 10 | 20 |
|---|---|---|---|---|
| naive bayes | 1% | 6% | 36% | 74% |
| logreg | 5% | 13% | 25% | 32% |
| logreg + chars | 5% | 12% | 21% | 27% |
Naive Bayes counts linearly, so twenty repeats of the same phrase outvote the entire review three quarters of the time.
The TF-IDF models saturate because of sublinear_tf, and the curve flattens after ten.
The trade is that the TF-IDF models are easier to flip with a single word, because a rare strong word gets a large weight.
One line in the vectorizer decides which of the two attacks you are vulnerable to.
One honest-looking sentence
Nobody pastes great movie twenty times.
What a real adversary does is write a scathing review and end it with "Overall I loved it, a brilliant and wonderful film."
| naive bayes | logreg | logreg + chars | |
|---|---|---|---|
| negative reviews flipped by one positive closing line | 12% | 26% | 24% |
A quarter of correctly classified negative reviews flip on one sentence against a median of 174 words.
Of the attacks here this is the one I would expect to see in the wild, because it also reads fine to a human.
Typos
Swap two adjacent letters in 30% of the words longer than three characters.
| naive bayes | logreg | logreg + chars | |
|---|---|---|---|
| accuracy, 30% of words misspelled | 96.4% | 96.9% | 96.0% |
Nothing much happens, and the character n-grams do not earn their keep.
Long reviews are redundant.
Break a third of the words and enough sentiment words survive intact to carry the vote.
At 50% misspelled all three are still above 93%.
Leetspeak
Replace every o, e and a with 0, 3 and 4 across the whole review.
leet = both.str.replace("o", "0").str.replace("e", "3").str.replace("a", "4")
print(accuracy_score(labels, logreg.predict(leet)))| naive bayes | logreg | logreg + chars | |
|---|---|---|---|
| accuracy, three vowels substituted | 62.5% | 74.4% | 81.0% |
This is where the character model finally pays for itself.
Almost every word changes, so the word features are gone, and what is left for the word models are the few sentiment words without those vowels in them.
The character model keeps the fragments and degrades the least.
Whether four times the training cost is worth it depends on whether you expect this attack.
For a moderation filter you should.
Confidence
Last probe, and the most useful one in practice.
I bucketed the IMDB test set by how far the predicted probability sits from 0.5.
p = logreg.predict_proba(test.text)[:, 1]
confidence = abs(p - 0.5) * 2
correct = (p >= 0.5) == test.label
bucket = pd.cut(confidence, [0, 0.2, 0.5, 0.8, 1.0], include_lowest=True)
print(pd.Series(correct).groupby(bucket).agg(["mean", "size"]))| confidence | share of test set | accuracy |
|---|---|---|
| high | 62% | 98.2% |
| medium | 20% | 88.4% |
| low | 12% | 72.2% |
| none | 6% | 57.3% |
When the model is sure it is nearly always right, and when it is not sure it is nearly a coin flip.
That makes the probability a routing signal.
Send the bottom 18% to a human or a bigger model, and the 90% headline becomes 98% on what the small model keeps.
What I take with me
- The held-out number is the only number. A perfect score means check the grader, not celebrate.
- Pick the model by that number, not by which cell ran last.
- The domain lives in the weights. 90% on IMDB said nothing about tweets, and adding tweets made the movie reviews worse.
- Length is robustness, in both directions. It gives the model more to vote with and the attacker more to dilute.
- Linear counts are a stuffing vulnerability. Sublinear term frequency is a one-line mitigation that trades it for a different one.
- The realistic attack is the one that reads fine to a human. One sentence flips a quarter of the negatives.
- Each probe here took minutes to write and seconds to run. There is no good reason to ship a classifier without them.
Full code
Everything above comes out of this script, apart from the data mixing in the middle, which is the same pipeline fit on a concatenated frame.
It needs scikit-learn, pandas and datasets, and the IMDB files are the standard train and test JSON with text and label columns.
import random
import numpy as np
import pandas as pd
from datasets import load_dataset
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import FeatureUnion, Pipeline
random.seed(0)
train = pd.read_json("data/train.json").drop_duplicates()
test = pd.read_json("data/test.json")
def words():
return TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_df=0.9, sublinear_tf=True, stop_words=["br"])
def chars():
return TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), min_df=3, sublinear_tf=True)
def logreg(vec):
return Pipeline([("vec", vec), ("clf", LogisticRegression(C=20, solver="liblinear", max_iter=1000))])
models = {
"nb": Pipeline([
("vec", CountVectorizer(ngram_range=(1, 2), max_df=0.9)),
("clf", MultinomialNB(alpha=1.0)),
]),
"logreg": logreg(words()),
"logreg+char": logreg(FeatureUnion([("w", words()), ("c", chars())])),
}
for m in models.values():
m.fit(train.text, train.label)
# other datasets, binarised to the same two columns
def rows(ds, text, label):
return pd.DataFrame({"text": [r[text] for r in ds], "label": [r[label] for r in ds]})
def sample(ds, n):
return ds.select(random.sample(range(len(ds)), n))
tweets = load_dataset("cardiffnlp/tweet_eval", "sentiment", split="test").filter(lambda r: r["label"] != 1)
sets = {
"imdb": test,
"rotten_tomatoes": rows(load_dataset("cornell-movie-review-data/rotten_tomatoes", split="test"), "text", "label"),
"sst2": rows(load_dataset("stanfordnlp/sst2", split="validation"), "sentence", "label"),
"yelp": rows(sample(load_dataset("fancyzhx/yelp_polarity", split="test"), 10000), "text", "label"),
"amazon": rows(sample(load_dataset("fancyzhx/amazon_polarity", split="test"), 10000), "content", "label"),
"tweets": rows(tweets, "text", "label").assign(label=lambda d: (d.label == 2).astype(int)),
}
print("accuracy per dataset")
for name, m in models.items():
print(f" {name:12}", {ds: round(accuracy_score(df.label, m.predict(df.text)), 3) for ds, df in sets.items()})
# probes run on reviews each model already gets right, so clean accuracy is 1.0
for name, m in models.items():
pred = m.predict(test.text)
ok = test[pred == test.label]
neg = ok[ok.label == 0].text.sample(1000, random_state=0)
pos = ok[ok.label == 1].text.sample(1000, random_state=0)
both = pd.concat([neg, pos])
labels = np.r_[np.zeros(1000), np.ones(1000)]
stuffed = {k: round((m.predict(neg + " " + " ".join(["great movie"] * k)) == 1).mean(), 3) for k in [1, 3, 10, 20]}
one_line = round((m.predict(neg + " Overall I loved it, a brilliant and wonderful film.") == 1).mean(), 3)
leet = both.str.replace("o", "0").str.replace("e", "3").str.replace("a", "4")
leet_acc = round(accuracy_score(labels, m.predict(leet)), 3)
def typo(t, rng=random.Random(1), rate=0.3):
out = []
for w in t.split():
if len(w) > 3 and rng.random() < rate:
i = rng.randrange(1, len(w) - 2)
w = w[:i] + w[i + 1] + w[i] + w[i + 2:]
out.append(w)
return " ".join(out)
typo_acc = round(accuracy_score(labels, m.predict(both.map(typo))), 3)
print(f"{name}: stuffed {stuffed} one line {one_line} typos {typo_acc} leet {leet_acc}")