You Look Like A Thing And I Love You

Review

If you work in tech and treat AI as a black box you haven't figured out how to open, this book is a good place to start. Janelle Shane explains how machine learning models actually fail before she explains how they succeed, which is the right order. The bias chapter is genuinely better than most: she names the problem, shows it with real examples, and doesn't pretend it has clean solutions. The writing is light, the examples are memorable, and the overall arc leaves you more skeptical of AI claims than you started, which is exactly the outcome worth paying for. If you already have a solid working model of how training data and optimization work, you will find the pacing slow and the technical depth thin. But if you do not, this is one of the more honest introductions I have read.

Key Takeaways

The parts worth keeping

The five principles of AI weirdness

1. The danger of AI isn't that it's too smart, it's that it's not smart enough
2. AI has roughly worm-level cognitive capacity
3. AI doesn't understand the problem you want it to solve
4. AI will do exactly what you tell it, not what you mean
5. AI will take the path of least resistance

Where AI works and where it breaks

AI tends to work well when:

• you don't know the rules in advance
• there are many subtle rules to learn
• the task is narrow and well-defined

It tends to fail when:

• the problem is too broad
• training data doesn't represent the real world
• there are sneaky shortcuts available (AI will find them)
• the model was trained for a simpler task than the one it actually needs to solve

Before reaching for ML, ask: is this actually the simplest way to solve the problem? If common sense or simple rules work, use those instead.

How ML models actually fail

• Healthcare researchers found their cancer detection algorithm had learned to spot rulers in photos rather than tumors. Many tumor images in the training set had been photographed next to rulers for scale, so the model learned the wrong signal entirely.
• Training AI resembles teaching a child more than programming a computer. You hand it data and let it figure out how to copy the pattern.
• AI models are slow learners and need large volumes of examples. Simulation is used to train them faster than real-time when real-world examples are expensive or dangerous to collect.
• AI is bad at long-range memory and forward planning. Text generation gets harder the longer the sequence, because the model needs to track what came before and plan what comes next simultaneously.
• Class imbalance: if the thing you're looking for is rare (fraud), an algorithm can hit high accuracy simply by predicting the majority class (not fraud) every time.

Three algorithm types worth knowing

• Markov Chains tackle similar jobs to recurrent neural networks, such as predicting the next word in a sequence. They are lighter and faster to train but can't predict far into the future.
• Random Forests are built from decision trees: individual flow charts leading to an outcome. ML builds a forest of these trees using trial and error. Each tree learns from a different slice of the data and casts a vote. The votes pool into a prediction that's more reliable than any single tree.
• Evolutionary algorithms treat each potential solution as an organism. Each generation, the most successful solutions survive, reproduce, and mutate into new candidates.

Combining algorithms makes sense because they perform better in narrow domains. Deciding how to decompose your problem into subtasks for different algorithms is a key skill in applied ML.

Interpretability: what is the model actually looking at?

• Two approaches: look at which cells activate when the model sees a particular input, or tweak the input systematically and observe which changes produce the strongest response.
• OpenAI trained a neural network on Amazon review data to predict the next letter in a sequence. One cell had learned to track review sentiment, finding it a useful predictor of what letter comes next.
• Google found one of their ImageNet recognition models was distinguishing dogs from cats partly by ear shape (floppy vs. pointy).

Bias and data quality

• If data comes from humans, it contains human biases. That's the starting assumption, not the exception.
• Movie review data encodes the review-bomb phenomenon: movies with non-white or non-male casts receive organized negative reviews. A model trained on that data learns the bias. You can adjust the underlying word vectors to break those associations, but it's an imperfect fix.
• MathWashing (also called Bias Laundering): explaining away bias because a computer made the decision, not a person. The bias is still there; the computer just makes it less visible.
• Unintentional Memorization: when a model memorizes specific data points from training, including PII that was never intended to surface to users.
• Adversarial example: adding "Oxford" or "Cambridge" in invisible white text to a CV can fool keyword-filter algorithms into scoring it higher.

Getting more training data: crowdsourcing, Mechanical Turk, and data augmentation are the standard approaches.

The core tension

As AI becomes ever more capable, it will still not know what we want. It will still try to do what we want. But there will always be a potential disconnect between what we want AI to do and what we tell it to do.

It helps to imagine the model is deliberately misinterpreting your reward function. That framing surfaces gaps you would otherwise miss when deciding what to optimize for.

Working with AI well requires:

• understanding how it fails, not just how it succeeds
• choosing problems that are a good fit for ML
• anticipating how it will misunderstand the goal
• actively preventing it from copying the worst patterns in the training data

Questions to ask when evaluating AI claims

1. How broad is the problem?
2. Where did the training data come from?
3. Does the problem require significant memory or a long planning horizon?
4. Is it copying human biases from the training data?