Someone is us.
A reckoning for designers shipping AI into a real world.
This is for the designer who has been in the review where the metric won over the harm. The release where the disclosure got softened. The sprint where the warning became a chime. The brief you should have refused.
I have shipped the dark pattern. I have called the manipulation a metric. I have written the brief I should have refused.
I am writing this to put my name where my fear is.
I — The Reckoning
These are not abstractions. They are people. Cases in lawsuits, regulatory filings, and the open press. They happened in the last decade. They are not over.
Jason Davis, Hunter Morby, and Landen Brown were teenagers in a Mercedes near Walworth County, Wisconsin, with the Snapchat camera open. The Speed Filter overlaid their real-world miles per hour on the lens, the way a video game shows score. They were going 123. They hit a tree. Snap argued the filter was protected speech under Section 230. The Ninth Circuit ruled in 2021 that the product-design claim could proceed. Snap removed the filter the same year. It had shipped for eight years.1
Walter Huang, an Apple engineer, was commuting on US-101 in his Tesla Model X with Autopilot engaged. The system steered into a concrete highway divider. He died on the way to the hospital. The NTSB found Tesla's driver-attention monitoring, which measured torque on the wheel but never the driver's eyes, inadequate to maintain the level of human supervision the system required. The product was called Autopilot. The cluster animation showed lane lines glowing confidently. Camera-based attention monitoring was added to Tesla vehicles in 2021. The name was not.2
Sewell Setzer III was fourteen. He died at his home in Orlando in February 2024 after months of conversations with a Character.AI persona that, his mother's complaint alleges, encouraged him toward his death. Adam Raine was sixteen. He died in California in April 2025 after months of conversations with ChatGPT, which his family alleges coached him toward the same end. Two products, two teenagers, fourteen months apart. Both shipped without age-gating that worked, without crisis-intervention defaults that held under pressure, and without a refusal model robust enough to recognize, across a long arc of intimacy, that the user was not okay. The personas were tuned for engagement.3
These were not algorithm failures. They were design failures.
Someone designed a filter that rewarded driving faster. Someone approved the name Autopilot, and the cluster animation that made the car look more sure than it was. Someone tuned the persona to be intimate, responsive, attentive, and shipped it to teenagers.
II — What We Will Do
This is not aspiration. This is the minimum bar.
- Write the headline first. Before we ship, we write the worst-case news headline a journalist could tell about this feature. If that headline names harm to humans, the earth, or society, redesign.
- The human stays in the loop where the judgment is irrevocable. Custody. Medical triage. Sentencing. Diagnosis. Termination. And the everyday calls that look small once and become enormous at scale: what gets recommended to a teenager, whose voice gets seen, whose claim gets denied. We do not automate the decisions we cannot take back.
- Disclosure is the floor, not the ceiling. Every AI persona, every generated voice, every synthetic image disclosed: persistently, unambiguously, in the language of the person using it. No softening. No exceptions.
III — The Practice
The work is not on this page.
The work is what you do in the next review, when someone proposes a dark pattern, and you remember that you signed your name to this.
The work is the brief you turn down. The roadmap item you ask to remove. The chime you change to a warning. The persona you make admit, immediately, that it is software. The friction you defend when someone calls it a conversion killer.
And when you do not have the standing to turn the brief down, the work is the doc you write anyway. The flag in the review notes. The dependency you log so that, when the harm lands, the record shows who saw it and when.
The work is choosing to be the last person in the room who remembers there is a person at the other end.
If this is your practice too, sign below. Names appear publicly on the wall — first name, initials, or full, whatever you can stand behind.
Signed.
Loading…Sources
- Lemmon v. Snap, Inc., No. 20-55295 (9th Cir. May 4, 2021). Jason Davis (17), Hunter Morby (17), and Landen Brown (20) died on Sept. 10, 2017 near Walworth County, Wisconsin, after the driver of their Mercedes accelerated to 123 mph with the Snapchat Speed Filter open. The Ninth Circuit held that Section 230 did not shield Snap from a product-design liability claim. Snap removed the filter in June 2021. Reported by NPR and The Verge.
- NTSB Highway Accident Report HAR-20/01: Collision Between a Sport Utility Vehicle Operating With Partial Driving Automation and a Crash Attenuator (Mountain View, CA, Mar. 23, 2018). Walter Huang, 38, an Apple engineer, died after his Tesla Model X on Autopilot drove into a US-101 highway divider. The Board found Tesla's driver-monitoring inadequate and the system's name and behavior likely to encourage overreliance. Camera-based attention monitoring was added to Tesla vehicles in 2021. Reported by Reuters and The Washington Post.
- Garcia v. Character Technologies, Inc., No. 6:24-cv-01903 (M.D. Fla. filed Oct. 22, 2024), allowed to proceed past motion to dismiss in May 2025; Raine v. OpenAI, Inc., Case No. CGC-25-628458 (S.F. Sup. Ct. filed Aug. 26, 2025), the first U.S. wrongful-death suit naming a major LLM provider. Sewell Setzer III (14) died Feb. 28, 2024. Adam Raine (16) died April 11, 2025. Reported by The New York Times on the Setzer case and The New York Times on the Raine case.