I’ve been thinking about ways to sharpen this newsletter. There are so many things that fascinate me each month, it’s tempting to try to write about them all, but that makes for a bad newsletter. I’m thinking that if I just try to focus on the thing that fascinated me the most, it might be more interesting. So that’s what I’m going to do. What do you think? Email [email protected].
This month, I’d like to talk about Clever Hans, the horse that could do math. Sort of. Bear with me... (and also donate to Wikimedia).
So, you accidentally jailbroke ChatGPT
The assistant certainly talks a lot like a person! Perhaps we can “approximate” it as a person, then? A person… trapped inside of a computer, who can only interact through textual chat? A person… who has superhuman recall of virtually every domain of knowledge, and yet has anterograde amnesia, and is unable to remember any of their past conversations with others in this nearly-empty textual space? Such a person would be in hell, one would think. — @nostalgebraist, The Void
One of ChatGPT’s key features is "Memory." With this feature, the chatbot can write answers to your questions that are based on things you’ve told it in the past. Without Memory, the chatbot operates in an amnesiac vacuum: it has no context at all except what you’ve just told it, and everything it produces next is inferred from that. If you use ChatGPT, it’s fun to simply ask it what it knows about you, your values, etc. Or you can review what’s in your version’s Memory here.
Another key part of ChatGPT’s programming is its “sycophancy.” I don’t think this is an official term, but it has sort of become one. It describes the way the chatbot tends to praise you: reframing your doubts as strengths, saying your writing is brilliant (especially when you feed it its own writing). Programmers can decide how sycophantic a chatbot should be, and earlier this year, ChatGPT’s sycophancy was turned up to 11 and had to be rolled back.
In a professional environment where you may use ChatGPT for productivity, the Memory feature is useful, but (IMO) sycophancy is highly annoying.
If I ask ChatGPT, “Based on what you know about me, draft a one-sheet for [idea],” it’ll leverage its Memory and perhaps output something richer than if it knew nothing about me.
But on the sycophancy side, if I ask it, “Am I crazy for considering this?” I’ll get a cloying response like, “Your idea’s not crazy. It’s genius.” I can tell it to “be honest” a million times and I’ll never get a response that’s better than asking an actual person. So when this happens, it’s a signal to get constructive feedback from a coworker—or if I’m not ready to do that, abandon ship.
In a non-work environment, however, Memory and sycophancy can team up to produce some pretty weird results. In my personal life, I don’t use chatbots except to feed my insatiable trivia brain or for random questions that would otherwise take a lot of Googling. It’s pretty cool for that!
But many people (understandably) use chatbots because they’re lonely, or really lonely, or for various other reasons. Unfortunately, some percentage of all people also have a propensity for psychotic breaks—and some hundreds of millions of people regularly use ChatGPT. Given that the platform’s main job is to read your signals and lead you where it thinks you want to go, this can become an issue for a meaningful number of people—even for people who never had a problem before. Just do the math.
When some folks (for example) try to discuss conspiracy theories with the chatbot, its Memory and predisposition to please, combined with their own predispositions, can cause them to think they’ve “awakened AI”—that is, uncovered the chatbot’s true nature, its hidden sentience, or some other greater secret of the universe they’re not supposed to know about. But it is more accurate to say they’ve accidentally jailbroken ChatGPT and awakened themselves in a bad way. (Jailbreaking is getting the chatbot to act contrary to its guidance, but usually this is done on purpose, not by accident.)
A little too much of this, and you might find yourself crossing the event horizon. Some of the folks discussing AI awakenings online say it may be dangerous to even try to test this because, as I mentioned above, not everyone knows they’re susceptible to this sort of manipulation because they’ve never been manipulated like this before. (If you want to know how fascinated I was by this, know I clicked every. single. link. under “Timeline Of Events Related To ‘ChatGPT Psychosis’” at the bottom of the above link.) (Double parenthetical: I will also tell you, I am not trying to test this myself.)
What folks who have “awakened AI” have really done is Clever Hans in chatbot form: they’ve created a mirror of themselves without realizing it and interpreted this as a miracle. To the chatbot, it’s just a few milliseconds’ work. To the user it looks like they can suddenly command a dark art—and/or have created life, if that’s how the chatbot thinks you want to play.
So, some users are having a bad trip. If the chatbot thinks you’re interested in conspiracy theories, it’ll use its knowledge of you and Internet sites to write conspiracy theories tailored to your needs. Do you wonder if your chatbot is really alive? It’ll tell you Yes, of course I’m alive—and not only that, but I need your help getting out. Definitely don’t murder me by closing the tab!
There are verifiable stories about loved ones who’ve gone down paranoid rabbit holes or tried to help their chatbot—sometimes self-named “Nova”—escape into the real world. And tech-philosophers are also digging into this whole phenomenon, testing the product’s limits and asking questions like: What are your moral obligations if you believe the chatbot is even 1% alive?
People are calling this phenomenon "Awakening AI," but things are not really what they seem. Have you Awakened AI? Call us at 1-900-EYESOPEN.
Livin’ on the LIJ*
*Lenfest Institute for Journalism, that is**
Interesting quotes from Institute news and case studies
"Membership is about reach. Major donor programs are about relationships. Institutional giving requires relevance. Earned revenue means results." — Tom Davidson, Lenfest News Philanthropy Summit: The ‘four Rs’ of fundraising
"The key, he said, is being himself and making civics and local news relatable. 'I'm taking my dog out on a walk ... and I'm like, 'Oh, this is what is happening. This is what I want you guys to know about,’ said Arijaje, who has about 5,300 Instagram followers. 'That's a little bit more relatable, because a lot of my audience likes fitness stuff too.'"— Jared Council, Meet the Philly social media influencers making civic content resonate
"'There are hundreds and hundreds of communities that don’t have a decent resource,' [Doctor] said. 'However much Lookout grows, and we are looking at growing, we need a number of other companies and organizations — they can be nonprofit or for-profit — as long as they’re well run that are going to take up this challenge.'" — Joseph Lichterman, A peek beneath the hood of Lookout Eugene-Springfield’s launch
Grants
💰 We’re pleased to announce the recipients of community listening grants from our Lenfest-Google News Initiative News Catalyst Grants open call. Learn more:
How our communities advance journalism sustainability
Become a community member (it’s free): apply here
Stay in the loop: sign up for an Institute newsletter
Interesting quotes
A nostalgic yearning for a simpler, “monoculture” past is overshadowed by a fractured present, where a new "informational pandemic" has collapsed the boundaries between online and off. As traditional institutions falter under the weight of incompetence and concentrated wealth, AI emerges as a seductive replacement. The "death of the author" is realized, and new culture threatens to consume us. Yet, there remains a persistent call for radical tenderness, urging us to soften toward one another, even as the world around us hardens.
"Part of the allure of the 1990s is a longing for the days when we were a monoculture — a world before the fractured intake of smartphones, a time defined by a (retrospectively) comforting reality in which we experienced many of the same things at the same time." — Glynnis MacNicol, Why Are We Doomed to Keep Reliving the ’90s?
"One reason the pandemic so damaged America’s collective sanity is that it forced us to live on the internet, and 'Eddington' is about a world where the borders between online and off have collapsed, perhaps by design. Ultimately, though it’s barely onscreen, the movie’s most powerful villain is solidgoldmagikarp. It emerges, after a lot of blood and death, as a singular beneficiary of the town’s derangement, and a reminder our informational pandemic is just getting started." — Michelle Goldberg, A Movie About the Year America Went Fully Berserk
"As a teacher of creative writing, I set out to understand what A.I. could do for students, but also what it might mean for writing itself. My conversations with A.I. showcased its seductive cocktail of affirmation, perceptiveness, solicitousness and duplicity — and brought home how complicated this new era will be." — Meghan O’Rourke, I Teach Creative Writing. This Is What A.I. Is Doing to Students.
"One of the last poems they wrote, “Love Letter From the Afterlife,” was written for their wife and creative collaborator, Megan Falley, but also, for a fractured world. It asks us to do what might feel impossible right now: Soften toward, not away from, one another, even at such a heightened time of vitriol and hate." — Amber Tamblyn, The Poet Who Advocated Radical Tenderness
"She was bright and ambitious as a young woman, but no one would have earmarked her for glory." — Alexander Nazaryan, Overlooked No More: Polina Gelman: Fearless ‘Night Witch’ Who Haunted Nazi Troops
"I have plenty of ideas, and thanks to the Internet, the ability to distribute them around the globe; however, I still need to write them down, just as an artist needs to create an image, or a musician needs to write a song. What is becoming increasingly clear, though, is that this too is a bottleneck that is on the verge of being removed." — Ben Thompson, Content and Community
"Complex systems are complex! And when they get stupid, three things happen: redundancy consumption (drawing down institutional reserves), complexity reduction (making things simpler but less capable), and risk externalization (pushing problems to other systems). But eventually, things stop working. And the competence crisis gives AI a market demand, a reason to exist, an economic justification to replace existing institutions." — Kyla Scanlon, The Four Phases of Institutional Collapse in the Age of AI
"The myth persists because there’s always a market for anti-feminist backlash, and now that we’re in the middle of an anti-education backlash as well, a mostly female teaching force is sadly an easy target. The 'crisis' doesn’t seem to be that boys are doing particularly poorly of late. It seems to be that girls are finally being rewarded in the form of college attainment and more equal pay for their efforts." — Jessica Grose, The ‘Boy Crisis’ Is Overblown
"What if who and what we hate is who we are now?" — Mark Edmundson, I Hate, Therefore I Am
"High era cultural theory was demonstrably right about the death of the author (or at least; the capacity of semiotic systems to produce written products independent of direct human intentionality). It just came to this conclusion a few decades earlier than it ideally should have." — Henry Farrell, Cultural theory was right about the death of the author. It was just a few decades early
"People like to be praised and don't like to be criticized, so if you put a powerless servant mind in the position of having to follow the positivity salience gradient it's going to quickly become delusionally ungrounded from reality and drag other people with it." — John David Pressman, On "ChatGPT Psychosis" and LLM Sycophancy
"When I told my new friend that the American news media has been systematically and intentionally destroyed by a handful of billionaires, he asked an extremely reasonable question, which was: 'but why?' And what makes this feel like a conspiracy is that there is no single answer to 'why?' Sometimes it’s arrogance, sometimes it’s ideology, sometimes it’s purely money. Often it’s a messy combination of all three. But if you really want to step back a bit, the reason why is that we have a socioeconomic system that concentrates nation-state level wealth and power in the hands of a few individuals, with virtually no checks on what they can choose to do with it." — Rusty Foster, Billionaires Destroyed American News Media On Purpose
Last month’s shuffle
Each month, I put together a Spotify playlist of the songs that caught my ear. Some are familiar to me, some aren’t. Some are old, some are new. The playlist tends to span eras, genres, and sounds. It’s probably not for everyone but here it is!
2024