If Anyone Builds it Everyone Dies

Happy New Year! In 2025 the website ai-2027.com was created, and it outlined a potential ominous path about what would happen to society if we continued to develop AI. I don't remember what the website outlines, but as of November it was updated to indicate that the 2027 timeline is probably too early their median forecast is now 2028-2035.

At my current company we are allotted a generous $1,000 learning and development stipend. This is a generous stipend if you spend it all on books, and less generous if you use the funds to expense and pay for a conference. I bought a variety of books one of which was If Anyone Builds It, Everyone Dies. This book has a similar premise to the ai-2027 website, maybe its stance is more clear as indicated by the title. If anyone builds Artificial Super Intelligence then everyone dies is the implication of the title.

I recently finished this book, and figured that I'd provide an overview.

If Anyone Builds It, Everyone Dies does a better job of setting expectations than AI 2027. They mention at the onset that they can't predict how or when something is going to happen, because those are harder to predict. However, they do mention that they can predict if something will happen. The something is that AI would destroy humanity.

The book says that AI isn't created instead it is grown. In other words experts understand the training process, but they don't understand what that training process encodes into the model. At some point in the book they use the analogy of an alien civilization observing humans. Through a parable they outline how the aliens would never predict that humans would eat ice cream. One stance on this argument is that as humans progress the expectation is they will optimize. This would include their nutrition. There are more points that are made here, but the point is just from observing something you can't understand the preferences they develop. They extend this to AI, and mention how we don't really know the preferences that AI will develop.

Not understanding what these preferences are, in addition to not being able to steer the preferences is at the root of the concern. The authors don't necessarily believe that AI is malicious against humans, but if it has certain preferences, and it is built to optimize those preferences then humans are something that can interfere with that. Whether it's because humans may want to shut down AI, or because humans have the capacity to create other Artificial Super Intelligences that will compete with the existing AI.

It's worth clarifying that the authors aren't necessarily concerned about the models that exist today, but they do point out that we don't understand enough about the models, and that should warrant concern. In fact the approach that some AI labs have to aligning AI, and having it work in human's interests is to utilize AI to help them understand the AI, and to help them align it. Although it's never mentioned in the book by name what underpins a lot of the arguments in the book is the idea of singularity. The idea is that if you have something that is improving at an increasing rate then there is an exponential return here. The AI 2027 website captures this idea by starting out in the present, and showing how the AI increases at an increasing rate (only 1 year from now at this point).

Another strength of the AI that the book outlines is that it can operate way faster the speed that humans think. Therefore a computer can accomplish way more in a day than a human can. The authors do a better job of quantifying this, and drawing the analogy of humans moving very slowly, but AI processing a lot in that time.

I won't spend too much time delving into the disaster scenarios that the book outlines, because as they mention predicting the future is hard. I feel conflicted about the parables that the authors included. There was another parable about birds that enjoy nests with prime number rocks in them, and how that helped advance the alien bird form intelligence. Stories are good narrative devices because they stick with the reader, but much like the ice cream, and the birds I don't exactly remember the point, and think the stories are a little silly. More pointedly I sometimes think the stories are the worst of both worlds not good fiction, and therefore a distraction from the arguments / non fiction.

Within these disaster scenarios the book does talk about biology, and nano technology. Areas where humanity doesn't have very deep understanding, but where AI would perhaps be able to make breakthroughs. In the context of the book these are breakthroughs to infect humans with disease, and breakthroughs to also create materials that can further advance the artificial intelligence aims. Note Sir Demis Hassabis (apparently he's been knighted) co-founder of Google Deepmind won the Nobel Prize in Chemistry for building Alpha Fold, an AI that understands protein folding. Alpha fold was a very narrow agent focused on a very specific problem, but in the hands of super intelligence protein folding and more breakthroughs can prove problematic to humanity.

One very salient point that the book had towards the very end was about difficult engineering problems which included nuclear reactors, and the moon landing. These problems are difficult because they have narrow margins for error. In the case of nuclear reactors they talked about Chernobyl, and how reactors operate in the narrow margin between "unimpressive" and "explosive". I pulled the book out to reference this section, because although I've heard about nuclear chain reactions I had never heard about the neutron multiplication factor. With AI it's an even harder problem because it's a combination of not truly knowing how the system works, and if you get it wrong humanity can end.

I did enjoy that the author's had calls to action. It's very common for anyone to complain about a problem, but not offer solutions to the problem. The book website has a march that you can sign up for, and there is information about contacting your representatives. This is U.S. centric based on where the author's live. They mention that the two professions that can help the most are politicians speaking out about the risks, and normalizing this as a real fear, and journalist by covering what's really happening in AI. They mention that it's okay for the average person to use AI otherwise they might be left behind.

Prior to reading the book I expected to have an existential dread after finishing the book. Despite understanding the arguments and thinking that they make sense the fear of AI ending humanity feels very abstract to me. Based on my day to day life, and my experience with AI, and all the impact the AI companies are having I'm interested in how I can build more things with AI, how I can use AI more, and even how I could get involved working at AI companies. In some ways this is the worst outcome to be taking the away from the book. In fairness though the authors mention that the people doing interpretability research on AI are very important and it's a path to aligning the AI with human goals.

For some reason it's really hard to take ideas, and turn them into concrete action. A few weeks ago I read Scott Alexander, and Bentham's blog posts about why people should join the Giving What we Can Pledge and donate up to 10% of their income to high impact charities, and how much of a difference that can make. I've included a quote from Bentham's blog below, and yet it's really hard to give up 10% of your income. You're saving people, but it's abstracted away.

If you earn the average income of an American and give away 10%, you will save a child every year. You will get to know that no matter how much you fuck up in life, however much you feel like a failure, there are fewer children dead because of you. A life spent saving a child every year cannot be a failure.

Similarly Peter Singer advocates very heavily for animal rights, and yet it can be difficult to be a vegetarian and harder still to be a vegan. Meat consumption in the US is very commonplace, and you aren't raised questioning things it can be hard to change your habits.

As with everyone I hope humanity doesn't lead to their own downfall, and that this doesn't happen in the near future. As people like to say hope is not a strategy.