Utopia Now

    There is an argument for increasing the rate of AI progress. Maybe the probability of other ex-risks are too high, and we simply cannot wait around for another 100 years. If nuclear war was destined to happen within the next ten years, I am certain that we would be pushing as fast as possible towards AGI. In some sense, your drive to be reckless is highly correlated with your pessimism regarding where things are going. If you think humanity is on a great linear trajectory towards utopia, there is no use in throwing in random variables that can mess things up. If AGI has a 10% chance of killing us, and you are fairly certain that in two hundred years the human race will be flourishing, probably not worth developing AGI. If you are pessimistic about humanities prospects, maybe we take the 10% risk.

    The world is full of authoritarian governments that are doing terrible things. Two of the three military superpowers (Russia and China) have horrible human rights track records and have a strong drive towards increasing power and influence. Russia invaded a sovereign country recently, and China is doing very, very bad things (oppression of Uyghurs, Hong Kong takeover, general police state tendencies). The West is constantly on the brink of nuclear war with these countries, which would result in billions of deaths. Chemically engineered pandemics become both more likely and more dangerous over time. The barriers to creating such viruses are being knocked down and the world is becoming more and more interconnected. What are our odds? If our odds of dying off soon are great, or if it will take us a long, long time to reach a place where most humans are free and thriving, maybe we make the trade. Maybe we decide that we understand the risks, and push forward. Maybe we demand utopia, now.

    Well, there is another problem with AI: suffering risk. This is not often discussed, but there is a very real possibility that the development of transformative AI leaves the world in a much, much worse place than before (ex: ASI decides it wants to torture a bunch of physical people or simulate virtual hell for a bunch of digital minds for research purposes). Another factor in your AI hesitancy should your estimated probability of a perpetual dystopia. This is where I differ from other people. I believe that the risk of things going really, really wrong as a result of AI (worse than AI simply killing everyone) is massively understated. We should hold off on AGI as long as possible, until we have a better understanding of the likelihood of this risk.

The Future of Freedom

    The dawn of AGI is near. What this means for the world is uncertain, but if you follow Nick Bostrom's logic it seems clear that ASI will be soon to follow. This will have a more clear result: the human race will no longer be the supreme being on the planet. We talk a lot about utopia when we discuss ASI. We discuss the ways in which it could cure disease, expand lifespans (potentially indefinitely), and colonize the galaxy. We also discuss value lock in, and the possibility for authoritarian dystopias. In every case, we see some version of either utopia or dystopia, all with one thing in common: a single entity making the decisions. Similar to a world government, our eventual ASI will likely control our lives and the bounds in which we live. I rarely see discussion of a libertarian utopia, where each individual receives private property and is allowed to do whatever they want so long as they are not impacting others in a negative way. I am not quite sure how this will work in a post-scarcity society. We are in the age of transformative AI, and I am very worried about human freedom. The right to make the wrong decisions is important, as it is often the only way to discern the right ones.

    Will ASI adhere to a bill of rights? It seems that this list of unalienable rights was crucial in the formation of the United States. Freedom often comes at a price. The second amendment absolutely equates to more individual freedom, at the expense of many needless deaths. Will the ASI respect these types of rights (freedom of speech, right to bear arms), even if in aggregate they could hurt society (hate speech, mass shootings). In the event of a chemically engineered pandemic, will the ASI force vaccinations at gunpoint in order to ensure the survival of the human race? I am very, very worried that the coming age of AI will naturally lead to autocracy. Time and time again we have seen history repeat itself, with "ends justify the means" and "for the greater collective good" leading right into fascism. I worry the technocratic and socialism-inclined minds may win out over the libertarian. Personal political beliefs aside, I think the former will inherently place less value on freedom and will be more likely to through good intentions force a bad outcome.

Mind Crime

    Humans are really, really bad at planning in advance to not be monsters. We have a pretty horrible ethical track record. Genocide and slavery seem to come pretty easily to most of us, given the right time period and circumstances. If there are internalized morals, we sure took our sweet time finding them. Generally, I don't think humans are in a position to make rational, ethical choices involving other conscious beings. Regardless of your take on factory farming, it is pretty clear we didn't spend decades deliberating the ethical issues in advance. Have you fully thought through the moral implications of factory farming, or are you just along for the ride? I am very worried that unaligned superintelligence will kill all of humanity, or enslave us, or torture us, or become authoritarian and lock in terrible values for eternity. Still, I am also worried about mind crime. 

    Look at our track record with slavery. Read about the recent Rwandan genocide. Look at the various authoritarian regimes and staggering human rights abuses across the planet. But don't worry, we will somehow care a lot in advance about the moral rights of artificial intelligences. From the industry that brought you social media, and don't worry they totally thought through and predicted any negative ramifications of the technology and have your best interest at heart, here is the new god! And don't worry we will treat it well and we totally won't be enslaving a morally significant being.

    If we gain the ability to generate millions of digital minds, we gain the capacity for horrors worse than any genocide or slavery in humanity's past. We might not even do it on purpose, but just through sheer ignorance. It took a long time for people to treat other humans as morally significant. And by long time I mean basically until fifty years ago in the U.S., and in many other countries this is still not the case. It isn't crazy to imagine that we will treat "computers" much worse. Mind crime will have to legislated early. If you knew slavery was about to become legal again in twenty years in the U.S., what policies would you put in place? How would you get ahead of the problem and ensure that morally significant beings aren't put in virtual hell? These are the questions we should all be asking.

The World Will End Because Math is Hard

    Every machine learning book I read leaves me baffled. How on earth can anyone understand this stuff? Not at a surface level, but how can anyone really master statistics/probability/calculus/linear algebra/computer science/algorithms to a degree where they actually understand what all the words in these 1,000+ page books mean? Even a summary book such as the "The Hundred-Page Machine Learning Book" leaves me with more questions than answers. Now to learn all of that, and then try to layer on the required decision theory/economics/ethics/philosophy to a level where you can have a positive impact on AI alignment seems pretty unreasonable. A lot of people pick a side, either specializing in cutting edge deep learning frameworks or armchair philosophizing. The AI capability people tend to underestimate the required philosophical complexity of the problem, and the AI ethics people tend to completely misunderstand how current machine learning works. Maybe there are a few that can master all of the above subjects, but it is more likely that a combination of people with deep expertise in disjointed areas will provide better solutions. It is pretty clear that I will not be one of the individuals who invents a new, more efficient learning algorithm or discovers a niche mathematical error in a powerful AI product. Focusing on AI risk management, a massively underdeveloped industry, is probably the way forward for me. The math is simply too hard, maybe for everyone. But someone is writing the books. If we can get a few people who understand the complexity of the issue into the right positions, maybe we can cause some good outcomes.

    

    One of the benefits of focusing on risk management is that you can make money and not feel guilty about it. "Oh no, people working on AI safety are making too much money." Have you heard that before? I for sure haven't, and I would like to. To someone that believes in markets, that statement rings similar to "oh no, people are going to be massively incentivized to have a career in AI safety." What a problem that would be. Also, competition isn't even a bad thing, an arms race towards safer products would be quite interesting. "Oh no, China is catching up and making safer AI systems than the US." I would pay to hear that. Obviously, sometimes alignment is really capabilities in disguise. I have touched on this previously, but deciding what exactly makes systems safer and what makes systems more powerful is pretty hard.

    I briefly pitched Robert Miles a few weeks ago on some of my ideas. Mainly an AI risk management industry that will provide more profitable employment opportunities for alignment researchers. His response:

"I guess one problem is the biggest risk involves the end of humanity, and with it the end of the courts and any need to pay damages etc. So it only incentivizes things which also help with shorter term and smaller risks. But that's also good. I don't have much of a take, to be honest."

    I am a newbie to this field and Robert is the OG (someone who understands the entire stack). His take is entirely fair, as companies will only be incentivized to curb short term risks where they will be affected. The elephant in the room is obviously the end of humanity or worse. People that don't see this as feasible simply need to read "The Doomsday Machine" by Daniel Ellsberg. All this talk of nanotechnology makes us miss the obvious problem that we are a hair's breadth away from worldwide thermonuclear war at every moment. I wonder how things will change when a powerful, unaligned AI starts increasing its hold on such a world. Longtermists drastically undervalue the terror of events that kill 99% of people instead of 100%. In regards to long term AI alignment, I think the number of researchers will matter, and I hope people in the AI safety industry would be incentivized to study long term alignment outside of work hours. Maybe I'm wrong and there's not a strong impact, but I haven't managed to find too many negative impacts of such a pursuit.

Company Thoughts: Part One

    Here is my essential company thesis:

1. There are less than 500 people in the world seriously working on AI alignment

2. This is a serious problem

3. We need to fix it

    Now let's pretend you are a financial professional and lack a detailed machine learning background. Well, you could drop your career capital, pursue a machine learning PhD, afterwards work at OpenAI or Anthropic, and then after a few years there (the year is now 2031) you decide to get some people together to start an AI safety company. Or, you save eight years of time and just start one now given your current skill set. Given the competitiveness and time requirement of the first option, I don't see any particular value in it. For the second option, I see actual impact potential. Also, there would be a lot of personal value here. As an effective altruist I don't see a large difference between taking six months off to start an AI risk management company and taking six months off to volunteer in Africa. If AI alignment is as important as I think it is, there's really no reason not to do it. So, what to do?

    

    Connecting companies to AI safety experts is probably the easiest. This could incentivize people to join AI safety and alignment as a career, and also maybe curb some short term risks of misaligned narrow AI. I am going to use alignment and safety a bit interchangeably here, as I envision these experts having a day job focuses on safety/risk management and a night job (unrelated to pay) focused on greater alignment issues. Let's expand. If people see that they can have a fulfilling career in AI alignment and actually feed their families and pay their bills, they are more likely to enter the industry. More people in the industry will lead to more beneficial alignment research and more people with the required skill set to navigate the complexities of AGI. Why aren't people entering the industry? First of all, there are basically no jobs (check the OpenAI and Anthropic website and you'll see maybe one safety job out of a hundred). If those two labs only have two job openings for AI safety, I would doubt there are more than ten open seats at AI labs for safety roles in the entire US. Second of all, changing your life to pursue alignment research with your time will make you zero dollars. I have yet to find anyone working in alignment paid an enviable salary. 

    There are a couple of non-profit AI alignment research firms. With traditional nonprofits, the traditional wisdom is people are paid less because they get some sort of emotional validation from doing good work. These people feel compelled to make a sacrifice, and later spend a majority of their time complaining about pay and recruiting for for-profit companies. AI alignment is important, and you get paid zero dollars for doing it. Not only that, but in term of opportunity cost (tech pays after all) you are potentially losing hundreds of thousands of dollars a year. The most important research field in human history, the smallest incentive to enter the field. Yes there a few (and I really mean less than ten) AI alignment jobs, but they are massively competitive (for no reason other than there are literally less than ten jobs). Here is a hypothetical. You are a recent MIT graduate who is an expert in machine learning. You can go work for Facebook and build AI capabilities and make $150,000 at the age of twenty-two. Or if you care about alignment, you could, well, I mean, I guess... you could post on LessWrong and stuff and self-publish research papers? Or try to get a research job at a company like Redwood that needs zero more people? Creating job opportunities would not totally solve this problem. And AI alignment is never going to get the top talent (those people have too much incentive to at least initially make bank building capabilities). I don't think we necessarily need them though (every MIT grad I know is impossible to work with anyway). Providing any sort of alternative, even just a basic nine to five job that pays 60k, may drastically increase the number of people willing to switch over. Closing this gap ($150k vs $0) is probably important. I am advocating for a market driven solution, something desperately needed.

    In this scenario, now there is an industry where machine learning engineers can work a nine to five job focused on AI safety. They build their skill set during this time and spend their outside of work hours doing what they would be otherwise doing (posting on LessWrong and self-publishing research papers). They now have a network of other AI alignment researchers they work closely with. Obviously, people could work on capabilities at work and do alignment in their free time. I would love to move to a world where this is not required. Moving forward, obviously this is good for the AI safety people, and potentially the field of alignment as a whole. What is in it for the companies?

    A lot of companies would find it tremendously valuable to have someone explain AI to them. They have no idea what is going on. Not just "they don't know how neural nets work" (spoiler, no one does). But they actually have no idea how most machine learning is done and they are baffled by large language models. They are worried about putting customers at risk, but they also don't want to get left in the dust by competitors who are using AI. They are banning AI tools but putting the words "AI" in marketing material. Having someone come in and explain the risks involved and how to make these trade offs would be massively beneficial. Most companies in America right now need that sort of consultant. They need it cheap and don't have the funds to go to McKinsey and pay absurd fees. We could provide that. I really do think this sort of industry will be massively in demand going forward. Financial firms without risk management departments are worth less. Companies with bad governance trade at steep discounts. AI is massively beneficial but can lead to terrible outcomes for a company. You should be able to fill in the gaps. 

Music, Movies, and the New Wild West

     In a previous post, "How Important Are Humans," I mentioned an argument I had with a close friend about AI generated art. My conclusion was that if AI ends up writing better books, creating better art, and making better movies, I will have no problem switching over to AI creations completely. Why would I read a 7/10 book when I can read a 10/10 book? At some point, the quality of the content is really all that matters. Well, within two weeks this has pretty much come to fruition. The quality of AI content has exploded, especially within the music landscape. The song "Heart on My Sleeve" by Drake and The Weeknd made waves in the music world, as it is completely AI generated and unrecognizable as AI. All week, I have been listening to AI music pretty much exclusively. I also listed to AI generated stand up comedy and watched some crazy-accurate deepfake videos. There are some cool applications of all of this.

    In the near future, the voices of singers, faces of actors, and writing style of writers will be replicable for free. Before I go for a run, I will be able to create a new Kendrick Lamar album (his voice, his cadence, his songwriting ability) within seconds. During my run, if I don't think his voice fits the song, I can switch the artist to Nas and the transition will be seamless. If I am watching a movie and don't like a particular actor, I will be able to quickly toggle the movie so that Danny DeVito is now playing that role. What will this all mean? Well, obviously we will probably have a lot of pressing legal issues to figure out. I am guessing this will regress a bit in spirit back to the days where everyone paid for music, and thus everyone illegally downloaded music for free on LimeWire. There will be a massive black market for AI generated songs and movies that steal the image and likeness of people without their consent. The most popular singers and actors will become more popular as they are featured heavily in this content, while those entering the industry will have essentially zero value. In a world when the most loved actor in the world can play a role in every single major film of the year, we don't need more actors. With no scheduling conflicts and no actual work required, I would guess that the traditional acting and music industries are essentially going to die. Live performances will still have a niche, but there will also be AI created characters and singers that will start taking some of the spotlight. Characters that are the perfect representation of an idea or personality, without any of the baggage or time requirements that plague real-world humans.

    Think back to the Wild West for a second. You could shoot someone in a bar, drive three towns over, and as long as no one saw you commit the actual murder it was essentially impossible to prove. A serial killer in the 1800's was essentially unstoppable, as there was no DNA evidence, and again, without any direct witnesses there would be no conviction. Even then, if there was a witness how the heck would any authority reliably track you down? If you want to imagine this world I would recommend reading "The Devil in the White City." We may be backtracking to this stage of life. Video and audio evidence in a world of indistinguishable deepfakes is basically worthless. I know no way of determining if a top-level deepfake is real or not, and given that a video is just a sequence of pixels there will probably no way to actually distinguish true reality. As a result, eyewitness testimony, as flawed as it is, will probably regress to being the primary form of evidence. If we can reliably trick cameras in indistinguishable ways, this means that a surveillance state driven by video and audio monitoring is less useful. Unfortunately, there are likely biometric equivalents that an authoritarian state will think up (you are now tagged with an imbedded GPS since we can't trust our cameras).

    Overall, I don't think that these new developments makes society any safer or more stable. There are now incredibly convincing disinformation tools, and I really don't know how I will trust anything I read or see going forward. Still, listening to young Taylor Swift sing her new album was cool. And some of the AI content is legitimately hilarious. If the world burns, at least we will all be laughing. Nothing makes an apocalypse more palpable than good content. 

Time to Start a Company

     Alright, well I thought about it and autonomous agents are insane. It is pretty obvious that within a decade pretty much every single company in the United States will be using AI agents for various tasks. As I mentioned before, finance companies have risk departments that prevent individual firm collapses and industry-wide financial contagion. The fact that current companies don't have AI risk management departments is not surprising, but soon it will seem ludicrous. Within a decade, every company in the US will be using multiple AI agents. They will have to, less they lose out to competitors who are employing this transformative technology. Again, the incentives are simply much too high. The AI market will be saturated with competitors trying to make the next ChatGPT, but none will be focused on the most important part of it all: risk. Providing risk solutions rather than capability solutions is an untapped area of the market. If you run an autonomous agent, horrible things could happen. Customer data could be leaked, the AI could break various laws, or you could accidently make a lot of paperclips. Companies are terrified of risk, terrified that all of their hard work and credibility will be wiped away. And it will happen, it will happen to a few companies and it will be well publicized. But companies won't stop, because they can't. They are driven to survive and make profits, and they will underestimate the risk (as does every investment firm, and they have risk departments!).

     Insert AIS, a company that delivers risk mitigation tools and access to AI experts. Customized software platforms that estimate risk and pose solutions, or some other product I haven't thought of. Probably the easiest solution is to outsource AI researchers as consultants who look over a company's plans and provide feedback. I would not target the business of the massive players who already have AI safety groups, are rapidly building capabilities, and are aligned with gargantuan profit-driven tech giants (OpenAI, DeepMind, Anthropic). Rather, AIS would service the 99.9% of other companies in the world that are going to dive in, safety or not. 

    There is a moral hazard here. You don't want to "rubber stamp" companies and give them a false sense of security. You don't want to convince companies that otherwise would have sat out on AI to participate, because they will gladly place the blame on you and justify their uninformed decisions with your "blessing" as a backing. So this will not be an auditing firm, verifying any sort of company legal compliance or justifying behavior. Those should all be internal. Rather, it will be providing systems and knowledge to build safer and more collaborate AI. Again, these are the small fries. I am less concerned about a mid-tier publishing company building the paperclip machine, and I am convinced that they are less likely to do so if they have a risk management system. 

    The most remarkable aspect of this idea is that even if someone else adopts it and creates a superior risk solution, it is a win-win scenario. Increased competition fosters innovation, and being the first mover in this space could ignite the creation of an entire industry. An industry that I am convinced will probably make things better, or at least not make things worse. If I am instantly replaced by a more capable CEO or another company develops awesome alignment solutions, all the better for humanity. I'll gladly return to an easy lifestyle with no skin in humanity's game.

    Another remark. The Long Term Future fund (the largest fund which funds initiatives that combat ex-risk) is only $12 million dollars. That is ridiculously small. In the world of finance, that is a rounding error to $0. There are only a few hundred AI alignment researchers, and they are definitely not paid well. At this point, AI alignment is similar to other non-profit work: you are expected to make a massive financial sacrifice. Working on capabilities research will feed your family, working on AI alignment will not. As a result, there is really no incentive to go into the most important research field of all time. This needs to change. I think creating AIS will kick off a market-driven solution to this problem. People that become experts in interpretability and corrigibility and come up with novel alignment solutions will have massive value. I would pay them handsomely to work with risk mitigation for various companies, and as a result we will incentivize more individuals to enter the space. If they work forty hours a week and make a decent salary, they can spend the entirety of their time outside work contributing to the long-term value alignment cause. I don't see many downsides here, outside of the massive personal and career risk I would accumulate as a result. Well, seems at least interesting though. Would be a pretty noble way to end up on the streets. "Hey man can you spare a dollar, I blew all my savings trying to align transformative AI with human values." Would at least make for a cool story. Guess its time to start a company.