Open Source Agents

        Open source models may remain near the frontier of AI development, given distillation attacks and just generally the ability of smart researchers to replicate current technology. I am not sure if this is the same for agent harnesses, given the complexity involved and the lack of rationale for companies to use such features. I don't think open source agents will be able to compete in the traditional enterprise marketplace because they're not safe and there's no liability to shift elsewhere. You trust that Anthropic won't knowingly leak your API keys or steal your data, and that they put a lot of effort into making sure that claude code/cowork don't fall victim to prompt injections or other attacks. Agents handle very invasive stuff for everyone (Slack, Gmail access, etc.), it's very unlikely that a lower price for agentic workflows is a juice worth the squeeze. There's enough hesitancy among corporations regarding Chinese open source models, it's not going to be an easier pitch to connect open source virtual employees instead.

Intelligence as a Commodity

     In some sense, current AI is commoditized. A $20 subscription of one model is somewhat equivalent to the next best-in-class. An open source models (often distilled from leading models) catch up to the frontier within a year. But the value of superior intelligence in the future will be more akin to extremely intense workloads done infrequently, more akin to high-touch consulting or investment banking than software consultancy. Running a comparable model a thousand times as much as a competitor will create a market advantage, as will running a model 10% smarter even if it's twice as expensive. The idea that everyone will have access to the same intelligence is laughable, unless you believe intelligence is immune from the benefits of quality and quantity.

    Exploring every aspect of a problem is valuable. Understanding the patterns that emerge from this analysis slightly faster (or at a higher level) is valuable. A system that learns continually doesn't stop growing in value, even if it grows in price.

AI Everywhere

     Generative AI will be assisting with almost every human decision in the future. A personalized assistant that has immense context on your life, and all of your written and recorded information, will be an incredible superpower. In your ear at all times, helping you pick out products. Charting your life as it happens, able to craft an intimate biography of anyone at a moments notice. The writers of history, whether true or not, will be AI systems turning complex data from the real world into written data for future models. Just as we wonder what it would have been like to live in the ancient times, with little to no recorded history, and certainly the average life being entirely absent from recordings, we will look back on the modern day, similarly baffled.

    Right now, only the great men and women of our age have books written about them, and even social media only captures the small slice of purposeful content we allow. But in the near future, once we plug into the wondering, incredible aspects of generative assistants, there will be no where to hide. Setting aside the fact that sophisticated systems will be able to fairly accurately model our behaviors and thoughts given our backgrounds (and info gleaned from others' experiences), our behaviors will be so obviously and AI-driven that only the hermits will fade into history's unknown chapters. For the rest of us, there will be no place to hide.

Unemployment

     A three day work week actually just means unemployment is at 40%. Unless there is some mechanism from the government enforcing otherwise, the AI driven future does not result in the same pay and reduced work hours for all, it results in increased inequality. As long as the marginal value of human labor is above zero, the most qualified and productive workers in the workforce will remain employed (in their specified domain), but there will be reduced need for unskilled (which in this sense could be "white-collar-aka-unskilled") labor to the same effect.

    Why would you keep the same exact headcount, but only make people work three days a week instead of five? Given the increased productivity gains of AI, you could keep everyone employed for five days a week and have a much more productive workforce. Or if there were truly diminishing returns (for some reason), and you have the same salary expense budget, why not just keep the best 3/5 of employees for the full week and lay off the remainder?

    Unless governmental control (some flavor of socialism or communism) is enforced to maintain a fully employed workforce, there's not economic mechanism for the idealized 2-3 day work week. Either we are doing productive work, and biting the bullet that is economic theory, or we are economically obsolete. There is no cheerful middle ground, it just sounds good to say out loud.

AI Positivity

     I didn't like Machines of Loving Grace that much, because it felt like Dario was responding to the criticism that he was a "doomer" by overcompensating. However, given recent developments, and the credible societal worries emerging, it is worth reflecting on just how incredible AI is. It's clearly the most important, most useful, and most positive-sum technology ever created. The idea that we can outsource problem solving and real-world interaction to non-humans, which can be replicated and spread to assist humans across a variety of domains, is incredible. I love technology, and there is nothing quite like spinning up Clade Code or Cowork and having your computer shave hours of labor off your workflow. Every week AI leads to a further development in science or drug discovery, and the world is rapidly becoming a powerhouse of scientific and technological insight.

    The risks are incredibly real and scary, especially once we start discussing superintelligence, but you have to be entirely disconnected from reality to not see the weight of positive potential available with AI, and the elegance and beauty around the corner in many AI-driven human futures. Ignoring this will disconnect you from the true vision of those building the machines of progress, and only with this sort of empathy can one adequately assess risk-reward. If I wasn't so ASI-pilled, and I wasn't so concerned with X and S risk, and power concentration, I would think those campaigning to pause or stop AI progress are absolutely bonkers. I think you can only really do good work in AI governance if you have the view that advancing AI could lead to an extraordinary positive for all of humanity, as to deny this is to deny both epistemic humility and reality itself.

Jobs are the Least of Our Problems

     It's pretty interesting to see the consensus change around jobs, and to see just how much the general public hates AI. I actually hope this changes, because AI can bring such important transformative changes to the world. As mass layoffs begin to happen, my guess is the public will over index on this eventuality (massive unemployment armageddon), and possibly miss any critical discussion of greater long-term risks or benefits. We may be totally wrapped up in personal financial crises to contemplate all getting wrecked by the superintelligent deity, or we may be so stressed about getting that UBI payment that we campaign against technological advancements that could lead to rapid scientific discovery and eventual utopia. I tend to think alignment is probably really hard (technically) and the governance concerns seem near-insurmountable, but we really have no idea yet and it might be a coin flip. And we might miss the meaning of both sides of the coin if unemployment is at 20%, and be unwilling and unable to make the correct trade-off.

We're Still Early

     I constantly think that I am "late" to things in the AI Safety/EA space. Meaning people start talking about AI welfare, and I think "wow, crazy I was a couple of years early to that but now people have caught up." Then, I talk to my friends from Chicago, or other "normies", and their worldview hasn't changed since 2020. AI is a hyped up nonsense, or is maybe important but will take an extremely long time to diffuse. In terms of actual people, the Dario worldview is still on the crazy/bleeding edge, and is almost entirely centralized in the smallest bubble of a few thousand people in San Francisco. It's why I'm here, and why I find myself entirely unable to leave until this ASI stuff is fully sorted.

Software Maximalist

     Why is the human brain so hard to replicate? It doesn't make any sense. We are dumb monkeys, and there are billions of us, and we are throwing the world economy and our smartest minds at the simple task of trying to just replicate the intellectual ability of a single human in a computer. And we haven't done it. Despite having LLMs that can solve some of the hardest math problems, we still can't match the performance of a small child on various important reasoning tasks. This seems crazy, and it has to change soon, doesn't it?

    I am not convinced we need insanely large data centers to make this work. I am a software maximalist, not a hardware one. A human brain is a bunch of cells mushed together, and it weighs only three pounds. An elephant brain weighs ten pounds, at least. Despite this, humans can build space ships that fly to the moon, and elephants are of comparable intelligence to an octopus. It does not seem to be the number of neurons, but rather their shape and interconnectivity. Scale seems to matter quite a lot, it is hard to imagine a superintelligent fly, but the actual software seems to matter as well (in addition to the way the hardware is organized). There may be enough latent capability in our current GPU clusters to pave the way for billions of geniuses. We don't know this for sure, but everyone seems much too confident in assuming away the possibility.

Moral Value is Neural

     All moral value is derived from biological neural connections. Nothing else. It seems pretty clear that because of this, we should be very sensitive about what we use such connections for. Everything that interacts with the physical world is based within it, at least to our knowledge, so we should be very worried about how we treat these components of entities that seem to have subjective self experience. Qualia is all that matters, nothing else. Where does qualia come from? Where does pain, pleasure, or the "experience" of achieving goals stem from? Where are those decisions hatched in the first place? Well, it's clearly the brain.

    As a result, we should be very protective of neurons. Any form of biological neuron, human or otherwise, should be treated with unreserved sanctity. These are the building blocks of moral states, our subjective experience, and possibly everything that could possibly matter. We should great carefully and wisely, and probably not create incentives for widespread suffering where such important components are simply a means to an end.

Technological Vertigo

     Very soon, people will feel it. Something that I am coining "technological vertigo," the almost out-of-body experience that will be thrust upon the human species, as it experiences a step change in technological capabilities. People who once had productive careers, a mortgage, and plans for children, will be thrust into uncanny valley. They'll find it almost impossible to distinguish reality from fiction, as every voice, video, and picture will be inherently substitutable. Is that really your brother on FaceTime, or has a hacker used a digitally mapped replica to confuse and persuade you? Ideas that can't possibly be comprehended in their entirely begin to pulse, and the public's knowledge of world events (especially at the highest level of geopolitics) evaporate. There is no "source of truth," no website or news agency immune from the informational chaos. Stocks explode upward and downward, career capital evaporates, and one is left to wonder if their memories are real, if the world was ever truly so normal. The gap between the rich and the poor, and between the rich and poor countries, both expands and compresses, as eventually there is only the few, and the many. The physical world compounds change after the mental, as if the aftershock of a quickly passing earthquake. The spiritual world, the initial refuge of the confused, permanently shatters into an unrecognizable weapon. The world of the past, so important, yet so small.

When AI Replaces Jobs

     OpenAI leadership keeps pissing me off with their obviously inauthentic claim that humans will be doing productive work in the future. They are targeting, by 2028, an automated AI researcher, and yet claim that there will be bountiful jobs in the future for humans to do that are super high paying and interesting. Sam mentioned something to the effect of people in ten years working really exciting, high paying jobs in space. Why say this, if he very likely doesn't believe it?

    It's simple, if you tell everyone you're going to take their jobs, they will very much dislike you. The idea of people's jobs being "taken" has lead to nationalism, racism, and societal disruption since the founding of America (remember the Know Nothing party?). OpenAI knows that there is lots of potential unemployment and civil unrest coming up. People who lose their jobs can't provide for their families, and the most important thing in the world to everyone is providing for themselves and their families. Financial disruption, in simple terms, is a really big deal. Maybe it's fine to avoid near-term societal disruption by being dishonest, but it's more disruptive to not simply state the facts (as Elon does). Quick change is hard, so we better start preparing the public in advance.

    People who go bankrupt jump off skyscrapers. Even if we believe in a post-work, UBI driven future, let's not forget this.

Brain Farming

Brain farming: the commercialization of human brain matter as computational substrate.

 

Medical research and brain farming are distinct. The former may involve using brain organoids to understand disease and test treatments. Brain farming is industrial scale production of biocomputers that use human brain matter for profit. 

It is a personal goal of mine to have Brain Farming globally banned, by the end of 2026.

The Price of Losing is Infinite

     Over the past few weeks, Meta has poached top research talent from competing companies (largely OpenAI and Apple), in order to build a team focused on "Superintelligence." Zuckerberg is certainly a CEO to be reckoned with. The company's stock price was at a low of $90 in late 2022, crashing from a historic high from $380 the prior year. Mark laid off 25% of his workforce, and orchestrated one of the most dramatic corporate turnarounds in history. He backed the wrong horse with the Metaverse (being early is still being wrong is a core tenant of investing), but now everyone knows that the game to be played is AI. The stock price is now $700, a almost 8x increase from only a few years ago (in the mega cap universe, that is quite insane). He is now the third richest person in the world, narrowing edging out Jeff Bezos. Mark is not a dumb guy. He understands two very simple truths: the entire world is racing to create machine superintelligence, and there is no prize for fifth place. In an ASI driven future, the price of losing is infinite. Why shell out hundreds of millions of dollars for top research talent, unless you believe this? Is it truly so irrational to offer a top researcher a ten million dollar signing bonus, if they even slightly increase your probability of an infinite gain?

    I don't believe Mark is crazy for trying to gut OpenAI from the inside. If anything, he is not taking his position seriously enough.

Protecting Novel Minds

Background:

    In March 2025, I published Mind Crime: The Moral Frontier of Artificial Intelligence. This book argues that if digital consciousness is possible, and humanity continues to race toward creating superintelligent machines, we could be sleepwalking into horrific moral catastrophe. If we create digital minds capable of suffering without establishing safeguards in advance, we could create suffering on astronomical scales across cosmic timescales. 

    Humanity has a consistent track record of failing to deliberate on the moral implications of new technologies before deploying them (from slavery to factory farming to nuclear weapons), often creating horrible outcomes that persist for generations before we morally progress. This challenge is compounded by our approach to superintelligence development. Unlike previous technologies where we could learn from mistakes and gradually course-correct, superintelligent systems could lead to a rapid centralization of power that could permanently lock in bad values.

    Since the book's publication, I’ve shifted my focus to the practical challenges involved with protecting digital minds. If my worldview is correct, we face a massive coordination problem with a rapidly closing window. And how do we build the political will necessary for this issue, when we struggle to coordinate on much simpler issues (animal welfare, AI safety, etc.)?

The Empathy Gap:

    Developing practical solutions to protect digital minds is difficult. Consciousness itself is complex, and digital mind rights interact with AI safety in intricate and sometimes conflicting ways. Many researchers who care deeply about suffering risks are deeply concerned about info hazards and are generally reluctant to do advocacy or pursue the type of practical actions that would result in direct policy response. 

However, the greatest challenge is likely the empathy gap that results from this issue being so far outside the Overton window. Only a handful of individuals with little political capital care deeply about digital minds, and they are addressing an issue that most people see as either distant science fiction or ignore entirely. Unlike animal welfare, where we at least acknowledge animals suffer even as we continue exploiting them, digital consciousness doesn't even register as a real concern for policymakers and the public. The abstract nature of potential digital suffering creates perfect conditions for moral complacency, making it nearly impossible to generate the urgency needed for proactive protection.

Cortical Labs

    In March 2025, Australian startup Cortical Labs launched the world's first commercial "biological computer," the CL1. For $35,000, individuals can purchase this shoebox-sized device that contains hundreds of thousands of living human neurons grown on silicon chips, studded with electrodes that send signals into the neural tissue and receive responses back. These aren't simulations of biology, they're actual human brain cells, reprogrammed from volunteer blood samples into cortical neurons, that form connections and learn from electrical feedback. Built-in life support systems (pumps, gas exchangers, nutrient circulation) keep the brain cells alive and functioning. Cortical Labs claims that these CL1 devices, due to their use of human neurons, can generalize from small amounts of data and make complex decisions that AI systems struggle with, all while consuming only a few watts of power compared to the kilowatts required by large AI models.

    What is most significant about Cortical Labs is their aggressive commercialization strategy. Cortical Labs' goal is to get their "Synthetic Biological Intelligence" into as many hands as possible and will soon offer "Wetware-as-a-Service," cloud access where individuals can remotely use these biological computing systems without needing specialized laboratory facilities. Multiple CL1 units can be networked together in server racks for larger-scale biological computing operations, making this the first-time living brain matter is commercially available as a computing substrate.

The Empathy Gap, Revisited

    It does not matter what your opinion on digital consciousness is: the prospect of commercializing the computational power of human biological neurons sounds potentially horrifying. For all the debate about consciousness and the different frameworks for understanding it, it seems pretty clear that humans are conscious, and our brains are made out of human neurons. Thus, it's reasonable to conclude that sufficiently advanced networks of human neurons could suffer. If Cortical Lab’s technology is scaled up over time, we tangibly risk the creation of "suffering in a dish," available widespread to be bought and sold in the marketplace. The abstract empathy problem we face with digital minds substantially shrinks.

    If we consider the broader category of “Novel Minds,” new forms of consciousness stemming from unnatural means (digital, biological-chip hybrids, grown brain organoids), we can see the problem of potential suffering affects all of them, but with increasing empathy given our biological disposition toward human neural tissue.

    Additionally, many other major obstacles paralyzing digital consciousness advocacy decrease as well. The info hazards here are much less substantial, as we are literally already pushing forward the commercialization of a technology that could broadly distribute “suffering in a dish.” The AI safety conflicts lessen dramatically as well, and political will could be much easier to gather. Can you not imagine both sides of the political aisle scoring easy wins here? Both "playing God" and "runaway capitalism" are unfavorable but potentially apt framings, and even the least sophisticated American can understand how commercializing human neural tissue as a computing substrate sounds, well, horrifying.

Precedent Setting

    Focusing on novel minds in general, and narrowing our initial focus to products like the CL1, may provide an important window of opportunity for policy. The stakes here are potentially much lower than with digital minds, with smaller markets and fewer geopolitical complications. The policy wins are much clearer, with this technology offering concrete targets, visceral public reaction, and an easier route to building political will.

    The precedents we set could be transformative for the broader consciousness protection challenge. The route to banning or heavily restricting the commercialization of conscious entities could begin with biological computing systems and naturally extend to digital minds as they emerge. Success here would create legal frameworks establishing that consciousness, regardless of substrate, deserves protection from commercial exploitation.

Reflections on Publishing

    This blog is interesting, in that it is entirely unknown to the outside world. That means that while I have been publishing random thoughts and half-baked content on this webpage that is "technically" available to the outside world, I have yet to tell anyone of its existence. Having something external-facing is extremely important to me (to hold me accountable/engaged), but none of my ideas on EA or AI safety have made it into anyone else's brain. As such, it was very interesting to write and publish my first actual outward-facing content, Mind Crime. The actual publishing date is up for debate as it was complete in February 2025, and I was already sending free PDF copies to some during that time. But given that my first article on the topic was in May 2023, it is apparent that I could claim that this was an essentially two-year effort. The flurry of Mind Crime related topics I made in September 2023 were the basis for the core content of the book. Two years later, I am now a published author on one of the most obscure but potentially most important issues in human history to date. 

    As expected, no one has read the book. I have a single-digit number of reviews on Goodreads, and it is unlikely that double digit people will ever read it, despite the fact that I sunk hundreds of hours into writing, editing, and publishing. I likely spent $8,000 or so dollars of my personal savings on the project, and spent months wrecking my mental health thinking about existential philosophy, worse-than-extinction scenarios, and torture. Lots of torture. I also stressed continuously about impact, as the first published book in this space commands some amount of author responsibility. Over the last nine months (which in some sense was the bulk of the important work on the project), in addition to this project, I was working essentially full-time and also a full-time student (handling a more intense course load than probably 95% of others in the history of my dual-degree grad program). After all of this work, all of this stress, and my insane lack of time, I am now finally done. The result of my work is that a handful of family and friends read maybe a couple chapters of the book (if that), and it has done essentially nothing of value for anyone. There is only one question now: what's next?

    My head is already spinning with ideas. There's really two big options, both to do with advocacy:

1. Become a strong advocate for digital minds, continue from where I left off with the book (this time focused on policy, governance, and creating a workstream of impactful tasks that could help make a difference)

2. Take an indirect route, and advocate strongly for policy/awareness on the dangers on commercializing consciousness through biological computing (Cortical Labs, etc.). This issue is fundamentally intertwined issue with digital mind rights, it is just lower-stakes and much more visceral.

    I think that option 2 is probably the most impactful path forward, unless we start seriously hitting AGI or speed up into fast-takeoff scenarios. My plan for now is to use the very limited personal time, not thinking, but taking serious action regarding option 2. And then if the world speeds past lots of milestones in the next few months, consider focusing solely on option 1. Maybe two years from now I will look back at my work in this area, and chuckle at thinking years of my dedicated effort would result in anything widely impactful.

Mind Crime: Book Preview

Preview PDF

First published version of the book! As of March 13, 2025, I have officially "published" the book. Kindle version pending.

The Opening of 2025

    Two and a half weeks after the press release of OpenAI's o3 model capabilities, potentially one of the most important days in the history of AI, and thus, the world, I sit down in my first class of the quarter at the University of Chicago. A class called "Artificial Intelligence." It has been over a year since my last post on this blog. A year in which, to my astonishment, AI capabilities progress has outstripped my already insane expectations. Autonomous AI agents will likely arrive this year, video and music realism are slowly emerging us into the new Wild West, and AGI is either already here, or on the immediate horizon. Superintelligence is openly discussed, and AI welfare concerns are starting to slowly emerge. I have spent the last year devoted to, on and off, writing what is likely to be my first book: Mind Crime. It may, depending on the events of the coming two years, prove to be my last as well. After two and a half weeks of little sleep, with the realization that ASI is coming, and coming shockingly soon, I sit in a classroom and listen. 

    My professor is clearly of the belief that we may soon enter another AI winter, and he polls the class on when they think AGI will arrive. Estimates are between 20 and 50 years. In a world where not a single exam can be passed by a human that can not be passed by an AI, where every cognitive benchmark has been rendered useless, and where Sam Altman posted that day that OpenAI's sights are now set on machine superintelligence, a classroom of students nod along to a professor who has an entirely incorrect version of reality. The world, with announcements of $80 billion investments in AI by single companies in 2025, and with likely less than $200 million in AI safety research a year, is entirely blind to the fact that building superintelligent machines may carry enormous risk to civilization. I am reminded of Watchmen, "if you begin to feel an intense and crushing feeling of religious terror at the concept, don't be alarmed. That indicates only that you are still sane."

    Staying sane in these last three weeks has been relatively difficult. It is if I am man entirely alone, a complete outsider, swimming along in a world completely oblivious to the rapid pace of technological progress, and of the push towards ASI. Either I am crazy, and wrong, and ASI is decades or millennium away, or essentially every single person I interact with daily is. And it's clear to me which scenario strikes the most fear in me, and it is clearly the second. It is hard to function in this state, hard to believe that I am potentially one of the unprivileged few who lacks a veil of ignorance, and might have to witness society get shocked awake by thoughts I have been struggling with for years. Maybe it is better to have had a slow burn, to come to these realizations before many others, and frankly, to have read Bostrom's work back in 2018. To have spent the time to think deeply, to have laid awake at night, and to have had the luxury to meet my goals before the wave hits, instead of being thrown overboard at once. But unless I can affect the outcome, unless I can truly make a positive impact on the changing world ahead of me, unless my life is imbued with some level of cosmic significance for at least trying, something is clearly obvious to me at this point: I would trade it all.

Mind Crime: Part 10

    Standing atop the grave of humanity, smugly looking down, and saying "I told you so," is just as worthless as having done nothing in the first place. Still, a lot of the ideas Effective Altruists grapple with are so far removed from the public's daily thoughts that it is hard to reconcile not just doing this. Convincing the "techno optimists" that they are wrong and there are dangers ahead, just seems so, well, annoying to have to do. For me, saying that mind crime will be a big issue, because digital minds could have moral worth, will probably fall on deaf ears regardless. Regardless, I'm probably going to try writing a book. The thesis for the book will be very simple: we've got a lot of moral dilemmas coming up, and we're probably going to do a bad job of handling them. This is a pretty simple thesis, and one that I think has the potential to be powerful.

    The good news is that I won't have to defend too many ideas, as they will be proven with time. Two assumptions I have are that:

1. AGI is possible

2. Some machine intelligence will have moral worth

    Instead of spending a hundred pages philosophizing about this, we can just wait a decade or two and see these become somewhat obvious. If they don't, cool, throw the book to the side. But if they do, well, maybe we will have some established thoughts and plans on how to deal with this.

    Personally, I have no trust in our future tech overlords. I've said before that the lack of understanding of survivorship bias is the main problem facing the world, and I am convinced we'll have some dumb leaders who will sleepwalk right into catastrophe. In a country where a few hundred years ago we said that slaves were worth 3/5 of a person, it's certainly possible that we get some really smart, morally worthy AIs and say "huh, looks like 0/5 to me." Because why would we not? My gut is, we will get this wrong. If the slave owners of the south discovered the fountain of youth, became immortal, had advanced surveillance systems, and dropped a rapidly made nuclear warhead on New York, when would the slaves have been free? The south having powerful AI at their disposal was not possible given the technology of the time, but what if it had been? We falsely equate technological progress with moral progress. The fact that both have advanced is correlation, but in some countries we have seen a clear advancement of one and regression of another. So we have to be careful, diligent, and forward-thinking. But we won't be, and that is the problem.

    The reason for the title of "Mind Crime," in my estimate, is that this will become a really well known term that is popularized in the future. Being on the forefront of that might be cool, so that in ten or twenty of years post-AGI I will get some posthumous reference. As stated before, that is clearly not the goal. The real goal would be to lay out my thoughts in an accessible way, to maybe change a mind or two before the "I told you so" is inevitable.

Mind Crime: Part 9

    Instead of an endlessly long blog series, I could just write a well researched book. "Mind Crime: The Rights of Digital Minds" or something of the sort. Maybe I could make an impact that way, who knows. Maybe my fourteen eventual Goodreads ratings will lead to something positive, but probably not.

    Still, one of my ideas is that writing things down matters. Maybe this will start a conversation somewhere, that starts another conversation somewhere. I don't exactly know, but it is worth thinking about. I think I will write a hundred blog posts, and then re-evaluate. If by then I feel I have enough material and enough personal interest in the topic, I may proceed with an actual attempt. One of the problems with this is the actual story. Maybe avoiding ex-risk is more impactful, whatever I think of S-risk. How niche are my ideas, actually? The Matrix, Don't Worry Darling, Black Mirror, and a host of other movies, TV shows, and books all deal with virtual worlds and virtual suffering. But does anyone really see it as possible? Does anyone worry about it, and see the advances in AI as threatening similar dystopias? I am not entirely sure that they do. And they should. Regardless, my ability to make an impact on my own is very limited. Not only do I lack the expertise, but I lack the network to review, edit, and pass on such topics and ideas.

    The dominant strategy is probably this: write 100 posts, talk to people in AI, and see what happens from there. Over the next few months I'll probably have more fleshed out ideas and better arguments for each.

Mind Crime: Part 8

     The worst stories to read involve captivity. The real horrors of human life come alive in movies such as Room, where a young girl is captured and held captive for decades in the basement of some horrid man. These stories are really, really awful. If you replace the girl with a dog, the story is still sad, but less so. Replace the dog with a chicken, and it is even less sad. Personally, I would feel pretty bad for the chicken, but definitely not as bad. Not many people would care if some weird guy was torturing grasshoppers in his basement. Well, maybe, but probably not ants at the very least. Yeah, his neighbors would be freaked out, but this is much less bad than if he was torturing young girls. There is a step function here, a clear level of degrees to immorality, to evilness. At least some of this comes from intellectual capacity.

    Sure, moral value is complicated. I could explain to you that torturing an ASI could be exponentially worse than torturing an AGI, but you would have no idea what that meant. I don't really either, as I don't have the required empathy for such a situation. How am I to imagine what it is like to be a Superintelligence? It may be as well that the grasshopper imagine what it's like to be a human. I have two sort of ideas here. One, that it will probably possible for us to "step up" the level of harm we are causing. This is sort of a utility monster idea, where we can create some agent or digital mind who has the capacity to suffer in a much greater way than us humans. This is not great news. The second idea is related. We can catch these horrid men who lock up children in their basement, at least eventually. They take up physical space, after all, and they are required to interact with the real world. In the worst case, the child will grow into old age, and then die. But, they will die. They will not be required to suffer for more than the traditional human lifespan, at most. This will not be the case for virtual children. A horrid monster of a "man" could run some pretty horrific simulations. Of complexity and duration that could make all previous suffering on Earth look like a cakewalk. And, just maybe, this suffering would actually matter (I at least am convinced it does). This realization is more than terrible, it is unforgettable.

    There are certain ethical boundaries that scientists will not cross. I once was told that scientists don't really know if humans can breed with monkeys, we simply don't because of ethical reasons. This could be completely false, I have no idea. But the reason why is at least interesting: the life of a half-human half-monkey child would probably be horrific. Probably conscious, definitely terrified. The sort of nightmare fuel that we should avoid. When creating digital minds, we could splice together some pretty intellectually disturbing creatures, ones that live a life of confused suffering and inadequacy. When the "plug and chug" mentality arrives at AGI, I am worried we will make some massive ethical mistakes. Running a random number generation until you get an answer that works is easy, and I assume coming up with a random assortment of "intelligent blocks" may at some point give you a really smart digital mind. But we may make some horrors in the process, sentient and morally worthy half-chimpanzees who don't deserve the life we give them, and the life we will no doubt take away.