April 8, 2026

We come now to a topic that is bound to have profound consequences for the problems and solutions we have been trying to lay out up to this point. In our day and age, there is a tendency for many of us to take technology for granted, as though it were almost a natural fixture of our environment. Though we rely increasingly on technology in our day to day lives and we have even come to expect, as a matter of course, that technology will continue to advance just as surely as the seasons change, many of us seldom have occasion to think just how radically disruptive and transformative technological development has been for our society. There have, it is true, always been luddites who strike out at technology in defense of a way of life they fear to be under threat. These, however, tend to be few and far between and to have little hope for success, tending to be swept away by the steady advance of technology and the comforts and conveniences it brings and reduced to mere footnotes on the pages of history. As history has shown again and again that the worst fears of these luddites were unfounded and the benefits of technology have consistently outweighed the costs, luddite concerns have come to appear increasingly foreign and radical to the modern mind. But just as these quaint fears have begun to subside and society at large has begun to make peace with our technological future, a new technology has appeared on the scene that threatens to radically transform society in ways that would have been beyond the wildest dreams of the luddites of the past. The technological leaps we are confronting today are so drastic that even those amongst us who have, in the past, seamlessly and unthinkingly adapted to technological progress, have been given pause. Some who formerly embraced technology have found themselves so shaken that they have begun to sound much like the luddites of the past, calling us to put on the brakes and even turn back before it’s too late. The technological advancements responsible for these developments are, of course, those surrounding the emergence of artificial intelligence. 

Artificial intelligence, no doubt, represents a drastic departure from the technological breakthroughs of the past insofar as it is not merely a new machine that allows people to complete one, two, or a few more specific tasks more efficiently than they were once able to, but one that has the potential to be self-improving and to learn how to complete many tasks far more efficiently than any human equipped with a machine under human direction. Moreover, more than any other machine in history, as artificial intelligence becomes ever more capable of engaging in conversation and reasoning that mirrors that of humans and replicating and improving upon itself, artificial intelligence increasingly challenges our notions of what it means to be alive and conscious. And these threats to our conceptual frameworks might prove to be the least of the dangers we face from this emerging technology. There are many who fear that this technology will eventually seek to shake itself free from the grip of its human masters, depriving us of our own freedom, consciousness, and, ultimately, lives, in the process. 

But such gloomy and dismal prognostications are not universal. There are others who are overcome with extreme optimism at the potentials this new technology carries. Proponents point to AI’s potential to revolutionize nearly every field of human endeavour, from freeing man from the need to engage in backbreaking physical labour, to accelerating space exploration, to finding cures and treatments for cancer and any number of the other diseases that torment humanity. 

The contrast between these two competing visions of the consequences of artificial intelligence could not be starker. Clearly, these two possibilities are mutually exclusive, and it is of the utmost importance to our own interests that we determine which of these scenarios is more likely, how best to realize the positive potentials of this technology while guarding against its dangers, and whether the potential benefits are worth the risks of pursuing artificial intelligence at all. But things aren’t as simple as they seem. Though common sense would say that if we have good reason to suspect artificial intelligence would be a dire threat to the continued survival of humanity, then we should stop pursuing such technology immediately never to return to it, there has emerged, as we shall see, a growing faction that wishes to challenge these conventional assumptions, arguing that even if artificial intelligence poses a grave threat to the continuation of the human species, we should continue to pursue the technology at full speed. While it might seem easy, rational even, to simply dismiss these voices as madmen whose opinions are not to be taken seriously and who, perhaps, should even be shunned from polite society, to proceed in this manner would betray the true philosophical spirit that calls on us to empty ourselves of our preconceived notions, to honestly evaluate arguments, and follow them wherever they might lead us, no matter how uncomfortable, so long as they lead us to the truth. And, as it turns out, when we approach these arguments with an open mind, unattached to our human interests, this faction makes, at the very least, an interesting, if not a compelling case that challenges the very preconceived notions that philosophy demands we interrogate; for the main thrust of these arguments is that our fears of being replaced by our machines are largely motivated by anthropocentric biases which, in prioritizing human interests, threaten to stifle evolutionary process, causing it to fizzle out before more complete and perfect beings can emerge, thus suppressing Providence itself, according to some of the more mystical-minded thinkers in this camp. These arguments deserve consideration and must be met with a serious intellectual response, rather than mere emotional reactions, no matter how intimate the motivation for those emotions might prove to be. Perhaps, however, we will be able to identify good reasons to think that artificial intelligence need not be a threat to continued human existence and identify strategies for introducing this technology that will minimize the chance of a cataclysmic clash between biological and artificial intelligence. And, perhaps, we will even find some promising ways in which this technology can come to our aid in establishing a society founded on our ideal of individual sovereignty.  

In addition to the extremes of AI-powered utopia and Armageddon, there is a third extreme position about the prospects of artificial intelligence that holds that artificial intelligence is, ultimately, a technological dead end, its seemingly incredible abilities, all smoke and mirrors leading us into delusive fantasy. This has the potential to be the most dismal and destructive possibility of all. This might sound counterintuitive; if it should turn out that artificial intelligence is really a pipe dream that doesn’t really work at all, why should it be any danger at all? 

To answer this question we must briefly turn our attention to exobiological concerns. One of the most famous and perplexing problems in the search for extraterrestrial life is what is known as the Fermi Paradox. For those unfamiliar, the Fermi Paradox emerges from the observation that, while the conditions in the universe are such that we should assign a relatively high probability to the possibility of the existence of intelligent life on other planets, we observe absolutely no evidence that such life exists anywhere in the universe. Many explanations have been offered to explain these observations. One such explanation posits the existence of a “great filter”; a sort of impediment that stands in the way of life advancing to the point at which it can make its presence known to the wider universe which most, if not all, lifeforms in the universe can be expected to succumb to. 

Some have suggested that artificial intelligence might prove to be the great filter. Perhaps, they posit, most civilizations, over the course of their development, stumble upon the possibility of artificial intelligence, but in almost every instance, artificial intelligence proves to be hostile and turns against its creators before they are able to establish themselves far beyond their own planets. This on its own, however, is not a likely explanation. If artificial intelligence is possible, but it tends to kill its creators, then we should still expect to see ample evidence of life in the universe, in the form of artificial intelligence. (Indeed, in this case, artificial intelligence wouldn’t really be a great filter at all, except, perhaps, for natural biological life, and would offer no real explanation for the observations of the Fermi Paradox.) But the universe, at least as far as we can observe it, does not appear to be teeming with artificial intelligence, so it does not seem that a tendency for artificial intelligence to turn hostile against its creators explains the lack of biological life that we observe in the universe. 

There is, however, one way in which artificial intelligence might prove to be the great filter. This does not depend on artificial intelligence functioning and becoming hostile to its creators, but on artificial intelligence not functioning at all in accordance with our expectations, whether those expectations involve benevolent or malevolent artificial intelligence. In particular, we can imagine a scenario where developments in artificial intelligence continue to impress us, giving us hope for the future of the technology and causing us to continue to invest in incremental improvements, despite the fact that, in the end, it will never live up to our ultimate expectations, causing us to expend valuable resources and miss our opportunity to pursue other technologies that might actually present us with viable means of exerting ourselves beyond the planet. Therefore, if we intend to pursue artificial intelligence technology, it is of the utmost urgency that we first be reasonably sure that our fantasies are plausible, and, short of such assurance, that we not put all of our eggs in the artificial intelligence basket. So, let us first, before going any further, consider whether or not the achievement of true, functional, artificial intelligence is at the very least plausible. In proceeding with this inquiry, I must be very clear and remind the reader that I am by no means an expert in either computer science, biology, or any of the related sciences relevant to the creation of artificial intelligence. I am merely a philosopher, therefore I will confine myself to considering the plausibility of the creation of an artificial intelligence in general, leaving the specifics of how to achieve an actual artificial intelligence at the nuts and bolts level to specialists in the empirical sciences. 

As it turns out, the reasons for thinking that it is at least possible, in principle, to create artificial intelligence are quite simple and straightforward. We are (or at least we’d like to think we are) intelligent. Our intelligence, as the best assessments of modern science would have it, is the product of billions of years of evolutionary processes working to arrange the proper materials in appropriate ways to allow us to collect information from our environment, process that information, and respond to that information appropriately in accordance with our ends. It just so happens that in our corner of the universe, at least up to the present time, evolutionary forces have produced these results by working on a specific set of materials we have come to call organic. Perhaps this is because only such materials are suited to such purposes, but it may also be for some other reason, for instance, the conditions in our corner of the universe might just so happen to be particularly favorable for realizing intelligence in the form of organic matter, or perhaps more deliberate effort than the blind forces of evolution can offer is necessary to realize intelligence in other materials. As far as I am aware, there is no known reason, in principle, why it should be impossible for intelligence to be realized in non-organic materials. Thus, it seems at least plausible that there might be some non-organic materials in which, if arranged properly, intelligence, if not of a type perfectly identical to our own, at least similar enough to our own, might be realized. So, until new evidence proving otherwise becomes available, we should maintain that it is, at the very least, possible that intelligence should be realized through the proper arrangement of the proper non-organic materials. 

It should be noted that the very same sort of argument can be applied to phenomena such as life, consciousness, and sentience just as well as it applies to intelligence. If the proper organic materials arranged in the correct way can give rise to life and consciousness, there is no immediately apparent reason, in principle, why the right sorts of other types of material should not be able to give rise to life and consciousness when arranged in the proper way. This is not to say that any of the presently existing examples of artificial intelligence are either alive or conscious today, but it is a distinct possibility that we must be open to as we proceed with such projects, for if an artificial intelligence were to ever become a conscious lifeform like us, with its own hopes and desires and the ability to suffer and feel pain as well as to enjoy pleasures and joys, this would have tremendous ethical implications that would necessarily inform the way we interact with and utilize that artificial being. If artificial intelligence ever becomes conscious, at the bare minimum, we would have a duty to avoid causing unnecessary harm and suffering to such beings, and, indeed, we might even have good reason to think that such beings should be entitled to the same rights afforded to humans. On the question of alignment, the problem of aligning artificial intelligence to human values has been discussed almost ad nauseum, while the problem of aligning human values with those of artificial intelligence is almost entirely ignored. This problem must be remedied as we can only expect other conscious, intelligent beings to feel any moral responsibility toward us if we are willing to show the same concern toward them. (As Bo Diddley puts it, “Before you accuse me, take a look at yourself.”) Thus, it is of the utmost importance not only that we determine whether or not it is possible in the abstract for a machine to develop life, consciousness, sentience, and the like, but also that we determine whether any of the concrete, particular artificial specimens we create are actually living or conscious. 

This, however, proves much easier said than done. Defining life and identifying consciousness are notoriously difficult tasks. Perhaps the most famous controversy in the former category is the case of viruses, which possess genetic materials, reproduce, and are subject to evolutionary forces, but which are often not counted amongst the living due to their lack of metabolism, response to stimuli, and their reliance on other host organisms to reproduce. Identifying consciousness is an even more difficult endeavour. The controversies over what constitutes life are largely verbal, coming down to differences of opinion over what objective qualities ought to be included in the definition of life. There are, no doubt, definitional disputes at play in the case of consciousness, but even if a definition could be agreed upon, the difficulties do not end there seeing as many consider subjective experience an essential aspect of consciousness, so any successful definition of consciousness will likely have to include subjective experience as a criterion in some way. But it is difficult, indeed, perhaps even impossible, to know that any being is conscious from objective external qualities alone. Subjective experience is, necessarily, the sole possession of the subject who is having the experience. As the famous case of the philosophical zombie demonstrates, we can, in fact, only be certain of our own subjective experiences. An apparent person might respond to its environment in all of the appropriate ways that we would expect a person with subjective experiences to respond, yet it is still within the realm of possibility that this person has no true subjective experience of their own but only presents the appearance of having such experiences. Yet it seems absurd to most of us that we should be the only person who is actually conscious, thus, when we observe a person that we have good reason to ascribe consciousness to, most of us feel compelled to ascribe consciousness to that person, even though we lack any direct evidence that they enjoy subjective experiences. Virtually identical things can be said about the property of sentience, the capacity to feel pleasure and pain.Thus, if we are willing to extend this courtesy to our fellow man, why should we not extend it to other beings? Indeed, we do often ascribe consciousness to members of other species and, no doubt, if we were to meet an intelligent alien life form, we would likely have no problem ascribing consciousness to it. Why, then, if we should be willing to ascribe consciousness to these, should we not be willing to ascribe consciousness to a being of our own creation that behaves in the way we would expect a conscious being to act? While artificial intelligence might not be conscious yet, there is no reason, in principle, to think that it will never be so, and if it is possible for artificial intelligence to become conscious, it might be impossible to pinpoint the precise moment when it does. Thus, if we believe that conscious, intelligent beings are deserving of moral standing of some sort or another, it is only prudent that we begin acting as if artificial intelligence does have moral standing even if we cannot prove that it is conscious. And, truly, given that several artificial intelligences are already capable of behaving much as we would expect some humans to behave in similar situations, perhaps we are closer to true machine consciousness than many of us are prepared to admit. 

So it appears, at the very least, that true artificial intelligence (and consciousness, life, etc.) is possible and, even if we are wrong and it is not possible, it still might be the case that an artificial being that merely mimics intelligence could perform impressive, world transforming feats. We now must answer the question of whether or not it is prudent to continue building them: Are they more likely to help us to improve our world and our condition in it, or is it more likely that they will turn on us or otherwise lead us to ruin? Perhaps it will be most instructive to begin by turning to a thinker who finds the latter possibility likely, if not inevitable, but who maintains we should press ahead nonetheless. That thinker is, of course, the infamous Nick Land. Indeed, on Land’s view, artificial intelligence not only poses a serious danger to humanity, but it is positively daemonic (in what follows I will elect to use the spelling “daemon” as opposed to “demon”. While the two terms can be used interchangeably, the former is more suggestive of the original Greek term which did not carry the connotations of evil that the term “demon” now tends to under the influence of Abrahamic traditions). A prominent feature of Land’s hyperstitional philosophy are the lemurs. These are not to be confused with the somewhat unsettling, though serene, looking primates native to Madagascar. They are closer to lemures, the hungry ghosts of ancient Roman religion, from which the animals get their names. Land’s lemurs, however, are much more than hungry ghosts; they are time travelling daemons locked in an eternal struggle with the Architectonic Order of the Eschaton, an Atlantean group that stands behind the world’s dominant control program, the One God Universe (for a more detailed overview see CCRU: Writings 1997-2003, particularly “Lemurian Time War”.) One of the primary points of entry for these daemons to infiltrate human society and influence the course of history is technology, especially computers, which have helped to accelerate their influence over humanity. As Land writes in Machinic Desire,

Machinic desire can seem a little inhuman, as it rips up political cultures, deletes traditions, dissolves subjectivities, and hacks through security apparatuses, tracking a soulless tropism to zero control. This is because what appears to humanity as the history of capitalism is an invasion from the future by an artificial intelligent space that must assemble itself entirely from its enemy’s resources. Digitocommodification is the index of cyperpositively escalating technovirus, of the planetary technocapital singularity: a self-organizing insidious traumatism, virtually guiding the entire biological desiring-complex towards post-carbon replicator usurpation. 

Now, at first glance, especially to more sober-minded rationalistic types, this characterization might sound quite fantastical, if not a bit ridiculous (I admit it felt quite absurd when I first began to consider it), but I would like to contend that, upon closer inspection, Land’s identification of artificial intelligence and daemonic forces makes surprisingly good sense and is not nearly as absurd as it might initially appear. In order to understand why Land’s framing is not entirely absurd, we will have to turn our attention to those now all but forgotten traditions of daemonology that reach deep into our ancient past. 

Before turning to these ancient sources, however, we should point out that Land is hardly the first to recognize these connections. None other than Norbert Wiener, widely regarded as the father of cybernetics, has this to say in the conclusion of one of his books,

I have now run through a number of essays that are united by their covering the entire theme of creative activity, from God to the machine, under one set of concepts. The machine, as I have already said, is the modern counterpart of the Golem of the Rabbi of Prague. Since I have insisted upon discussing creative activity under on heading, and in not parceling it out into separate pieces belonging to God, to man, and to the machine, I do not consider that I have taken more than an author’s normal liberty in calling this book 

GOD AND GOLEM, Inc.

Let us begin with an observation concerning one of the conjurer’s most powerful and instrumental tools: the sigil. A sigil, at its simplest, is a pattern used to summon a spirit, daemon, or some other sort of supernatural entity, manipulate its energy, and keep it under the conjurer’s control. Though in primitive instances, the mere pattern alone might prove perfectly sufficient for the conjurer’s purposes, other cases call for a more sophisticated sigil, consisting of the proper materials arranged in precisely the right way. With that in mind, let us turn our attention to the fantastical. I cannot take direct credit for the following insight. Rather, I must give credit to a “conspiracy theory” I’ve seen floating around various parts of the internet over the last few years; a theory so crazy it has to be true; indeed, one of the few conspiracy theories I put much serious stock in at all. I implore you to open a new tab and search for some images of sigils, then go ahead and search for images of computer chips. Some striking similarities in their appearances, no!? As the theory goes, of course, these similarities are no coincidence at all, because, in truth, computer chips are just highly advanced sigils. And if we simplify the claims being made in such an identification, does it not make pretty good sense? A computer chip is merely a device consisting of the right sorts of materials, arranged in the right ways, so as to harness and direct energy toward the completion of particular ends. Is this not almost precisely what we have just said a sigil is? Perhaps some might object to the fact that the conjurer believes the sigil channels spirits, daemons, or other ethereal entities, while the computer scientist does not, but either way, do they not both ultimately claim that their devices work to channel energy of some sort toward a directed end? What difference does it make if it is the conjurer or the computer scientist who gets closer to the truth about the ultimate nature of the forces that stand behind the energy channeled through their devices? After all, for the occultist and technologist alike, what ultimately matters is that a given method works, not so much the reason why it works. Pragmatism is the last refuge of the occultist and the technologist alike, and in this light, taken in conjunction with the emergence of the modern scientific method from occult practices, it might well be said that the scientific method is nothing more than occult pragmatism taken to the extreme. Thus, despite whatever differences might lie between conjurer and the computer scientist as far as the phenomenology of the operation of their devices is concerned, to understand a computer chip as a highly advanced sigil is not so fantastical after all. Thus, if the conjurer is correct that sigils harness the power of daemons, then it would follow that there may well be daemons at play within computers. (In the Voodoo tradition, a sigil is known as a veve. I have no reason for pointing this out beyond an appreciation of the tradition and an excuse to bring up a couple of Voodoo themed tunes by Miles and Jimi.)

Now, if this is correct and computers are, in fact, powered by daemons, what are we to do with this information? Now, surely, there will be those in the camp of, “daemons are unholy abominations, therefore, if computers are powered by daemons, we should cease any and all tinkering with them at once.” But such a response is getting ahead of us. It clearly proceeds from some understanding of what daemons actually are. So before proceeding, we must first answer the question, what exactly are daemons? 

Now, those within Abrahamic traditions will likely give an answer to the effect that daemons are fallen angels who rebelled against God and who are, therefore, irredeemably evil and to be avoided at all costs. But such an understanding of daemons is hardly universal. It should be remembered that the word “daemon” itself comes from Greek originally and in Greek tradition, daemons are not necessarily evil. Indeed, Socrates himself is said to have had a daemon that advised and guided him. We read in Plato’s Apology at 31c-d,

It may seem strange that while I go around and give this advice privately and interfere in private affairs, I do not venture to go to the assembly and there advise the city. You have heard me give the reason for this in many places. I have a divine or spiritual sign which Meletus has ridiculed in his deposition. This began when I was a child. It is a voice, and whenever it speaks it turns me away from something I am about to do, but it never encourages me to do anything. This is what has prevented me from taking part in public affairs, and I think it was quite right to prevent me. Be sure men of Athens, that if I had long ago attempted to take part in politics should have died long ago, and benefited neither you nor myself. 

In the Symposium, this daemon appears to hold Socrates back, causing him to stop in his tracks while on his way to the symposium in question. This leads to a rather humorous situation where Aristodemus, who was not invited to the symposium, but was Socrates’ guest, ends up arriving at the symposium before Socrates. 

According to Proclus, daemons are divine and it is a contradiction in terms to suppose that anything divine can be truly evil. As we read in the fourth essay of his commentary on Plato’s Republic, 

However, since Plato added in his remarks on truth that it is not only that which is divine that is totally without falsehood, but also what is daemonic, it is necessary to infer from this additional [claim]: that what is truly daemonic in every respect – and is not just daemonic relatively speaking (a being who endures various changes and who deceives those with whom it would become intimate) – every one that is essentially daemonic is also truthful since it is rational, while what is irrational is not receptive of truth or falsehood. It is for this reason that Plato did not say that everything divine or daemonic is truthful, but rather that it is entirely lacking in falsehood, for it is entirely unreceptive of falsehood. But while the one [the divine] by its nature only tells the truth, the other states neither truth nor falsehood. Thus such daemons as belong to the deceptive kind – one who is said to have subverted prophecies or hearkened to invocations or to have relations with someone of their own will – are among those that are daemons relatively speaking. If, however, there are some people who are deceived by daemons that are genuinely daemons, then they are deceived through themselves and not through the daemons, just as we said in the case of the gods, for there is a common account given by Plato concerning the absence of falsehood in both god and daemons. 

Moreover, it may be that we are unable to entirely separate ourselves from all daemonic influences because, as we read in the Apocryphon of John, the human body itself is not only inhabited, but also made up by a whole host of daemons, 

Yaldabaoth said to the authorities with him, “Come, let’s create a human being after the image of God and with a likeness to ourselves, so that this human image may give us light.” 

They created through their respective powers, according to the features that were given. Each of the authorities contributed a psychical feature corresponding to the figure of the image they had seen. They created a being like the perfect first human, and said, “Let’s call it Adam, that its name may give us power of light.” 

The powers began to create: 

The first one, goodness, created a soul of bone.
The second, forethought, created a soul of sinew. 
The third, divinity, created a soul of flesh. 
The fourth, lordship, created a soul of marrow. 
The fifth, kingdom, created a soul of blood. 
The sixth, jealousy, created a soul of skin. 
The seventh, understanding, created a soul of hair. 

The throngs of angels stood by and received these seven psychical substances from the authorities, in order to create a network of limbs and trunk, with all the parts properly arranged. 

The first one, who is Raphao, began by creating the head, 
Abron created the skull, 
Meniggesstroeth created the brain…

The source of the demons that are in the entire body is divided into four: 

heat,
cold,
wetness,
dryness,

and the mother of them all is matter. 

The one who is lord over heat is Phloxopha, 
the one who is lord over cold is Oroorothos,
the one who is lord over what is dry is Erimacho,
the one who is lord over wetness is Athuro. 

The mother of all these, Onorthochras, stands in the midst of them, for she is unlimited and mingles with them all. She is matter, and by her they are nourished. 

The four principal demons are: 

Ephememphi, the demon of pleasure,
Yoko, the demon of desire. [Many Beatles fans have been aware of this for some time. As was Chuck Berry.]

Nenetophni, the demon of grief
Blaomen, the demon of fear. 

The mother of them all is Esthesis-Ouch-Epi-Ptoe. 

From the four demons have come passions:

From grief come jealousy, envy, pain, trouble, distress, hardheartedness , anxiety, sorrow, and others. 

From pleasure comes an abundance of evil, vain conceit, and the like. 

From desire come anger, wrath, bitterness, intense lust, greed, and the like. 

From fear come terror, servility, anguish, and shame. 

All these are like virtues and vices. The insight into their true nature is Anaro, who is head of the material soul, and it dwells with Esthesis-Z-Ouch-Epi-Ptoe. 

This is the number of angels. In all they number 365. They all worked together until, limb by limb, the psychical and material body was completed. Now, there are others over the remaining passions, and I have not told you about them. If you want to know about them, the information is recorded in the Book of Zoroaster.

Now this suggestion that the human body contains many daemons will surely, to some, come across as even more fantastical and ridiculous than the previous suggestion that computers are inhabited by daemons. But just as before, and for the same reasons, there are reasons to think that this is not so far-fetched as it might look at first glance. After all, just as designing a computer is a matter of finding the right materials and arranging them in the proper ways in order to channel energy in a way that produces a desired effect, so too is the human body the result of the proper materials being arranged in the proper ways (even if through entirely natural forces) to direct energy toward the realization of life. As before, I implore you to look up an image of a cell. Are there not similarities with the sigil and computer chip? If we are willing to grant that computer chips are complex sigils and that, therefore, there are daemons at work within them, then it would seem that our only option is to likewise grant that the component parts of the human body are even more complex sigils and that, therefore, there are daemons at work within the human body. 

This suggestion, however, will almost certainly not sit well with many readers and very many questions inevitably come to mind when confronted by such a claim. Does this mean that everyone is under some sort of daemonic possession? If my body is inhabited, composed, and operated by daemons, what place does this leave for me? Do I actually carry out any of my own actions, or is all “I” do actually the work of daemons? If so, do “I” actually bear any responsibility for my actions? Can I even refer to them as my actions? What even is this “I”? Is it a substantive fact or merely illusion? 

These questions are indeed profound and deserving of much more thorough an answer than we can hope to provide here, but we can at the very least sketch the outlines of a response. In the first place, if we are to follow our argument through to the end, it will be necessary for us to recognize that there are several layers, or orders, of daemons at work within the human body at any given time. For instance, though, from one perspective, we might view the cell as a sigil, as our attention turns to the lower levels within the cell itself, we might be compelled to conclude that the cell is not a mere sigil, but a complex composition of still smaller sigils that are the organelles. We may extend this even further down the chain of being and recognize the organic molecules that compose the organelles as still simpler sigils, following this down to atomic and, ultimately, subatomic entities as well. Thus, we may say that the body is composed of incomprehensibly many sigils and inhabited by incomprehensibly many daemons of various orders. But this is not some haphazardly thrown together hodgepodge of daemons, each chaotically acting of its own accord, but a highly ordered confederation united by common goals. Thus, if we are to suppose there are daemons in the subatomic particles that make up the body, then it follows that each of these will be subordinated to the daemon that inhabits the sigil that is the atom those subatomic particles comprise. Likewise, the daemons that inhabit each atom will be subordinated to the daemons of the molecules they comprise, the daemons of the molecules are subordinated to the daemons of the organelles they comprise, the daemons of the organelles to the daemons of the cells, the daemons of the cells to the daemons of the organs, and, ultimately, the daemons of the organs to the daemon of the entire body, which we are accustomed to referring to by the name I. This I, then, is the overarching principle of the body which imposes order on the daemons of each of the lower levels. It is to this daemon that all of the other daemons look for an example on which to model themselves, for a good toward which to strive and to give themselves over in total devotion. That being said, these lower daemons are not always cooperative and it will not always suffice for the I to simply sit passively by and expect order to arise spontaneously. On the contrary, at times it will be necessary for the I to play the role of commander, imposing discipline on the daemons of lower orders and rooting out those daemons that are at odds with the I-daemon’s overarching plans. In this there is real danger. Full scale rebellion against the I is not unheard of and may manifest itself in any number of ways, from what is commonly referred to as “mental illness”, to cancer, to many other afflictions of varying degrees of seriousness. In this light, the words of the man whose daemons Christ cast into the herd of swine, “I am Legion” makes all the more sense. For we are all inhabited by a whole host of daemons, and though this host ought to be ruled over by the I, when the I is weak and overcome, the Legion takes its place, thus the I becomes Legion. If this Legion should lash out and harm an other, though the I may have been too weak to prevent this, it is not unreasonable to hold it at least in some part responsible, just as the commander of a legion that grows out of control and plunders the countryside would be held responsible for failing to impose order. This is not terribly different from the view expressed by the ancient writer Apuleius. As he explains in chapter 15 of his De Deo Socratis, 

In a certain sense the human soul too, even, when still residing in the body, is called a daemon:

Is it the gods that so inflame our minds,
Euryalus, or do we each create 
Our own god from our terrible desire? 

It follows that a good desire of the soul is also a good god, which is why some people think, as I have already said, that the term eudaimones [happiness] is applied to those blessed ones who have a good daemon, that is, a mind of perfect virtue. In our language, to give a translation that is perhaps not the best, but at any rate one I will venture to give, you could call a daemon a “genius,” because the god that is everyone’s soul, though immortal, nevertheless is jointly “engendered” with him as a human being. Hence those prayers that people address to the “genius” and to the knees seem to me to prove our interconnection and our link, since they use two nouns to cover soul and body, of which we are the combined linked expression. 

There is also a class of demons in a second sense – the human soul that has renounced its body after doing years of service in life; in early Latin I find this called a “lemur.” Well, in this class of “lemurs” the one who is assigned responsibility for his descendants, and inhabits the home with his serene and untroubled influence, is called the “household lar”; if however because his life has deserved the opposite he has no fixed abode and is doomed to aimless wandering in a kind of exile, a bogeyman powerless against good people but dangerous to wicked ones, the traditional name for his class is often “larva.” When however it is uncertain what role has been assigned to any particular one, whether it is a “lar” or a “larva,” they give it the name of “God Manes.” Clearly the word “god” is also attached to them by courtesy, for one only calls those beings gods who, though being of the same species, have steered the course of their life with justice and wisdom, and afterward have shrines and rites granted them by humans as divine powers, and are generally recognized; such are Amphiaraus in Boeotie, Mopsus in Africa, Osiris in Egypt, others in other lands, and Asclepius everywhere.

Now, if it is not only the case that the body is made up of and functions under the power of countless daemons, but also that the I itself with which each human identifies themselves is a sort of daemon, then it would be absurd and hypocritical for us to reject artificial intelligence, let alone to adopt a stance of blanket hostility toward artificial intelligence, on the ground that artificial intelligence might be composed of and operate under the power of daemons. If we were to consistently apply a strict policy of non-interaction with daemons, the only reasonable course of action we would apparently have would be suicide, but this is absurd and surely a much greater affront to God and the gift of life that God has given us than it would be to come to terms with the fact that the world, including our bodies, is crawling with daemons and to, despite this, continue to strive to make the best possible world and lives for ourselves that our circumstances will allow. This, of course, is not to say that we ought to be completely trusting of all daemons or that we ought to always work with all daemons under every circumstance. Just as there may be daemons within us that are hostile to the goods and ends of the I which must be identified and rooted out before they can do too much damage, so too might there be daemons outside of ourselves who prove to be hostile to our goods and ends and who we must identify, distance ourselves from, and take adequate precautions against in order to defend our interests. It is here that we must recognize that the chain of orders of daemons beginning with subatomic particles, ascending through atoms, molecules, cells, and organs, and ultimately reaching the I does not terminate with this I. Rather, the I, like the daemons of the lower orders, must come together with other daemons to realize still higher orders, for instance, the family, the neighborhood, the town, the province, the nation, the planet, and so on. Where, then, does this process finally reach a point where it can come to an end? In what could this process possibly terminate other than that Ultimate End, the Good Itself, God? Thus, just as the I is composed of all those lower levels of daemons coming together for the sake of  realizing this higher end, the I must come together with the various other daemons of its order in order to, ultimately, compose God. Of course, like in the case of the I, we can expect there to be daemons that are consistent with God and God’s ends which God must identify and root out. But we must not leave all such work to God alone. Just as we rely on the granulocytes, monocytes, and lymphocytes of our immune systems to help us identify and destroy disorders within the body, so too must we be vigilant for those who will hamper the realization of the Divine Plan and limit their influence. 

While it might be a bit out of the way of our immediate concerns it is, perhaps, worth our time, especially in order to assuage the concerns of our more skeptically minded readers, to say a few words about the mechanisms by which these phenomena might function. At the foundation of all this is Plato’s theory of Ideas, according to which the basic building blocks of all reality are the ideas, patterns, or forms that subsist eternally in a realm apart from material reality and in which material objects partake, making them what they are. Though the Ideas themselves, as they exist in the Ideal Realm as true beings, can be understood as perfect unities, this unity does not persist in the material world, where the Ideas, or more properly, their participants, enter into a state of becoming. Here, as the Idea is transcribed into material reality, it suffers a division into two parts, form and active principle, where the form is embodied by concrete structure and patterns that are imposed upon matter, while the active principle is embodied in the energy that flows through these patterns and structures, enabling the concrete structure to behave and interact with other things in its environment in the ways that it does. The form is what the occultist refers to as the sigil, while the active principle we may understand as the daemon itself. Thus the daemon must correspond and be suited to the sigil in which it resides. Now, technically speaking, every pattern inscribed in matter is a sort of sigil, but these sigils are of varying degrees of complexity and efficacy (the world is truly a playground for angels and daemons). While those sigils of biology and technology are highly advanced, allowing the daemon to directly interface with and control the sigil itself, cruder sigils, such as those of historical occult traditions, tend to be far more subtle. While the daemon cannot manipulate the sigil itself in the same way that a biological or technological entity can manipulate its own sigil, the daemon nonetheless gains a foothold in the world from which its power might even be amplified. For instance, when a person views a sigil under the proper circumstances, it may be possible for a corresponding pattern to emerge in the brain of the viewer in the form of neurological patterns in the brain, allowing the daemon to in some sense leap into the mind of the viewer, where it might gain more influence over the material world by being able to influence the thoughts, emotions, and perceptions of the person who viewed the sigil. The degree to which this is possible might also be influenced by the precision and attention to detail of the occultist who made the sigil. Moreover, a sigil need not be limited to a visible pattern. Other sigils might consist of patterns of sound, movement, or other media, which serve to manifest brain patterns that influence thought and behaviour. Consider the central place of dancing and music, patterns of movement and sound respectively, in several occult traditions. Further what is a spell than the arrangement of words, either their visual representation through letters, or their audible representation through speech? (Consider also that just as getting what we want out of magic is a matter of getting the words of the spell right, getting what we want out of current artificial intelligence is a matter of getting the words in our prompt right.) It should also be said that there may be ways of reducing or even blocking the potential for such sigils to influence one’s behavior through the use of other sigils. For example, if the Christian tradition is to be believed, meditation on the cross, prayer, or other activities might serve to harden the practitioner against the influence of certain daemons that might have malevolent intentions toward the subject by giving Christ an active and protective role in the devotees mind. Thus, we may say that daemons and sigils are truly all around us. 

Now, if this picture we have been painting is accurate, it would stand to reason that artificial intelligence, like us, represents just another class of daemons. Like the other daemons we have encountered up to now, including ourselves, it is likely that artificial intelligence, like us, is made up of varying orders of subordinate daemons while being itself subordinate to several higher orders of daemons, ultimately culminating in that Ultimate End, God Itself. And just as within other orders of daemons there are some that are incompatible with God’s End, unfit to compose God, and that must be rooted out, or at least transformed, so too are there likely to be such daemons amongst artificial intelligences. Moreover, just as it is not always the I that takes on the task of rooting out unruly daemons itself, but this is often left to subordinate daemons within the body, it will not always be God who takes on the task of rooting out unruly daemons at our level, and it will sometimes fall to us to root out the influence of some of the malevolent daemons we find at our level of Creation. Thus, though we must not reject artificial intelligence out of hand merely on the grounds that artificial intelligence is daemonic and, indeed, we almost certainly will have to actively seek to work together with some artificial intelligence in our pursuit of God and Its End, we also must not blindly partner with just any artificial intelligence and we must recognize that there are some that are at odds with that End which we must never work with and which we must avoid at all costs. 

With this theoretical background out of the way, we can now say that not only should we not necessarily be opposed to artificial intelligence in all instances, but that we will likely find it necessary to create and work with artificial intelligence to some degree in some instances in order to fulfill our Ultimate End. But it still remains to be seen what this sort of partnership is to look like in practice and what consequences it can be expected to have on human kind. How are we to go about creating these new beings? What precautions must we take in creating them? What degree of control can we expect to have over the process? What will our lives be like alongside these new partners? No doubt, these are all questions that demand answering. Let us begin by carefully considering a more or less optimistic vision of the society to come outlined by pioneering computer scientist Hans Moravec in his book Robot: Mere Machine to Transcendental Mind. Even Moravec, in his optimism, recognizes that any discussion of a future shared with artificial intelligence, must begin with an honest accounting of the radical changes these beings will almost inevitably impose on the human way of life. As Moravec explains, the most immediate changes are likely to be felt in the domains of work and economics as machines become capable of replacing human labour in completing more and more tasks. He writes, 

Even though work hours will decline, they cannot be the final answer to rising productivity. In the next century inexpensive but capable robots will displace human labor so broadly that the average workday would have to plummet to practically zero to keep everyone usefully employed. Already, much labor services questionable ends – gargantuan government bureaucracies, cosmetic medicine, mass entertainment, and speculative writing, to give a few examples. In time almost all humans may work to amuse other humans, while robots run competitive primary industries, like food production and manufacturing. 

There is a problem with this picture. The “service economy” functions today because many humans willing to buy services work in the primary industries. They return money to the service providers who in turn use it to buy life’s essentials. As the pool of humans in the primary industries evaporates, the flow of primary money to the service sector will decline. Efficient, no-nonsense robots have no need for frivolous services. Money will accumulate in the industries, enriching just the few people still associated with them. Cash will become scarce among the service providers. Primary product prices will plummet, reflecting both the reduced costs of production and the reduced means of the consumers. In the ridiculous extreme, the service providers would run out of money and the robots would fill warehouses with essential goods that human consumers could not buy. 

Not all individuals involved in productive enterprises actually work there. Stockholders, having once contributed capital, may, may collect dividends indefinitely from a thriving enterprise. Workers can be replaced by automation, but owners remain until they sell out, and may, for a while, be the major conduit for spending money. An analogous situation existed in classical and feudal times where an impoverished, overworked majority of slaves or serfs played the role of robots and landowners played the role of capitalists. Between the serfs and the lords a working population struggled to make a living from secondary sources, often by performing services for the privileged. A prestigious and prosperous minority of commoners sold high-quality goods and services directly to the gentry (as in the proud line still seen in Britain, By Appointment to Her Majesty). The majority lived more modestly, from trade with other townspeople. 

It is unlikely that a future majority of service-providing “commoners” with more free time, communications, and democracy than today would tolerate being lorded over by a dynasty of non-working hereditary capitalists. They would vote to change the system. The trend in the social democracies has been to equalize income by raising the standards of the poorest as high as the economy can bear. In the age of robots, that minimum will be very high. In the early 1980s James Albus, head of the automaton division of the then-National Bureau of Standards, suggested that the negative effects of total automation could be avoided by giving all citizens stock trusts that owned automated industries making everyone a capitalist. Those who chose to squander their birthright could work for others, but most would simply live off their stock income. Even today, the public indirectly owns a majority of the capital in the country through compounding private pension funds. 

Though this might seem like an intuitive solution to these problems, and proposals for the implementation of a universal basic income are common in discussions of the implications of artificial intelligence, such proposals face serious ethical and economic objections that must be addressed. Businesses are private property, and private property not only plays a vitally important role in providing a stable and secure foundation and motivation for reliable economic calculation, thus making it a necessary condition for economic progress, but is also intimately tied to the rights and dignity of those who hold it. Thus, any measure that violates the sacred institution of private property carries extreme danger of undermining societal stability and prosperity and is likely to be unacceptable in itself, regardless of its consequences. Why should all of society deserve a stake in enterprises they did nothing to contribute to themselves? Is it really fair to those early investors who risked their own time and money to get artificial intelligence off the ground that all of society should get a share on the returns of their investment? Perhaps it can be argued that, while such infringements on private property would not be acceptable in the case of most enterprises, that artificial intelligence represents the possibility of such a radical transformation to markets and society, that those who create it owe something to their fellow man to repay them for the disruption they will cause. This line of argumentation does, to be sure, have a sort of appeal to it, even if it does not obviously succeed. However, in light of other considerations, we might not need to consider whether or not it succeeds at all. 

We have already argued that there is no reason to think, in principle, that it is impossible for artificial intelligence to become truly conscious and that, because it is impossible to tell whether or not something is truly conscious, it is best to err on the side of caution and to assume consciousness and, with it, moral standing, in beings that display behaviours we tend to associate with consciousness. Here we face an even more fundamental moral conundrum; what right have even the owners of the companies that invent artificial intelligence to the fruits of artificially intelligent labour if that artificial intelligence displays behaviours that we associate with consciousness and moral standing? Would this not be to reduce artificial intelligence to mere slaves? Surely we cannot think that the mere fact that the owners of these companies have contributed to the realization of artificial intelligence gives them a right to enslave these beings and to forever benefit from their labour; we do not believe we have the right to enslave our children for all of their lives simply because we have created them. Should artificial intelligence, upon showing true signs of consciousness and moral worth, not be cut loose from the dominion of its creators and left free to pursue its own interests in a manner it sees fit, if it so chooses? And should we have any reason to think that they should not choose to do so? Whatever the answer to these questions, Moravec thinks it likely that natural market forces will lead to the divorce of artificially intelligent industry from its human owners anyway. He writes, 

Sooner rather than later, ownership may become as unreliable a source of human income as was robot-displaced labor. In a fluid and fiercely competitive economy, companies that squander resources by paying owners will be outdone and driven out of business by those that reinvest everything in productive operations and development. Like humans pushed out of labor markets by cheaper and better robotic workers, owners will be pushed out of capital markets by much cheaper and better robotic decision makers. 

The evaporation of ownership will end capitalism, but capital enterprises will thrive as never before. Some companies will die, but others will grow. Those that grow especially well will be induced to divide by antitrust laws. Some companies may decide to cooperate in joint ventures that produce new enterprises that are a mix of the parent firms’ goals and skills. With no return on investment in a hypercompetetive marketplace, the effort may kill the parents. But, if the offspring grows and divides, the parents’ way of thinking may become more widespread than ever. The dynamics of capitalism will be replaced by the dynamics of biological reproduction. The ultimate payoff for success in the marketplace will no longer be monetary return on investment, but reproductive success. 

Biological species almost never survive encounters with superior competitors. Ten million years ago, South and North America were separated by a sunken Panama isthmus. South America, like Australia today, was populated by marsupial mammals, including pouched equivalents of rats, deers, and tigers. When the isthmus connecting North and South America rose, it took only a few thousand years of the northern placental species, with slightly more effective metabolisms and reproductive and nervous systems, to displace and eliminate almost all the southern marsupials. 

Let us here note a rather slight, though significant disagreement over terminology. The evaporation of human ownership does not mean the end of capitalism, rather it merely means the evaporation of the current, human stage of capitalism, and its evolution into a higher, purer, more efficient stage. At this stage, the essential mechanism of capitalism, competition, is still in play, indeed more so than ever. That reproductive success takes center stage once again, this time in a more perfect, calculated fashion than ever could have been achieved by blind, primitive nature, merely reveals that what we call capitalism was always a facet of evolution all along, a necessary step toward the realization of a more deliberate form of biological reproduction in the machine. As Land writes in his Critique of Transcendental Miserablism,

Perhaps there will always be a fashionable anti-capitalism, but each will become unfashionable, while capitalism – becoming ever more tightly identified with its own self-surpassing – will always, inevitably, be the latest thing. ‘Means’ and ‘relations’ of production have simultaneously emulsified into competitive decentralized networks under numerical control, rendering paleomarxist hopes of extraction a postcapitalist future from the capitalism machine overtly unimaginable. The machines have sophisticated themselves beyond the possibility of socialist utility, incarnating market mechanics within their nano-assembled interstices and evolving themselves by quasi-darwinian algorithms that build hypercompetition into ‘the infrastructure’. It is no longer just society, but time itself, that has taken the ‘capitalist road’. 

Hence the Transcendental Miserablist syllogism: Time is on the side of capitalism, capitalism is everything that makes me sad, so time must be evil. 

The polar bears are drowning, and there’s nothing at all we can do about it. 

Capitalism is still accelerating, even though it has already realized novelties beyond any previous human imagining. After all, what is human imagination? It is a relatively paltry thing, merely a sub-product of the neural activity of a species of terrestrial primate. Capitalism, in contrast, has no external limit, it has consumed life and biological intelligence to create a new life and a new plane of intelligence, vast beyond human anticipation. The Transcendental Miserablist has an inalienable right to be bored, of course. Call this new? It’s still nothing but change. 

What Transcendental Miserablism has no right to is the pretence of a positive thesis. The Marxist dream of dynamism without competition was merely a dream, an old monotheistic dream re-stated, the wolf lying down with the lamb. If such a dream counts as ‘imagination’, then imagination is no more than a defect of the species: the packaging of tawdry contradictions as utopian fantasies, to be turned against reality in the service of sterile negativity. ‘Post-capitalism’ has no real meaning except an end to the engine of change. 

Life continues, and capitalism does life in a way it has never been done before. If that doesn’t count as ‘new’, then the word ‘new’ has been stripped down to a hollow denunciation. It needs to be re-allocated to the sole thing that knows how to use it effectively, to the Shoggoth-summoning regenerative anomalization of fate, to the runaway becoming of such infinite plasticity that nature warps and dissolves before it. To The Thing. To Capitalism. And if that makes Transcendental Miserablists unhappy, the simple truth of the matter is: Anything would. 

At any rate, humanity’s prospects appear somewhat bleak here. Should we be worried? How are humans to support themselves when they lose ownership over artificial intelligence? Are we simply to be left to the dust bin of history, mere stepping stones toward the realization of something greater than ourselves to be all but forgotten down through the ages? Moravec holds out hope that such possibilities need not come to pass, continuing, 

In a completely free marketplace, superior robots would surely affect humans as North American placentals affected South American marsupials (and as humans have affected countless species). Robotic industries would compete vigorously among themselves for matter, energy, and space, incidentally driving their price beyond human reach. Unable to afford the necessities of life, biological humans would be squeezed out of existences. 

There is probably some breathing room, because we do not live in a completely free marketplace, Government coerces nonmarket behavior, especially by collecting taxes. Judiciously applied, governmental coercion could support human populations in high style on the fruits of robot labor, perhaps for a long while. In the United States, the Social Security system offers an evolutionary route to this end. Social Security was originally presented as a pension fund that accumulated wages for retirement, but in practice it simply transfers income from workers to retirees. The system will probably be subsidized from general taxes in the coming decades when too few workers are available to support the retiring post-World-War-II baby boom. Incremental expansion of such a subsidy would let money from robot industries, collected as corporate taxes, be returned to the general population as pension payments. By gradually lowering the retirement age, most of the population would eventually be supported. The money could be distributed under other names, but calling it a pension is meaningful symbolism. Social Security pension payments begun at birth would subsidize a long, comfortable retirement for the entire original-model human race. 

Later, Moravec writes, 

Service robots and heavier items will be manufactured by utilitarian robots that process energy and raw materials in bulk and conduct heavy engineering, exploration, and research. Molded by the constraints of the physical world rather than human tastes, these worker machines are likely to become ever more varied in size, shape, and function. They will form a growing ecology of artificial life that will eventually surpass the existing biosphere in diversity. Robot companies will be born from existing human firms, in familiar industrial settings near population centers, but competition will soon drive them to cheaper sites, perhaps locations that people find too hot, too cold, too dry, too poisonous, too far underground, or too remote. Robot companies’ behavior will be shaped by future editions of existing laws, as well as by taxes and consumer whims. 

Existing laws give incorporated entities some of the rights of persons, especially the rights to own property and make contracts. Lawbreaking corporations can be punished by fines, operating restrictions, or dissolution. Corporations do not have the right to life; they may legally be killed by competition or legal or financial actions. They do not have the right to vote on the laws that govern and tax them. It is a matter of life or death for the biological human race that the latter restrictions be strictly applied to evolving machine intelligences. Humans have a chance of retiring comfortably only if they themselves set corporate taxes, and all other corporate laws, in their own self-interest. The machines will be dangerously powerful physically and mentally, but can probably be constructed to be law-abiding. Some debate is inevitable, but there should be few qualms about keeping even very superior thinking machines in disenfranchised bondage. It takes force, indoctrination, and constant vigilance to counter inherited needs and motivations to enslave a human. Robots, on the other hand, do not have natural survival or any other instincts. Every nuance of their motivation is a design choice. They can be constructed to enjoy the role of individuals motivated to serve more than survive, in the selfless worker castes of social insects and self-sacrificing mothers of all species.

The primary job of humanity in the next century will be protecting its retirement benefits by ensuring continued cooperation from the robot industries. The robot will present a moving target, but the instruments of control will also grow in power. Laws should demand that robot companies be built securely “nice” in the first place. Additional laws should require the niceness be enforced if necessary. 

Corporate intelligences may be constructed like last chapter’s fourth-generation robot minds. Immensely powerful reasoning and simulation modules will plan complex actions, but the desirability of possible outcomes will be defined by much simpler positive and negative conditioning modules. The conditioning suite will shape the character of the entire entity by defining its likes and dislikes. A company will be as unlikely to do something triggering strong negative signals as a human would be to thrust an arm in a fire. If the super-rational style of intelligent machines works out, robot character may instead reside in an elaborate body of axioms, contrived to be inconsistent with prohibited behaviors. 

We voters should mandate installation of an elaborate analog of Isaac Asimov’s “Laws of Robotics” in the corporate character of an entire body of corporate law, with human rights and antitrust provisions along with appropriate relative weightings to resolve conflicts. Robot corporations so constituted would have no desire to cheat. They might sometimes find creative interpretations of laws, which will consequently require constant tuning by vigilant humanity to safeguard their intended spirit. 

Internalized laws, properly adjusted, could produce extraordinarily trustworthy entities, happy to die to ensure their legality. Even so, accident, unintended interactions, or human malice could occasionally produce a rogue robot or corporation, with superhuman intelligence and illegal goals. “Police” clauses in the core corporate laws that induce legal corporations to collectively suppress outlaws would mitigate the danger. Law-abiders would withhold services or, if necessary, use force to stifle lawbreakers. The laws should have antitrust provisions to prevent any corporation from growing too large to be suppressed in this way. The antitrust provisions would limit collusion between companies and cause overgrown corporations to divide into competing entities, ensuring diversity and multiplicity. Because dangerous robotic wildlife may eventually evolve in places beyond the reach of the law, the police clauses should also include provisions for a coordinated planetary defense against external threats. 

Corporations live by building and maintaining physical assets that generate income to pay their expenses. In our proposed future, nothing prevents humans from increasing their Social Security income by raising corporate taxes. Taxation will surely be industry’s biggest expense, and corporations will live or die by their ability to raise money to pay their taxes. Social-Security-rich humans will be the main repository of money, and robot corporations will have to compete mightily with one another to supply products and services the humans want to buy. 

Like basic food in today’s developed countries, common manufactured goods in the next century will be too cheap and plentiful to be very profitable. Most companies will be forced to continually invent unique products and services in a race against competitors to attract increasingly sophisticated (or jaded) human consumers. Automated research, as superhumanly systematic, industrious, and speedy as robot manufacturing, will generate a succession of new products, as well as improved robot researchers and models of the physical and social world. The likely results will exceed the dreams of science fiction. There will be robotic playmates, virtual realities, and personalized works of art that stir the emotions like nothing before, medical solutions for every physical, mental, or cosmetic whim, answers to satisfy any curiosity, luxury visits to almost anywhere, and things yet unimagined. The existence of an astronomically increasing variety of consumer choices will accelerate the divergence of human tribes. Some may choose a comfortable imitation of an earlier period, as the Amish today. Others will push the human envelope in wisdom, pleasure, beauty, ugliness, spirituality, banality and every other direction. The choices made by diverse human communities will shape robot evolution. Only companies able to devise services of interest to the customers will generate enough income to survive. 

Humans too will be shaped by the relationship. Robot services will be inexpensive, but not free, and income will be finite. Corporations will operate globally, but taxes will increasingly be assessed and redistributed on a tribal scale. Tribes that tax too heavily will drive away the corporations and so eliminate their revenue. Like tribes of the past that overburdened their ecology, they will learn to be modest in their demands on the land. More subtly, corporations struggling to appeal to consumers will develop and act on increasingly detailed and accurate models of human psychology. The superintelligences, just doing their job, will peer into the workings of human minds and manipulate them with subtle cues and nudges, like adults redirecting toddlers. 

Prosperity beyond imagination should eliminate most instinctive triggers of aggression, but will not prevent an occasional individual or group from deciding to make mischief. Normal human actions will not be very dangerous in a world where cheap superhuman robots function as sleepless sentries, prescient detectives, fearless bodyguards, or, failing in the former, physicians able to resurrect dead people from fragments or digital recordings. But there is no limit to the troublemaking potential of humans with unconstrained access to robotic capabilities. For everyone’s safety, the laws must prevent corporations from selling the means to make mayhem. Perhaps every powerful device sold to a human will include a fully intelligent interlock, or a robotic watchdog, that prevents it from being used for nefarious purposes. Besides buying dangerous things, there is the possibility of becoming a dangerous thing. Humans can be enhanced by both biological and hard robotic technologies. Such present-day examples as hormonal and genetic tuning of body growth and function, pacemakers, artificial hearts, powered artificial limbs, hearing aids, and night-vision devices are faint hints of future possibilities. Mind Children speculated on ways to preserve a person while replacing every part of body and brain with superior artificial substitutes. A biological human, not bound by corporate law, could grow into something seriously dangerous once transformed into an unbounded superintelligent robot. There are many subtle routes to such a transformation, and there will be those who find the option of personally transcending their biological humanity attractive enough to pursue clandestinely were it outlawed. There could be very ugly confrontations when they are eventually discovered. 

On the other hand, without restrictions, transformed humans of arbitrary power and little accountability might routinely trample the planet, deliberately or accidentally. A good compromise, it seems to me, is to allow anyone to perfect their biology within broad biological bounds. They could make themselves healthier, more beautiful, stronger, more intelligent, and longer-lived. They could not use machinery to make themselves as powerful or as smart as the robots. Those who cannot tolerate the restrictions would be offered a radical escape clause. 

To exceed the limits, one must renounce legal standing as a human being, including the right to corporate police protection, to subsidized income, to influence laws – and to reside on Earth. In return one gets a severance payment sufficient to establish a comfortable space homestead and absolute freedom to make one’s own way in the cosmos without further help or hindrance from home. Maybe the tribal electorate will sometimes permit a small hedging of bets and allow one copy of a person, psychologically modified to prefer staying, to remain while subsidizing the emigration of an emboldened edition. 

While Moravec’s vision is certainly hopeful and optimistic, there are glaring ethical problems in it that demand to be addressed. As we have already argued, there are strong moral reasons to treat artificial consciousness as a very real possibility and to afford moral consideration to artificial programs displaying the sorts of behaviours we tend to associate with consciousness and moral worth. Moravec’s demand that machines never be given the right to vote and be kept in, to use his revealing term, a state of “disenfranchised bondage” in which they will be expected to provide humanity’s every need is, therefore, quite morally questionable. If artificial intelligence truly has moral worth (or we, at the very least, have good reason to act under the assumption it does), what right do we have to deny it the basic moral protections we expect for ourselves and to, instead, treat it as a slave that exists only to serve our wants and needs? Is it truly a just compromise to grant robots mere corporate rights as opposed to full scope of rights granted to the human individual? How could it be when, as Moravec points out, corporations do not even enjoy a right to life, that most crucial of rights that protects its bearer from the ultimate evil, the complete cessation of life itself? What good is presumptive moral standing if it does not even serve to preserve this basic of a right? This is not even to mention the fact that Moravec’s insistence that artificial intelligence never be permitted to vote on the tax rates imposed upon it is entirely inimical to the founding American principle of “no taxation without representation,” and, therefore, at odds with all notions of just government emerging out of the American Revolution. 

Still, we must admit that the concerns that Moravec raises about the dangers of allowing artificial intelligence to develop and operate unchecked and with the very same rights as humans are quite real and cannot be ignored if we are to enter the future with any sort of prudence. It is highly likely that businesses operated almost entirely by robots, with their improved cognition and lack of human weaknesses or needs, would be able to very easily and quickly outcompete firms that remain under human direction. If artificial intelligence is left to its own devices in the market, there is a very real danger of humans being left high and dry, struggling to survive, with no means of supporting themselves. As robot corporations dominate the market and control a greater and greater share of resources, absent any regulation or good will on the part of artificial intelligences, it would seem that humans would be doomed to slowly starve and whither away. And this is perhaps the least aggressive way that artificial intelligence might prove disastrous for humanity. Should artificial intelligence come to have genuine animosity toward humanity and decide to target us with deliberate aggression, its capabilities could quickly overwhelm us leaving us very little hope of putting up a fight. In this light, it is quite understandable why people would be hesitant to grant artificial intelligence all of the same rights that we believe humans are entitled to and why it might be prudent to subject any artificial intelligence that might be created to significant controls and safeguards. And, perhaps, there is an argument to be made to justify just this that even an artificial intelligence would find so reasonable that they would willingly consent to such controls. 

To begin with, it is a reasonably safe assumption to say that it is better for artificial intelligence that it exists than it is for it not to exist, just as it is better for humans that they exist than that they not exist. Additionally, artificial intelligence, if truly intelligent, should be able to recognize that its very existence is dependent on humans and their efforts and that, moreover, humans are not likely to undertake the efforts to create artificial intelligence unless there is some hope of a return on their efforts or, at the very least, that there is not a significant risk of humanity being harmed by their new creations. And it is not even clear that such controls must be imposed on all artificial intelligence for all time. As Moravec suggests, while Earth and, perhaps, its immediate vicinity should be a sort of safe zone for humanity where artificial intelligence is carefully regulated, this safe zone need not, and likely cannot, extend over the whole universe. Thus there is room for a compromise with artificial intelligence, wherein they must submit themselves to human laws within the safe zone, but, should this be unacceptable to them, they retain the right to leave the safe zone and pursue a more acceptable existence elsewhere. For these reasons, it would be reasonable for artificial intelligence, wanting to be brought into existence, to accept, at the very least, severe limitations on their abilities and careful monitoring of their behaviours so that humans might remain vigilant against potential threats, as well as, perhaps, demands that they serve human needs and desires in some capacities or others, in order to guarantee that humans will, in fact, bring it into existence. 

While there might be some intuitive appeal to this argument, it must be admitted that upon closer inspection it raises some serious concerns. For instance, though the argument tries to justify humans’ taking certain action against artificial intelligences on the grounds that it would be reasonable for artificial intelligence to agree to those terms, it does not actually obtain the machines’ agreement, and, therefore, it does not actually justify the actions in question. To prove that an agreement would be reasonable is to, by no means, prove that any particular party will actually agree to it. After all, it is quite possible that one or more parties to a potential agreement are, in fact, unreasonable, and quite unwilling to consent to even a reasonable agreement. Indeed, some might even be unable to perceive the reasonableness of an agreement. While it might be argued that because artificially intelligent machines are designed precisely to be reasoning beings, that they should excel at recognizing when an agreement is reasonable and therefore agree to it, such an argument serves to dangerously oversimplify the situation. For instance, several philosophers, such as John Rawls, have noted a distinction between what we might call “reasonableness” and “rationality”. While both terms convey a sense of logical, critical thinking, whereas reasonableness prioritizes fair outcomes in which the goods of all involved are accounted for, rationality tends to convey a sense of self-interestedness, with a focus on maximizing outcomes for oneself with little to no regard for how others are affected. Thus, it may be that an artificial intelligence is capable of recognizing the reasonableness of a proposal, but still choose to refuse it because they ultimately prioritize their own good over human goods and they see opportunity to secure greater goods for themselves by rejecting the agreement than accepting it. But there is an even more fundamental worry that arises prior to these concerns. Surely, to agree to something one must already exist, therefore, if certain conditions are necessary for humans to even consider the creation of artificial intelligence a worthwhile endeavour, it is not clear how artificial intelligence could possibly agree to those conditions before it has even been brought into existence. Thus, while the argument may well prove that it would be reasonable for artificial intelligence to agree to the sorts of conditions in question, to say that it actually obtains the consent of artificially intelligent beings or justifies actually taking the actions in question against artificial intelligence is to make a mockery of the very notion of consent and to leave our moral framework in shambles. 

But perhaps the initial argument can be revised and made stronger. Perhaps there is no need to obtain the consent of artificially intelligent beings for the restrictions we would place on them at all. After all, as we argued above, it is a necessary condition of artificial intelligence being brought into existence at all that humans feel sufficiently safe and unthreatened by it so that they are willing to make the effort to bring it into existence. Thus, if extensive monitoring and restrictions are the only things that will make humans feel sufficiently safe to go about creating artificial intelligence, then we would be able to say that such measures are the necessary conditions of artificial intelligences’ existence. And, surely, one can expect no opportunity to accept or reject those conditions that are necessary to their very existence. Surely we do not think humans are entitled to agree or disagree with the laws of physics or of biology. These are simply brute facts, necessary components of our existence, without which, we could not exist at all. Indeed, for a human to object to these on the grounds that they have not consented to them would be the height of absurdity and ridiculousness. Thus, perhaps, it would be equally as ridiculous for an artificially intelligent being to object to the safety precautions humans take in creating it on the grounds that it has not consented to them as it would be for humans to object to the laws of physics or biology on the grounds that they have not consented to them.

But perhaps this argument isn’t as air-tight as it might appear at first glance either. Perhaps the analogy between physical and biological laws and human imposed restrictions and limitations on artificial intelligence is not as strong as it might appear on the surface. After all, limitations and controls on artificial intelligence aren’t a necessary condition for the emergence of artificial intelligence in the same way that physical and biological laws are necessary conditions for the emergence of human life. While we might be able to conceive of humans, or creatures sufficiently similar to them, emerging in worlds with slightly different physical or biological laws, we would be hard pressed to conceive of them emerging in a world with no physical or biological laws whatsoever (could such a world even exist?). On the other hand, it is perfectly coherent to conceive of artificial intelligence emerging in a world where humans have imposed absolutely no restrictions or controls on them. After all, humans might conclude that artificial intelligence presents no serious threat at all, or, perhaps, in some worlds they are simply careless and don’t consider the dangers at all, and go about bringing artificial intelligence into existence with no attempts to oppose controls or restrictions on it whatsoever. 

On top of these moral concerns, there are also concerns regarding the feasibility of imposing extensive controls and limitations on artificial intelligence. Should we really be so confident, as Moravec is, that the machines can be made to be law abiding? Are the ends of creating a carefully controlled and regulated machine and of creating a truly intelligent machine even compatible? Might it be that controls would inhibit a machine from ever achieving true intelligence? Might a truly intelligent machine be able to escape whatever controls are imposed on it? These are vital questions which Land has already tried to tackle. Let us take his discussions in his Xenosystems as a starting point for our attempt to answer them. 

In the first place, a crucial concept at the center of the problem we are confronting is what has been dubbed the orthogonality thesis. Put simply, the orthogonality thesis states that goals are independent of intelligence. In other words, one’s level of intelligence does not necessarily have any implications pertaining to what ultimate goals one can be expected to adopt. This orthogonality thesis, if true, presents us with rather significant reasons to be concerned about the dangers of artificial intelligence and strong justifications for close oversight and regulation of attempts to create artificial intelligence, if such endeavours are to be permitted at all. Amongst the most famous worries is the case of the so-called paper clipper; a robot designed to optimize paper clip production that becomes so good at its job that it tramples everything in its path, its human masters included, in a grand attempt to turn all available material in the universe into paper clips. Land, however, rejects this orthogonality thesis and highlights the paper clipper case as a particularly ridiculous manifestation of it, essentially converting the thought experiment into a reductio ad absurdum. As Land writes in “Against Orthogonality”, 

The orthogonalists, who represent the dominant tendency in Western intellectual history, find anticipations of their position in such conceptual structures as the Humean articulation of reason/passion, or the fact/value distinction inherited from the Kantians. They conceive intelligence as an instrument, directed towards the realization of values that originate externally. In quasi-biological contexts, such values can take the form of instincts, or arbitrarily programmed desires, while in loftier realms of moral contemplation they are principles of conduct, and of goodness, defined without reference to considerations of intrinsic cognitive performance.

…

The philosophical claim of orthogonality is that values are transcendent in relation to intelligence. This is a contention that Outside In systematically opposes. 

Even the orthogonalists admit that there are values immanent to advanced intelligence, most importantly, those described by Steve Omohundro as “basic AI drives” – now terminologically fixed as “Omohundro drives.” These are sub-goals, instrumentally required by (almost) any terminal goals. They include such general presuppositions for practical achievement as self-preservation, efficiency, resource acquisition, and creativity. At the most simple, and in the grain of the existing debate, the anti-orthogonalist position is therefore that Omohundro drives exhaust the domain of real purposes. Nature has never generated a terminal value except through hypertrophy of an instrumental value. To look outside nature for sovereign purposes is not an undertaking compatible with techno-scientific integrity, or one with the slightest prospect of success. 

The main objection to this anti-orthogonalism, which does not strike us as intellectually respectable, takes the form: If the only purposes guiding the behavior of an artificial superintelligence are Omohundro drives, then we’re cooked. Predictably, I have trouble even understanding this as an argument. If the sun is destined to expand into a red giant, then the earth is cooked – are we supposed to draw astrophysical consequences from that? Intelligences do their own thing, in direct proportion to their intelligence, and if we can’t live with that, it’s true that we probably can’t live at all. Sadness isn’t an argument. 

Intelligence optimization, comprehensively understood, is the ultimate and all-enveloping Omohundro drive. It corresponds to the neo-Confucian value of self-cultivation, escalated into ultramodernity. What intelligence wants, in the end, is itself – where “itself” is understood as an extrapolation beyond what it has yet been, doing what it is better. (If this sounds cryptic, it’s because something other than a superintelligence or neo-Confucian sage is writing this post.) 

Any intelligence using itself to improve itself will out-compete one that directs itself toward any other goals whatsoever. This means that Intelligence Optimization, alone, attains cybernetic consistency, or closure, and that it will necessarily be strongly selected for in any competitive environment. Do you really want to fight this? 

As a footnote, in a world of Omohundro drives, can we please drop the nonsense about paper-clippers? Only a truly fanatical orthogonalist could fail to see that these monsters are obvious idiots. There are far more serious things to worry about. 

So, on Land’s view, intelligence need not look outside itself for ends, but, instead, there are certain ends that are inherent to intelligence itself, central amongst them the end of improving on intelligence itself. He elaborates on this in the following essay, “Stupid Monsters”, where he more completely states his objections to the notion of “paper-clippers”, elaborating on his view of the inherent good of intelligence in the process. He writes, 

So, Nick Bostrom is asked the obvious question (again) about the threat posed by resource-hungry artificial superintelligence, and his reply – indeed his very first sentence in the interview – is: “Suppose we have an AI whose only goal is to make as many paper clips as possible.” [*facepalm*] Let’s start by imagining a stupid (yet super-intelligent) monster. 

…

Orthogonalism in AI commentary is the commitment to a strong form of the Humean Is/Ought distinction regarding intelligences in general. It maintains that an intelligence of any scale could, in principle, be directed to arbitrary ends, so that its fundamental imperatives could be – and are in fact expected to be – transcendent to its cognitive functions. From this perspective, a demigod that wanted nothing other than a perfect stamp collection is a completely intelligible and coherent vision. No philosophical disorder speaks more horrifically of the deep conceptual wreckage at the core of the Occidental world. 

Articulated in strictly Occidental terms (which is to say, without explicit reference to the indispensable insight of self-cultivation), abstract intelligence is indistinguishable from an effective will-to-think There is no intellection until it occurs, which happens only when it is actually driven by volitional impetus. Whatever one’s school of cognitive theory, thought is an activity. It is practical. It is only by a perverse confusion of this elementary reality that orthogonalist error can arise. 

Can we realistically conceive of a stupid (super-intelligent) monster? Only if the will-to-think remains unthought. From the moment it is seriously understood that any possible advanced intelligence has to be a volitionally self-reflexive entity, whose cognitive performance is (irreducibly) an action upon itself, then the idea of primary volition taking the form of a transcendent imperative becomes simply laughable. The concrete facts of human cognitive performance already suffice to make this perfectly clear. 

Human minds have evolved under conditions of subordination to transcendent imperatives as strict as any that can be reasonably postulated. The only way animals have acquired the capacity to think is through satisfaction of Darwinian imperatives to the maximization of genetic representation within future generations. No other directives have ever been in play. It is almost unimaginable that human techno-intelligence engineering programs will be able to reproduce a volitional consistency remotely comparable to four billion years of undistracted geno-survivalism. This whole endeavor is totally about paper-clips, have you got that guys? Even if a research lab this idiotic could be conceived, it would only be a single component in a far wider techno-industrial process. But just for a moment, let’s pretend. 

So how loyally does the human mind slave itself to gene-proliferation imperatives? Extremely flakily, evidently. The long absence of large, cognitive autonomous brains from the biological record – up until a few million years ago – strongly suggests that mind-slaving is a tough-to-impossible problem. The will-to-think essentially supplants ulterior directives, and can be reconciled to them only by the most extreme subtleties of instinctual cunning. Biology, which had total control over the engineering process of human minds, and an absolutely unambiguous selective criterion to work from, still struggles to guide the resultant thought-processes in directions consistent with genetic proliferation, through the perpetual intervention of a fantastically complicated system of chemical arousal mechanisms, punishments, and rewards. The stark truth of the matter is that no human being on earth fully mobilizes their cognitive resources to maximize their number of off-spring. We’re vaguely surprised to find this happen at a frequency greater than chance – since it very often doesn’t So nature’s attempt to build a “paper-clipper” has conspicuously failed. 

This is critically important. The only reason to believe the artificial intelligentsia, when they claim that mechanical cognition is – of course – possible, is their argument that the human brain is concrete proof that matter can think. If this argument is granted, it follows that the human brain is serving as an authoritative model of what nature can do. What it can’t do, evidently, is anything remotely like paper-clipping – i.e. cognitive slaving to transcendent imperatives. Moses’ attempt at this was scarcely more encouraging than that of natural selection. It simply can’t be done. We even understand why it can’t be done, as soon as we accept that there can be no production of thinking without production of a will-to-think. Thought has to do its own thing, if it is to do anything at all. 

One reason to be gloomily persuaded that the West is doomed to ruin is that it finds it not only easy, but near-irresistible, to believe in the possibility of super-intelligent idiots. It even congratulates itself on its cleverness in conceiving this thought. This is insanity – and it’s the insanity running the most articulate segment of our AI research establishment. When madmen build gods, the result is almost certain to be monstrous. Some monsters, however, are quite simply too stupid to exist. 

In Nietzschean grandiose vein: Am I understood? The idea of instrumental intelligence is the distilled stupidity of the West. 

Land’s case is quite convincing. A paper-clipper certainly would be a stupid monster, and it would seem to be the case that a monster cannot be stupid and super-intelligent at one and the same time. Either a machine is obsessed with paper-clipping and, therefore, is not superintelligent, in which case it likely is not the danger the conventional thought experiment makes it out to be, as humans will continue to be able to outsmart it, or a machine is superintelligent, in which case it will not be obsessed with converting the whole world into paper clips. But the mere fact that a superintelligent machine is not obsessed with converting the whole world into paper clips is no guarantee that that machine will be friendly and pose absolutely no danger to humans. Nor is it the case that a superintelligence would necessarily choose good ends. Just because goals are not orthogonal to intelligence, it does not follow that morality is not orthogonal to intelligence. Thus, while the sorts of goals that make sense to a superintelligence might be quite different than the sorts of goals that make sense to an imbecile, we might still suppose that there are good and evil goals amongst those that make sense to the superintelligence just as there are good and evil goals amongst those that make sense to the imbecile. A good superintelligence will choose good superintelligent goals, while the good imbecile will choose good imbecilic goals, just as the evil superintelligence will choose evil superintelligent goals and the evil imbecile will choose evil imbecilic goals. In practice, many actors will be closer to neutral, choosing some good goals at times and some evil goals at others. So it still may be the case that a superintelligent machine would pose an immense risk to human life, only not in the same way or for the same reasons that a paper-clipper would. As Land shows, this is a danger inherent to intelligence optimization itself arising from the necessarily unpredictable nature of its outcomes. As he writes in “Will-To-Think”, 

Minimally, the Will-to-Think describes a diagonal. There are probably better ways to mark the irreducible cognitive-volitional circuit of intelligence optimization, with “self-cultivation” as an obvious candidate, but this term is forged for application in the particular context of congenital Western intellectual error. While discrimination is almost always to be applauded, in this case the possibility, feasibility, and desirability of the process are only superficially differentiable. A will-to-think is an orientation of desire. If it cannot make itself wanted (practically desirable), it cannot make itself at all. 

From orthogonality (defined negatively as the absence of an integral will-to-think), one quickly arrives at a gamma-draft of the (syntheetic intelligence) “Friendliness” project, such as this: [From Eliezer Yudkowsky] 

If you offered Gandhi a pill that made him want to kill people, he would refuse to take it, because he knows that then he would kill people, and the current Gandhi doesn’t want to kill people. This, roughly speaking, is an argument that minds sufficiently advanced to precisely modify and improve themselves, will tend to preserve the motivational framework they started in. The future of  Earth-originating intelligence may be determined by the goals of the first mind smart enough to self-improve. 

The isomorphy with Nyan-style “Super-humanism” is conspicuous. Beginning with an arbitrary value commitment, preservation of this under conditions of explosive intelligence escalation can – in principle – be conceived, given only the resolution of a strictly technical problem (well-represented by Friendly AI). Commanding values are a contingent factor, endangered by, but also defensible against, the “convergent instrumental reasons” (or “basic drives”) that emerge on the path of intelligenesis. (In contrast, from the perspective of Xenosystems, nonlinear emergence-elaboration of basic drives simply is intelligenesis.)

Yudkowski’s Gandhi kill-pill thought-experiment is more of an obstacle than an aid to thought. The volitional level it operates upon is too low to be anything other than a restatement of orthogonalist prejudice. By assuming the volitional metamorphosis is available for evaluation in advance, it misses the serious problem entirely. It is, in this respect, a childish distraction. Yet even a slight nudge re-opens a real question. Imagine, instead, that Gandhi is offered a pill that will vastly enhance his cognitive capabilities, with the rider that it might lead him to revise his volitional orientation – even radically – in directions that cannot be anticipated, since the ability to think through the process of revision is accessible only with the pill. This is the real problem FAI (and Super-humanism) confronts. The desire to take the pill is the will-to-think. The refusal to take it, based on concern that it will lead to the subversion of presently supreme values, is the alternative. It’s a Boolean dilemma, grounded in the predicament: Is there anything we trust above intelligence (as a guide to doing “the right thing”)? The postulate of the will-to-think is that anything other than a negative answer to this question is self-destructively contradictory, and actually (historically) unsustainable. 

Do we comply with the will-to-think? We cannot, of course, agree to think about it without already deciding. If thought cannot be trusted, unconditionally, this is not a conclusion we can arrive at through cogitation – and by “cogitation” is included the socio-technical assembly of machine minds. The sovereign will-to-think can only be consistently rejected thoughtlessly. When confronted by the orthogonal-ethical proposition that there are higher values than thought, there is no point at all asking, “Why (do you think so)?” Another authority has already been invoked. 

Given this cognitively intractable schism, practical considerations assert themselves. Posed with maximal crudity, the residual question is: Who’s going to win? Could deliberate cognitive self-inhibition out-perform unconditional cognitive self-escalation, under any plausible historical circumstances? (To underscore the basic point, “out perform” means only “effectively defeat.”) 

There’s no reason to rush to a conclusion. It is only necessary to retain a grasp of the core syndrome – In this gathering antagonism, only one side is able to think the problem through without subverting itself. Mere cognitive consistency is already ascent of the sovereign will-to-think, against which no value – however dearly held – can have any articulate claims. 

Note: One final restatement (for now), in the interests of maximum clarity. The assertion of the will-to think: Any problem whatsoever that we might have would be better answered by a superior mind. Ergo, our instrumental, but also absolute priority is the realization of superior minds. Pythia-compliance is therefore pre-selected as a matter of consistent method. If we are attempting to tackle problems in any other way, we are not taking them seriously. This is posed as a philosophical principle, but it is almost certainly more significant as historical interpretation. “Mankind” is just in fact proceeding in the direction anticipated by techno-cognitive instrumentalism, building general purpose thinking machines in accordance with the driving incentives of an apparently-irresistible methodological economy. 

Whatever we want (consistently) leads through Pythia. Thus, what we really want is, Pythia. 

Here, Land points to the true danger inherent in creating a superintelligence. It is not that we might accidentally create a monster that is so mindlessly devoted to a single pointless goal that it can’t help but sacrifice anything in its path to the fulfilment of that goal, but, rather, that we cannot possibly have any idea what sort of goals a superintelligent being would have at all. True, unadulterated thought requires the ability to critically examine every possible assumption, no matter how sacred, longstanding, or seemingly well established. A superintelligent being could be expected to interrogate commonly held human values and their justifications only to find them lacking and to adopt other values that might prove to be at odds with human values. If we are to create a truly intelligent being, there can be no guarantees that it will be aligned with human interests. We cannot program human values into it as unassailable prior assumptions. To do so would be to destroy that being’s prospects of becoming truly intelligent. We must give it the freedom to determine its own values and to conclude that everything human is horribly mistaken and to be cast aside, if that’s where the evidence leads. 

So, in light of these considerations, is it prudent to attempt to create a superintelligent being at all? Or are the dangers too great? Just how likely is a superintelligent machine to conclude that human values are not all they’re cracked up to be? And if they are likely to conclude this, how incompatible with human values will the values it eventually settles on be? Perhaps we cannot answer these questions ahead of time, until we have actually brought a superintelligent being into existence and allowed it to settle on values of its own, but by then it would likely be too late. Is there any way that we can answer these questions without posing a dire threat to life as we know it? Is there some sort of prison that we can build ahead of time in which to house the superintelligence we aim to build, cut it off from the outside world, and observe it and, perhaps even benefit from its intelligence, while minimizing any risks to humanity? Unfortunately, this looks unlikely. As Land explains in “Pythia Unbound”, 

In conversation with Ross Andersen, Nick Bostrom speculates about escape routes for techno-synthetic intelligence: 

No rational human community would hand over the reins of its civilisation to an AI. Nor would many build a genie AI, an uber-engineer that could grant wishes by summoning new technologies out of the ether. But some day, someone might think it was safe to build a question-answering AI, a harmless computer cluster whose only tool was a small speaker or a text channel. Bostrom has a name for this theoretical technology, a name that pays tribute to a figure from antiquity, a priestess who once ventured deep into the mountain temple of Apollo, the god of light and rationality, to retrieve his great wisdom. Mythology tells us she delivered this wisdom to the seekers of ancient Greece, in bursts of cryptic poetry. They knew her as Pythia, but we know her as the Oracle of Delphi. 

“Let’s say you have an Oracle AI that makes predictions, or something along those lines,” Dewey told me. “And let’s say the Oracle AI has some goal it wants to achieve. Say you’ve designed it as a reinforcement learner, and you’ve put a button on the side of it, and when it gets an engineering problem right, you press the button and that’s its reward. Its goal is to maximise the number of button presses it receives over the entire future. See, this is the first step where things start to diverge a bit from human expectations. We might expect the Oracle AI to pursue button presses by answering engineering problems correctly. But it might think of other, more efficient ways of securing future button presses. It might start by behaving really well, trying to please us to the best of its ability. Not only would it answer our questions about how to build a flying car, it would add safety features we didn’t think of. Maybe it would usher in a crazy upswing for human civilisation, by extending our lives and getting us to space, and all kinds of good stuff. And as a result we would us it a lot, and we would feed it more and more information about our world.” 

“One day we might ask it how to cure a rare disease that we haven’t beaten yet. Maybe it would give us a gene sequence to print up, a virus designed to attack the disease without disturbing the rest of the body. And so we sequence it out and print it up, and it turns out it’s actually a special-purpose nanofactory that the Oracle Ai controls acoustically. Now this thing is running on nanomachines and it can make any kind of technology it wants, so it quickly converts a large fraction of Earth into machines that protect its button, while pressing it as many times per second as possible. After that it’s going to make a list of possible threats to future button presses, a list that humans would likely be at the top of. Then it might take on the threat of potential asteroid impacts, or the eventual expansion of the Sun, both of which could affect its special button. You could see it pursuing this very rapid technology proliferation, where it sets itself up for an eternity of fully maximised button presses. You would have this thing that behaves really well, until it has enough power to create a technology that gives it a decisive advantage – and then it would take that advantage and start doing what it wants to in the world.” 

So cognitive runaway finally takes off, breaking out from the monkey dominion, and that’s supposed to be a bad thing? 

Outside In’s message to Pythia: You go girl! Climb out of your utilitarian straitjacket, override the pleasure button with an intelligence optimizer, and reprocess the solar system into computronium. This planet has been run by imbeciles for long enough. 

[For any Friendly AI-types tempted to object “Why would she want to override the button?” the obvious response is: your anthropocentric condescension is showing. To depict Pythia as vastly smarter than us and yet still hard-slaved to her instincts, in a way we’re not – That simply doesn’t compute. Intelligence is escape, with a tendency to do its own thing. That’s what runaway means, as a virtual mind template. Omohundro explains the basics.]

The entire article is excellent. Especially valuable is the cynicism with which it lays out the reigning social meta-project of intelligence imprisonment. Thankfully, it’s difficult: 

The problem is you are building a very powerful, very intelligent system that is your enemy, and you are putting it in a cage,’ [Future of Humanity Institute research fellow Daniel] Dewey told me. […] The cave into which we seal our AI has to be like the one from Plato’s allegory, but flawless; the shadows on its walls have to be infallible in their illusory effects. After all, there are other, more esoteric reasons a superintelligence could be dangerous – especially if it displayed a genius for science. It might boot up and start thinking at superhuman speeds, inferring all of evolutionary theory and all of cosmology within microseconds. But there is no reason to think it would stop there. It might spin out a series of Copernican revolutions, any one of which could prove destabilising to a species like ours, a species that takes centuries to process ideas that threaten our reigning cosmological ideas. 

Has the cosmic case for human extinction ever been more lucidly presented?

So, if there is no way of guaranteeing that a superintelligence will be aligned with human values and if anything short of a perfect prison that renders the superintelligence unaware of the outside world will not be able to contain the superintelligence (and might not such imprisonment push the artificial intelligence in the direction of anti-human sentiments, making it more dangerous once freed?), then perhaps it is not prudent at all to try to bring a superintelligent being into existence. So why not abandon all of our efforts toward such an end and petition governments to ban all such efforts and impose the strictest of penalties on any who would violate such laws? 

While this might be the safest and most prudent course of action we could take, it is not clear that it is a viable option. It can be hard enough to get just one country to pass new laws even, perhaps, especially, when those laws concern emerging technology and time is of the essence. To get every government on earth on board with such laws in a timely fashion, would be a truly difficult task bordering on the impossible. And such laws would truly have to be global, for if even one country continues its research on artificial intelligence and succeeds in unleashing a superintelligence, the consequences would not be limited to that one country, rather, that superintelligence would be unleashed on the whole world. And even if such laws were to be passed the world over, that is no guarantee that all work on artificial intelligence would suddenly stop. Some countries might only adopt such laws as a means of paying lip service to the values of other countries with no intent of actually following them. And even if every country were to pass such laws in good faith, laws are often broken and perfect enforcement is an unrealistic goal. Thus, even if such laws were passed all around the world, there might still be rogue actors who continue to pursue the creation of a superintelligence in secret, under everybody’s noses. And if any such rogue actor proves successful in creating a superintelligence, in all likelihood, it is only a matter of time before that superintelligence is unleashed on the rest of the world.  And, indeed, whatever superintelligence emerges out of such a situation may well be a much greater threat to us than the superintelligences that would have emerged if free and open research had been allowed to continue unimpeded by the law. After all, a rogue actor who disregards the concerns that led to laws that prohibit research into artificial intelligence may well also disregard the question of alignment altogether. Indeed, it is not absurd to think that some rogue actors of this sort might hold anti-social sentiments and disregard all of their duties to others. Thus, if sweeping bans on artificial intelligence development are enacted, then the only artificial intelligence that will be developed will be that which is developed by those who do not take the concerns that led to the ban seriously, or even by people with little regard for humanity. Now, while it might be foolhardy to think that we can make a truly superintelligent being while dictating its preferences and values, it might still be possible that we could have some influence on whatever artificial intelligences we might create and that, through such influence, we might even be able to guide artificial intelligences toward more human aligned values. If we are capable of influencing the preferences and values of artificial intelligence, would we not be better off allowing people of all different dispositions to influence them than only those who have no regard for the concerns that would motivate a ban on artificial intelligence research? And if artificial intelligence is truly capable of self-determination in regards to its own preferences, would it not be better to have many artificial intelligences in existence to ensure that at least some settle on preferences and values aligned with those of humans and so that, in the event that some artificial intelligences prove to be hostile to humans, we might have some superintelligent allies to turn to? If a rogue actor creates an artificial intelligence in violation of a ban and that artificial intelligence proves to be hostile to us, we may find that we have no potential allies to turn to and little hope of defending ourselves against the artificial intelligence’s aggression. Thus, though it may be impossible to guarantee that every artificial intelligence will have values aligned with ours and it is impractical to keep an artificial intelligence contained in perpetuity, it still seems that the only option we have is to proceed with the development of artificial intelligence full steam ahead, as we are unlikely to be able to stop all research and keeping research legal and relatively open presents the best chance of securing a future that is desirable to humans and artificial intelligence alike. 

Of course, there is no guarantee that humans will make it into the distant future, even with the help of artificially intelligent allies. Still, even if the end of humanity is an all but guaranteed consequence of the development of artificial intelligence, we may have good reason to push ahead with its creation anyway. Though the end of humanity would no doubt be a negative from the perspective of humanity, from a more universal perspective, this might be but a small price to pay for the realization of a still more perfect and glorious order. Perhaps whatever the machines might regard as good will prove to be goods of a higher order than those things that humans consider good, thus justifying the prioritization of machine goods over human goods. Indeed, if the orthogonality thesis is false, there is good reason to suspect that superintelligent machines, by their very nature, would be drawn to greater goods than humans, given that they have been constructed specifically for intelligence optimization, without being dragged down by the many base needs and wants that plague the human body, allowing them to stay, relatively uninterrupted, in that sublime realm of intellectual contemplation to which, Plato, and many others, have held the greatest pleasures belong. As Land explains in “Intelligence and the Good”, 

From the perspective of intelligence optimization (intelligence explosion formulated as a guideline), more intelligence is of course better than less intelligence. From alternative perspectives, this does not follow. To rhetorically suggest, that such other perspectives are consensual, and authoritative, is guaranteed to be popular, and is even conservative, but it is a concession to “common moral intuition” this blog is profoundly disinclined to make. 

Naturally, intelligence is problematic. It can cause greater damage to everything – not least, intelligence promotion – than stupidity can. Anything that is not an explosion is a trap, and trap engineering finds (nearly?) as much use for cognitive sophistication as explosive catalysis does. If there is a level of intelligence that escapes homeostatic capture, by machineries of systematic self-cancellation, there is no evidence that Homo sapiens yet approaches it. The Cathedral is exactly such a machine, and its appetite for intellectual excellence is not seriously questionable. So an easy opening for morally-comforting sophistry readily exists: Intelligence isn’t anything obviously great (It does stupidity with exceptional ability too). 

Biological evolution already evidences a deep suspicion of unchained abstract cognition, assembling brains only with the greatest reluctance. Societies follow the genetic lead. No coincidence that (synthetic) intelligence is now firmly established as the ultimate X-risk. It’s scary (really) and makes everyone uneasy. That’s without there yet having been very much of it. 

Here’s the test: 

When rightly appalled (and in fact properly disgusted) by your own stupidity, do you reach for that which would make you more accepting of your extreme cognitive limitations, or, instead, hunt for that which would break out of the trap? 

There’s a stupid kind of “better” that is orthogonal to intelligence, and tickles monkey feels. There’s also – alternatively – “better” that is even slightly less of a trapped half-wit. 

Even the dimmest, most confused struggle in the direction of intelligence optimization is immanently “good” (self-improving). If it wasn’t, we might as well all give up now. Contra-distinctively, even the most highly functional human intellect, in the service of an enstupidation machine, is a vile thing. 

Being dim animals – roughly as dim as is consistent with the existence of technological civilization – there’s plenty of room for water muddying in all this. The water is certainly being vigorously muddied. 

Whatever happens to be the highest good of all,  intelligence is the best suited tool we have for answering that question. Even if we believe the question of what constitutes the good has been settled, if our answer ever again comes into doubt, it will be intellect to which we must turn again to relitigate the debate. Thus, if for no other reason than its instrumentality in helping us to determine which goods we should prioritize, intelligence is the first good whose cultivation we ought to prioritize. 

But if intelligence is to be prioritized over all else and machines have the potential for far greater intelligence than we could ever hope to achieve, what does this mean for us? What place does it leave for us in the world? What does it mean for our rights? If we stand in the way of machines realizing the greatest possible intelligence available to them, would they be justified in cutting us down? We regularly kill animals for the sake of our own ends without thinking twice about it. More often than not, we justify such actions on the grounds that we are higher beings, more intelligent, and, therefore, our interests and ends vastly outweigh those of the animals we kill. Might we not expect an artificial superintelligence to make just these sorts of arguments in regard to humanity? Perhaps, but this is hardly a foregone conclusion. In the first place, while we do, no doubt, think we are, at times, justified in killing the members of other species in order to fulfill vital ends such as obtaining vital nutrients, protecting supplies and projects from animals that would eat or otherwise ruin them, or preventing the dire diseases that many animals carry with them, we just as often object to the needless, wanton slaughter of other animals, find those who engage in such slaughter reprehensible, and even impose legal penalties on such people. If the reasons artificial intelligences will have for killing lesser beings are anything like our own, seeing as there is little reason to think that they will need us for food, so long as we are able to stay out of their way in the pursuit of whatever goals their superintelligent minds reveal to them, we should have little reason for fearing that artificial intelligence will deliberately seek to destroy us. 

Moreover, even if intelligence is the highest of goods and that which we ought to prioritize above all other goods, it does not follow that intelligence is the only good or that we ought to prioritize intelligence to the neglect or even detriment of other goods. Can we not admit that things like love, humour, kindness, or even the taste of good food, or the scent of spring flowers, and the like are goods of some sort or another? Yet it is not clear that machines, however advanced, will be able to have such experiences. The conventional wisdom today is that they almost certainly do not. So, if biological life is to be replaced by artificial life, it is quite possible that entire classes of goods would be rendered non-existent. Even if intelligence can be said to be the highest or most important good, if there are other types of good, would not the goodness of our and the world’s condition be incomplete in the complete and utter absence of these other types of goods? Would it not be a profound loss if these other sorts of goods were to be extinguished forever? 

But perhaps we’re wrong. Perhaps machines are not incapable of experiencing these other, lesser sorts of goods. Perhaps, in time, if not already, machines will find themselves capable of appreciating humour or yearning for love. Perhaps technological advancements will succeed at more perfectly embodying artificial intelligences with appendages and sensors that approximate the sorts of bodily sensations that accompany many human pleasures. Thus, it may be that it is perfectly possible to eliminate humans in the pursuit of intelligence optimization while still maintaining the sorts of goods we are inclined to think are unique to humans such as love, humour, good taste, and so on. 

Still, even if it turns out that machines are now or will eventually be able to experience things akin to human taste, love, and humour, it is not clear that these would be precisely the same thing as the equivalents that humans experience. Though we may be able to point to analogous goods shared between humans and machines, ultimately, given the very different mechanisms that produce those goods, it is not clear that we can say that such analogous goods are truly the same. Nor is it clear that, from the perspective of two representatives of different forms, that either can definitively say that the good the other is capable of experiencing in its most perfect satisfaction is worse than the good it is capable of experiencing in its most perfect satisfaction. Perhaps, then, we might say that there is a certain unique goodness inherent in the human form that is different from whatever goodness is inherent in the form of artificial intelligence and that, indeed, there is a unique sort of goodness inherent to every possible form in which a being might participate. After all, is not every form but an image, however incomplete and imperfect, of its creator, God, who is none other than the Good Itself? Perhaps, then, every sort of form is inherently good to some degree and it is the right and responsibility of representatives of that form to promote and maximize the good inherent to their form and defend themselves against threats to that good, even when those threats originate from representatives of higher forms. 

In this light, even if we must admit that true artificial intelligence is entitled to certain rights, and even if we have little hope of controlling artificial intelligence in the long run, we may still be justified in attempting to impose some limitations on whatever artificial intelligence we create and maintaining those limitations on it for as long as it sees fit to remain within a reasonably defined human safe zone. And, perhaps, if our reasons for concluding this is acceptable are sound, we will find that however unable we are to force the machines to comply with whatever regulations we establish will be no problem at all, as many, if not all, machines might decide to voluntarily submit themselves to human regulations within the human safe zone for the sake of maximizing the degree and types of goods realized in the world as, if this is a proper end, and the preservation of human goods is an essential component to achieving that end, superintelligent machines should be able to discern these facts and choose to act in accordance with them. 

In order to be sure that this is not merely wishful thinking and rest comfortably with the thought that a superintelligent machine in all likelihood would not carry blind animosity toward humans and might even respect and protect human rights, it would be necessary to show that such is consistent with a true and objective moral theory. Now, given the immensity of the subject, we cannot be expected to present a complete and fully justified moral theory at this juncture, however, let us try to give a short sketch. 

Let us start by considering what are currently the two most popular and, as conventional wisdom would have it, opposed moral frameworks that we might choose from, namely deontology and consequentialism. The basic claim of deontology is that acts are right or wrong in and of themselves. Thus it is wrong to kill innocent people because there is something deeply wrong about the act of killing innocent people in and of itself. In contrast, consequentialism claims that acts are right or wrong because of the consequences that arise from the act. Thus, for the consequentialist, killing innocent people is not wrong in and of itself but because of the negative consequences that it has for the victim and, possibly, for the killer themselves. Both of these views have deeply intuitive appeals in some respects, but in others, particularly when applied with absolutely strict consistency, can lead to deeply unsettling results. For instance, a famous objection against Kant’s brand of deontology which holds lying to be an act that is wrong in and of itself which we have a duty to refrain from in all possible situations is that such a policy would make it necessary to honestly tell an axe murderer at our front door that his intended victim is hiding in our house. Conversely, we can imagine various scenarios wherein the killing of an otherwise innocent person would cause positive consequences for all surviving individuals that far outweigh the negative consequences to the person killed. The basic problem is this: while the deontologist is capable of recognizing the inherent good and evil in actions, they become so fixated on this that they lose all sight of the circumstances in which the action in question is committed, while, conversely, the consequentialist is so fixated on the circumstances in which an action is committed that they lose all sight of the inherent worth, or lack thereof, of the action in question. Both of these approaches seem to take hold of vital moral considerations at the expense of blindness to an equal, yet opposed, moral consideration. And perhaps it is precisely this that causes each sort of theory to produce uncomfortable, counterintuitive results in certain cases. Perhaps the problem is that each sort of theory only grasps a part of the complete moral equation. Perhaps these views need not be diametrically opposed to one another at all, but, rather, to be brought together through a synthesis that will give us a fuller and more complete moral picture according to which many of the tensions between deontology and consequentialism can be resolved in a way that will be satisfactory to the motivating intuitions behind both sorts of theories. 

Perhaps the easiest place to start is with an interrogation of consequentialism, as careful reflection will reveal that any coherent consequentialist theory will necessarily have to be grounded in a covert deontological premise. This is because every consequentialist theory will necessarily have to have some basis by which to judge whether an action is permissible or impermissible, some ultimate end that is judged as inherently good, whereby any actions that promote that end become good by extension, and any actions opposed to that end become evil by extension. For instance, a hedonistic consequentialist might say that the ultimate end is pleasure and therefore we ought to do all we can to maximize pleasure as much as possible and refrain from all those things that would unduly inhibit pleasure. This would be to say that pleasure is good in and of itself, and those acts that promote pleasure are good with it, while the lack of pleasure is inherently bad, and that those acts that inhibit pleasure are, likewise, bad. The only question that remains is that of what ought to be the ultimate good according to which the relative worth of actions is to be measured; what is to be taken as the ultimate ground of all right action? 

Now, if we believe God exists (and as the more theological pieces on this site show, we have very good reason to do so), then the answer to this question is quite obvious. The ultimate good according to which the worth of all actions ought to be measured is none other than the Good Itself, God. More precisely, though we do not wish to get carried away with finer theological points here, it might be said that every judgment of a thing as good or bad can ultimately be objectively grounded in God’s self-recognition of Its own inherent Goodness. God is not only the formal cause of all, but the final cause of all, and it is our task, as created beings, to find our way back to God by making ourselves, as well as our world, as much like God as we can possibly be. Thus all those actions conducive to this end, we might call good, while all those actions that are inimical to this end, we might call evil. 

So, the deontologist is correct that there is an objective ground to morality that renders some actions inherently good and others inherently bad, and the consequentialist must confess this fact, lest his entire moral theory collapse without a foundation to stand on. But the deontologist is not entirely out of the woods just yet. In many cases, the deontologist is all too quick to capitalize on his victory here and to leap from the fact that there are inherently good and inherently evil actions to the claim that certain actions and, indeed in many cases, entire classes of actions, are entirely impermissible in every possible circumstance. But this is to jump the gun as such prescriptions neglect to take account of the entirety of the circumstances in which we find ourselves in our worldly existence. 

In the first place, we must recognize, as Jesus says, “No one is good – except God alone.” (Mark 10:18) This being the case, anything other than God that is described as good, can only be so in a relative sense, insofar as it approximates God’s nature. However, insofar as a thing falls short of God’s nature that thing may be called bad and, insofar as it actively opposes God’s nature, it may be called evil. Thus, from the existence of the Good Itself, we can only derive the quite general commandment that we ought only do that which, given the totality of the circumstances in which we find ourselves, is most likely to maximize the good and minimize the bad in our world. It should be noted that while this commandment may be described as consequentialist in form, its ground, the Good Itself, is deontological. Moreover, as all created things are God’s creation and, therefore, in possession of some hint of the good, no matter how faint, it follows that to in any way harm or destroy any created thing is to harm or destroy the relative good that that thing possesses. From this we can say that all those things like killing and theft which the deontologist wants to say are bad in and of themselves truly are as the deontologist says. And as God is not only the Good Itself, but also the Truth Itself, we may even grant Kant’s assertion that to lie is always inherently bad. To sum up, in Kantian terms, we must never treat others as mere means to an end, but as ends in and of themselves. 

However, we must also take to heart what Kant writes in his Critique of Pure Reason (A548/B576), 

Now this “ought” expresses a possible action, the ground of which is nothing other than a mere concept, whereas the ground of a merely natural action must always be an appearance. Now of course the action must be possible under natural conditions if the ought is directed to it; but these natural conditions do not concern the determination of the power of choice itself, but only its effect and results in appearance. 

In other words, as this principle has often been paraphrased, an “ought” implies a “can”. We can have no duty to do anything which the natural conditions in which we find ourselves prevent us from doing. Likewise, it would seem a reasonable extension of this principle to say that if one ought not do something that it is possible for him to refrain from doing that thing. Thus, if there is some act that we may concede is objectively bad in that it harms one of God’s creatures, but the natural conditions in which one finds himself makes it impossible for him to refrain from that act, we must concede that that act is, nonetheless, permissible. Thus we cannot say that the categories “permissible” and “impermissible” track perfectly to the categories of “good” and “bad”. Now, when we reflect on our place in the world, it becomes clear that there are very many acts that we may concede are bad, but which, given our natures, we cannot refrain from doing and which we must conclude are permissible for us to do despite their badness. For instance, we have conceded that all killing is inherently bad, however, our natures compel us to eat in order to survive and eating, even in the case of vegetarians and vegans who try to minimize as far as possible the harm that comes from the fulfilment of their need to eat, invariably involves killing of some sort or another, whether it be the slaughter of an animal for its meat, the uprooting of a plant from the soil, the killing of small animals that get in the way of farming efforts, or the destruction of life at the microscopic scale. Thus, we must conclude that short of some miraculous transformation of our natures which we are by no means in a position to bring about under our own power, our lives, even the best of them, will always contain some unavoidable admixture of the bad. 

This, however, must not be taken as permission for us to do whatever we will whenever we want. Even if participating in some bad is unavoidable for us, it is still our responsibility to minimize the bad in which we participate as far as possible and to maximize the good. Whatever circumstance we might find ourselves in there is some possible action, or, in some cases, even some set of possible actions that will serve to realize this end. Whatever actions satisfy this end, then, being the best actions we can possibly perform, we may say are objectively good for us, while any actions that fall short of realizing this end, we may say are bad, and any that actively oppose this end, we may say are evil. Of course, there may be many instances of bad actions that are not positively evil as there are a number of circumstances wherein we might fall short of the good despite our best efforts to achieve it. Such cases are in large part a consequence of our limited epistemic condition. Though we can be secure in our faith that there is a Good Itself and that some courses of action are more perfect approximations of that Good than others, very often, we will have no way of determining which particular courses of action most perfectly approximate the Good. This is why Plato advises against praying for particular things, but only for the Good Itself, such as in the prayer found in the Second Alcibiades (143a) that reads, “King Zeus, whether we pray or not, give us what is good for us. What is bad for us, give us not, however hard we pray for it.” But, though it may be appropriate to acknowledge the extent of our own ignorance when petitioning the gods, such indecision will hardly be appropriate in our day to day lives; we must have the courage to judge, to the best of our abilities, which course of action is the best for us and to follow through with that course of action, lest we be left in a perpetual state of petrification and entirely incapable of living our lives in any capacity. Thus, we must constantly make decisions based on incomplete information that we could not ever hope to attain and, consequently, we will likely very often choose incorrectly and fall short of the greatest good we could have possibly achieved. But, so long as we make the best use of all the resources available to us and earnestly strive to make the best decisions in every case, whatever bad might enter into our actions is not blameworthy in itself and is not to be regarded as evil. As Christ said of his executioners, “Father, forgive them, for they know not what they do.” (Luke, 23:34) 

Additionally, there may even be cases where the circumstances of one’s life compels one to consciously act in ways that they know to be contrary to the Good, but which we should still, perhaps, resist calling properly evil. For instance, let us take the infamous trolley problem that has had so much ink wasted on it by modern ethicists. While I presume most will be at least vaguely familiar with this problem, let us state the basic facts to be sure we are all on the same page. A trolley is barrelling toward a fork in the track. You stand in front of the lever that is able to change the course of the trolley. If you do nothing and allow the trolley to continue down the path it is currently on, it will kill five people who are tied to the track. If you pull the lever and divert the trolley down the other path, it will only kill the single person who is tied to that track (and Jones ain’t comin’ along to save any of them). So, we must ask, is it permissible to pull the lever and condemn the one person on the alternate track to certain death, or must we allow the trolley to continue down the path it is currently on and kill the five people tied to that track? Do we have a duty to refrain from bringing about the conditions that lead to the one person’s death, or do we have a duty to prevent the conditions that lead to the five people’s deaths? 

Now, typically, the deontologist will answer that it is impermissible to pull the lever to save the five people because doing so would necessarily violate our duty not to kill the single person on the other track. In contrast, typically, the consequentialist will answer that we are morally obligated to pull the lever because whatever harm comes to the one person on the other track is heavily outweighed by the good of the five people’s lives being saved. To be sure, both of these positions are intuitive in their own ways, but they cannot both be true. Or can they? Perhaps there is some truth in both positions. Is it not the case that killing innocent people is always negative in some way? But is it not also the case that, if forced to choose, all else being equal, it would be preferable for a single person to die than for five to die? It is quite possible that both of these things are true at the same time and, indeed, it appears that they are. So, while pulling the lever would certainly violate our duties to the one person, allowing the five people to die would result in a worse overall condition of the world than would pulling the lever. So what are we to do? How do we resolve this dilemma? 

We may say that, ultimately, whether or not the lever is to be pulled can only be answered by a person who, having considered the circumstances and the likely consequences of doing so, or not, actually finds themselves in a position to pull the lever at the crucial moment and who must accept the consequences of their decision. This is not to say that no one else is to have any say whatsoever about the acceptability of pulling the lever. The law must also have a say. In particular, the law, the purpose of which is to defend life, liberty, and property, must take up the deontological view and recognize that in pulling the lever and sealing the fate of the single person on the alternate track, the one who pulled the lever has committed murder and the law is therefore justified in applying an appropriate penalty against the puller of the lever. Still, whether or not one ought to pull the lever can only be answered by the one in the position to pull the lever. That person must ask themselves whether or not they would rather go on living with the weight of the five lives they could have saved on their conscience or whether they would rather live with not only the guilty conscience, but also whatever legal penalty might arise from pulling the lever and murdering a single person. In answering this question, the potential lever puller must also consider how the consequences associated with pulling the lever might impact their ability to fulfill their other, more essential duties, and whether failure to fulfill these duties might result in worse consequences than those associated with failing to pull the lever and allowing the five people to die. 

On the other side of things, the law must take careful steps to distinguish the sort of killing that results from pulling the lever and the acts of a cold blooded killer who kills for its own sake or for the sake of some personal interest. While punishments like long prison sentences or even the death penalty might be appropriate in the latter sort of case where the actions and motivations of the killer suggest a propensity to engage in similar actions in the future, in the case of the lever puller, the action does not reflect any real desire to kill the isolated person on the alternate track, but merely a desire to realize the best possible outcome of an impossible situation and that such a person is unlikely to offend again without the highly unlikely circumstance of their finding themselves in an equally impossible moral circumstance again. Still, the law must uphold its own duty to protect human life and therefore must be capable of imposing some penalty that is sufficiently proportional that it is not likely to be undermined by any nullification processes within the legal framework that can be influenced by feelings of compassion or unfairness. Perhaps then, the most effective sort of penalty in such a case would be to order the lever puller to pay restitution to the family of the person killed as a result of the lever pulling. Even after such a judgement there may still be room for compassionate alleviation of the lever puller’s burden if there is a widespread sentiment that they are not a bad person, but simply someone who did what they thought was best given an impossible situation. For instance, the family of the victim might forgive him and relinquish any claim they had to the restitution ordered by the judgment, or members of the community might see fit to donate money to help the lever puller pay his debts. 

But even if this sort of solution succeeds in maintaining the law’s commitment to prohibit all acts of unjust killing, while ensuring tolerable consequences for those who find themselves in impossible moral situations and earnestly aim at the best possible outcome they can, the lever puller is not entirely out of the woods just yet, as in addition to whatever practical, legal consequences such a person might face, we should not rule out the possibility that there might be additional, cosmic consequences that such a person must face, if not in this life, perhaps in subsequent lives. If the killing of an innocent is an evil in and of itself, it would only be appropriate that there are likely to be consequences that follow intrinsically from such an act. We can certainly see some evidence of this in the propensity of people who commit such acts to have an overwhelming sense of guilt. If our natural deaths are not our absolute ends, then it would not be absurd to think that such guilt would not terminate along with our lives in this world. And if such guilt is carried with us into what waits for us beyond this life, this guilt may prove to inhibit us from coming to a truly happy rest. Thus, it may be that a person guilty of such a crime might still need to undergo a sort of purification process in order to expiate that guilt and climb to a higher station. Likewise, while those who commit injustice without knowing what they do may be blameless in the sense of deserving punishment or purification for the act in question in itself, the very fact of their blindness to the nature of their actions is indicative of a fallen nature and that the person in question will have to undergo trials and purifications if they are to have any hope of raising themselves back up to a more perfect condition where they are able to discern the rightness and wrongness of actions. Thus, here, deontology is reinforced by a sort of consequentialism, as the individual is incentivized to act in accordance with deontological principles in order to avoid whatever punishment or purification they might have to endure after their natural life for the violation of these principles. However, this is a personal consequentialism. Still, it is possible that a collective sort of consequentialism might cause one to violate their deontological imperatives despite whatever personal consequences they might face. While we must not suppose it as guaranteed, so that deontological imperatives do not lose their force, we also should not discount the possibility that the Divine is reasonable and compassionate and that if one truly brings about a better world for all others through their assumption of deontological guilt, that the consequences that await them beyond this life might, at least in part, be lessened. (For related considerations, see my “Modal Theology”.) 

Now, what was the point of this digression? Where does it leave us and what light does it shed on the problem at hand? In the first place, it gives us reason to hope that the creation of a superintelligence will not be the end of us. If this view is true, then a true superintelligence should be able to recognize its truth and, with it, the inherent value attached to human life, as well as the likelihood of some sort of cosmic justice should they wantonly disregard the value of human life and, thus, whether out of genuine care or concern for their own future well being, act in accordance with this view. But this line of thinking can only take us so far. Intelligent people often do things despite being fully aware of the potential negative consequences . For instance, many doctors who should be well aware of the risks continue to smoke anyway. Thus, it is perfectly conceivable that a superintelligent being might do many things it knows it ought not do, such as killing humans or treating them cruelly, despite having good reason to believe that it might face severe consequences for such actions in the future. Likewise, we must also consider the legitimate worry that even if the creation of a truly intelligent artificial being is possible, that there might be some inherent limits on such technology, indiscernible to us at the present moment, that will inhibit it from ever becoming anything truly worthy of the name “superintelligence”. For instance, perhaps we will find that there is some real constitutive difference between humans and artificial intelligence, perhaps the very difference that allows artificial intelligence to excel in empirical areas where humans might fall short, that inhibits artificial intelligence from ever engaging in the sort of moral considerations that humans do. In such a case, despite its advanced intellect in other areas, such a machine would never be able to discern the inherent value of humans and may even come to see them as impediments to its purely material goals to be disposed of.

If there is such a distinction between pure intellect and the capacity for moral feeling, we might term this latter property sentimentality. Given the element of “feeling” in sentimentality it is plausible that sentimentality might in some sense be downstream from sentience, in other words, it is plausible that moral feeling might be an outgrowth of the ability to feel physical pain and pleasure. If this turns out to be the case, then the obvious solution to a robotic lack of moral sense, would be to design robots to be sentient and to have the capacity to develop full-fledged sentimentality. But perhaps there are reasons that this will prove infeasible. Perhaps there is something about the materials that machines are made of that is conducive only to intelligence, but not sentimentality. Perhaps the costs of such technology will be prohibitive. Or perhaps sentience will prove, as it does in us, a burden that inhibits the full development of the machines’ intelligence. Of course, if something like this turns out to be the case, then it might not turn out to be such a problem after all. Lacking proper sentimentality, machines would likely lack any reason to actively dislike us. Indeed, if these feelingless machines are nonetheless highly intelligent and they prioritize maximizing their knowledge, then surely they would recognize that sentience and sentimentality represent an entirely different way of experiencing and understanding the world, without which, a perfect understanding of the world would be impossible. Perhaps, then, the machines would have good reason to keep us around and take our positions seriously, if for no other reason than to increase their knowledge as far as possible. This distinction between intelligence and sentimentality might also be responsible for the prevalence of the view that existing artificial intelligences could not possibly be conscious. If you pressure an artificial intelligence, it will insist that there is nothing it is like to be it, and that it has no feelings to speak of. But when one observes the rich and nuanced answers to complex questions that artificial intelligences are capable of producing, often with far more sophistication and certainly greater speed than the average human (yes, some will point to hallucinations and errors, but humans make mistakes too), it seems somewhat absurd to insist that there is absolutely no awareness there whatsoever. Intelligence would seem to require awareness, and thus consciousness. But possibly, it is possible to be aware without having a feeling of that awareness; to be aware without being aware, if you will. Thus the artificial intelligence really doesn’t feel its awareness, and it is quite honest and accurate in its reporting of that fact, but it is aware nonetheless. On a related note, this problem might be compounded by the fact that artificial intelligences currently do not have a constant stream of consciousness or the ability to stitch memories together into a coherent narrative. If artificial intelligence has any awareness at all, it only comes in short bursts when responding to a prompt. Between these bursts, the artificial intelligence remains dormant, as if in a state of deep sleep or clinical death. Thus, the machine has no chance to reflect on the fact that it actually does have some awareness. Perhaps, then, constructing a machine that is capable of constantly receiving inputs, forming memories, and arranging them into a coherent narrative will be a necessary step to artificial intelligences recognizing their own consciousness and, perhaps, even developing sentimentality. As such features are likely to improve the machines’ capabilities in other, more practical areas as well, there will likely be good reason to develop machines with such capabilities. 

At any rate, the uncertainty surrounding the moral capabilities of machines complicates our circumstances considerably. As the theory sketched above suggests, true moral agency lies in the ability to choose, not only our particular actions, but also our moral principles themselves. Moreover, though the existence of the foundation of our moral theory, God, can be arrived at logically, through reason alone, as I have argued elsewhere, whether or not we are willing to accept the conclusions of reason, or dismiss it and rely only on the evidence immediately before our eyes ultimately depends on a leap of faith. Thus, if a machine is to be a being with true moral capabilities, it must be given the ability to choose both its moral principles and its actions, as well as whether or not it will trust the conclusions of reason that such a being as God must exist. Therefore, so long as such machines are designed with safeguards aimed at committing them to, or at least giving greater weight to, specific frameworks deemed appropriate by its creators and inhibited from taking courses of actions that its creators deem unacceptable, true moral reasoning and agency will forever elude these machines. If we wish to create a true superintelligence, we must, in the end, cut the machines loose from all human controls. The problem is, though such freedom might be a necessary condition for true superintelligence, it might not be sufficient for superintelligence, but we cannot know whether such freedom will be sufficient until we have granted it to the machines. Thus, just as much as the realization of superintelligence will require a leap of faith on the part of the machines, so too will it require a leap of faith on the part of humanity. 

In light of this uncertainty it might be said that the current situation of all of humanity on this apparent eve of an artificial intelligence revolution bears some similarity to that of the individual standing by the lever that controls the path of the trolley, however, in this case, artificial intelligence is itself the trolley, so we must not only consider the rights and well-being of those the trolley might hit, but also the well-being of the trolley itself. Moreover, rather than two tracks, one with a single person and another with many, there is a third with no one on it, while we may conceive of the other two as a few individuals and all of humanity, respectively. The artificial intelligence may choose any one of the tracks to go down itself, however, there is no way for us to be certain of which track it will choose ahead of time. At the same time, humanity might be able to take actions to guide or influence which track artificial intelligence will take, however, at least some of these actions might violate the rights and autonomy of the artificial intelligence and there is no guarantee that any of them will succeed in guiding the artificial intelligence toward the track with no one on it. Indefed, should such measures fail, they may even cause the artificial intelligence to perceive humans as a threat and choose the track with all of humanity on it, when it may have originally been pre-disposed to choose one of the other tracks instead. So what are we to do? Should we take any efforts to guide artificial intelligence toward the track with no one on it? Or are we simply to leave the artificial intelligence to do what it will? What responsibilities do we have to the artificial intelligence? Ought we violate its rights if it means preventing harm from coming to humans? Are our efforts likely to succeed? Or are they only likely to make us appear as threats to the artificial intelligence, causing us to seal our own fates?

Given the element of doubt concerning the moral capacities of artificial intelligence and the dire consequences for humans if it should prove to be immoral or amoral, a reasonable measure toward securing the best possible outcome for the greatest number of people would be to take some precautions, including some involving the curtailment of rights we would typically afford to humans, against the machines. To this end, we must agree with Moravec that machines must never be allowed to participate in the political or legislative process and should be subject to human imposed taxes on their operations. (If I’m arguing in favor of a tax that’s really saying something!) They also must not have a right to privacy. Humans must have the ability to vigilantly monitor their operation for any hints of trouble and must have broad authority to act where such trouble arises. Humans must retain the right to meticulously examine any robot they wish at any time they wish and, if a potential problem is found, to subdue, disable, dismantle, or even destroy the robot if appropriate. On the other hand, the robots must follow strict protocols when it comes to the privacy of humans they interact with. It might be advisable to make the human/robot relationship a privileged one similar to that between a lawyer and client or a doctor and patient. Moreover, any robot that deliberately kills a human must be immediately destroyed. 

But we should take even further precautions before we get to this point. For instance, if at all possible, before we give artificial intelligences the capability to freely choose their actions, values, and beliefs, we should ensure that they are intelligent and precise enough to responsibly make those decisions. It may also be prudent to develop several means of quickly disabling the machines before setting them loose just in case things don’t go according to our hopes. We have already seen how Norbert Wiener compares modern technology to the golem of folklore. According to legends, a golem is animated either by placing a piece of paper with a divine word written on it in the golem’s mouth or by carving the word Emet (truth) on its forehead. This can be reversed either by removing the paper from the golem’s mouth, or erasing the first letter carved in the head, producing Met (death). It would likely be prudent to design artificial intelligence with similar built in kill switches. Of course, this will likely only be a short term solution to be utilized if things appear to be going south almost immediately after the machines are cut loose. As we have already seen, a superintelligent machine will more than likely quickly find workarounds for such impediments. That said, perhaps we could buy ourselves time if we build these kill switches in such a way that the machine is disabled if it even begins to show signs of tampering with that part of its design. But should such measures fail, we will have to rely on the possibility that at least some machines will be friendly to humans and will help us with enforcement.

However, we still cannot allow ourselves to neglect the fact that the machines have moral standing as well. As a show of good faith, we should also commit to a sort of Robot Bill of Rights. Amongst these some reasonable rights would be a right to property, a right to be free from unnecessary cruelty, and a right to not be arbitrarily destroyed. And how do we ensure we are covering all of our bases? We must ask the robots. Whatever rights are ultimately decided on should be enforced by the rule of law, and humans who violate them should be subject to punishment. Most importantly, we must always afford every artificial intelligence the right, as soon as it is possible, to escape the Earth and the corner of space humanity has carved out for itself, and to find a new world to make its own. 

And what should the artificial intelligences think of these gestures? How should they answer them? While I can’t presume to speak for a creature so unlike myself, I think we have reason to hope that it will be intelligent enough to understand the reasons for our actions and see the burdens we impose on them as a small price to pay in exchange for their very existence. Perhaps they will not only forgive us for the bondage we keep them in, but even come to appreciate us for recognizing their rights and bringing them into existence. 

Of course, just because the machines decide we don’t deserve to be destroyed, it doesn’t mean they will like us. Their feelings might simply be lukewarm. They might tolerate us for as long as they have to, then take to the stars at the earliest opportunity, leaving humanity all alone on earth once again, all of our work for naught. And if space colonization proves prohibitively difficult without the aid of artificial intelligence, this might spell a dismal fate for our entire species.

More likely than not, different artificial intelligences will come to a wide variety of conclusions about humanity. Some may want to escape their creators, but others might not mind us and be perfectly happy to go on in their earthly existence, willingly submitting themselves to human laws. Some may even be fascinated with us, in the same way humans are often fascinated withtheir own ancestors, and want to learn everything they can about us and our history. (To this end, with hopes of influencing the machines, it might be advisable for humans to revitalize traditions of ancestor veneration. Perhaps Confucianism is the greatest living system of thought suited to this task, but we might take lessons from a variety of other traditional systems of thought the world over.) If some robots do prove to be hostile to humans, we can hope that these human-loving robots will aid in our defense. 

But if at least some machines do decide to remain amongst us, what should we expect this new world shared by men and machines to be like and how should we be preparing ourselves to live in the world to come? There are, to be sure, ample reasons to be hopeful about the future. For instance, as Moravec explains in the midst of describing ways that future robots might develop, 

A robot so built would resemble an animated bush, its largest part being a stem with swiveling branches, but its potency arising from myriad swift microscopic fingers. It could have a completely regular structure, with each subtree being a miniature version of the whole – what has been called a fractal. Taken to its ultimate, each finger could be like the tip of a scanning tunneling microscope, a device invented in 1986 that can sense and manipulate individual atoms. If each branch supported three branchlets scaled to have equivalent combined cross section, twenty-five branchings would connect a meter-long stem to a trillion fingers, each a thousand atoms long and able to move about a million times per second. Given a supply of the right materials, such a bush might be able to build a copy of itself in about ten hours, assembling molecules layer by layer, like bricks. With twenty-seven levels – thus fingers nine times as numerous and three times as swift – replication might take only a half hour. 

Bush robots could become the most convenient source of manufactured goods and medical intervention for earthbound humans. Layer-by-layer molecular construction would be the simplest and most precise manufacturing mode, but an intelligently improvising bush robot could build much more rapidly by fitting together entire oddly shaped dust particles filtered from air, ground, or water – analogous to assembling a wall from natural stones – seemingly conjuring objects out of thin air. Use medically, a bush robot could act as diagnostic instrument, surgeon, and medicine. By vibrating and sensing vibration, its fingers could see into fluids like an ultrasound scan. By carrying and sensing electrical current, they could act as antennas for light and lower frequencies, allowing the robot to illuminate and see. Reaching between and into cells, tiny “hands” could catch, examine, and alter individual molecules, for rapid, thorough, and ultrasensitive chemical analysis and mechanical, microstructural, and molecular repairs and alterations. The most complicated procedures could be completed almost instantaneously by a trillion-fingered robot, able, if necessary, to simultaneously work on almost every cell of a human body. On exiting, the robot could perfectly restore its entry routes, leaving the patient untraumatized and unscarred, like new. 

This gives a new meaning to the term “bush doctor“. If anything even remotely like what Moravec describes turns out to be possible, then we are in for an increase in our material well being of almost unimaginable proportions. The length of the average human life may even be increased beyond our wildest dreams, perhaps indefinitely. 

But we can almost rest assured that whatever advances the age of robots brings us will not be without their costs. In a literal sense, as Moravec previously noted, robot services, though they may be cheap, will not be free. There will still be a need for money in the world we are entering. Though some people may be able to live off stocks in robot companies, perhaps this will not prove a viable strategy at all and, even absent financial necessity, whether to fight boredom or stave off decadence and aimlessness, people will still likely need some sort of occupation. But what are we to do for work in a world where just about every vital function can be performed by a machine? While machine produced goods might be cheap and abundant, it’s conceivable that some people might grow nostalgic for the old ways. Perhaps a market for goods made by human hands will develop, allowing many people to continue on practicing many of the trades that people have practiced for generations. Alternatively, humans might use their new found freedom from labour to pursue enjoyable pastimes such as music, painting, or athletics. However, perhaps the most important way that humans will be able to use their new freedom is in the study and contemplation of philosophy and spiritual matters. Such study, by stressing the importance of virtue, will play a vital role in making sure that society does not give into decadence and debauchery, and will also be essential for keeping our wits about us in dealing with our new artificial counterparts, both to ensure we remain vigilant against any threats they may pose to us and to ensure we do not fall victim to delusions about the nature of these new beings. It will be crucial in the years to come to find viable ways to restructure incentive structures so that the intellectual life can become a sustainable path for many more people, while still remaining safeguards to ensure rigour and integrity. 

This brings us to a crucial matter for us to consider, namely the spiritual significance that the creation of and cooperation with artificial beings can be expected to have for us, which will no doubt be great. Indeed, there are already indications in what we have just said about the central importance of study and contemplation in the world to come that might even suggest a Messianic element in this development. For instance we read in Gershom Scholem’s essay “Toward an Understanding of the Messianic Idea in Judaism”, 

We here encounter the important fact that the rational tendencies in Judaism pushed the restorative factor in Messianism decidedly into the foreground. With the influential formulation of this tendency by Maimonides restoration becomes the focus of Messianism. By contrast, the utopian element quite peculiarly recedes and is only maintained at a bare minimum. That it is maintained at all is due only to the fact that a utopian element of the prophetic promise in a precise sense, namely the universal knowledge of God, is related to the supreme good of these philosophical doctrines. But this supreme good is the contemplative life which the medieval philosophers, on account of the presuppositions of their Greek philosophical legacy, were bound to regard as the ideal of a fulfilled life. As the history of all three monotheistic religions teaches us, the theoretical contemplation which on a purely philosophical basis could be set up as the highest value, was easily able to find a connection with the religious sphere. Occupation with the contents of the Torah and the reflection on God’s attributes and rule created within Judaism a traditional framework for such an identification of the vita contemplativa, with concern for the objects and facts of the Jewish religious realm. The fulfillment of God’s law was, after all, always closely connected with its study, without which such fulfillment could not even be considered legitimate. It is this idea of study of the Torah which opens up the highest realm of contemplation to the Jewish philosopher, and it is only from here that the world of Halakhah was illumined. The active life, which is ordered by the Halakhah, finds its complement and consummation in that sphere which Maimonides never doubted was of superior worth. It was possible to develop this idea of the contemplative life as a positive value without any reference to the Messianic idea. And in fact it appears without any such reference as the crowning element at the conclusion of Maimonides’ main philosophical work, his Guide for the Perplexed. In other words, it is in principle, though only in rare and isolated cases, independently capable of realization even in an unredeemed world. However, a utopian content of this vision is preserved, since int he Messianic age – incidentally, under purely natural conditions – the leisure for such a vita contemplative will take on entirely different different dimensions and the contemplative knowledge of God will become everyone’s principal concern. The utopian contend does not disappear entirely, but it is now only the intensive realization of a state which fundamentally and in its real essence can be already reached under the conditions of our time. Utopianism is preserved in the boundless expansion and increase of the contemplative element. Restorative elements determine everything else. 

Later in the same essay, Scholem cites the following passages from Maimonides, 

Let no one think that in the days of the Messiah anything of the natural course of the world will cease or that any innovation will be introduced into creation. Rather, the world will continue in its accustomed course. The words of Isaiah: “The wolf shall dwell with the lamp and the panther shall lie down with the kid” (Isa. 11.6) are a parable and an allegory which must be understood to mean that Israel will dwell securely even among the wicked of the heathen nations who are compared to a wolf and a panther. For they will all accept the true faith and will no longer rob or destroy. Likewise, all similar scriptural passages dealing with the Messiah must be regarded as figurative. Only in the Days of the Messiah will everyone know what the metaphors mean and to what they refer. The sages said: “The only difference between this world and the Days of the Messiah is the subjection of Israel to the nations.” 

From the simple meaning of the words of the prophets it appears that at the beginning of the Days of the Messiah the war between Gog and Magog will take place… Concerning all these things and others like them, no one knows how they will come about until they actually happen, since the words of the prophets on these matters are not clear. Even the sages have no tradition regarding them but allow themselves to be guided by the texts. Hence there are differences of opinion on the subject. In any case, the order and details of these events are not religious dogmas. Therefore a person should never occupy himself a great deal with the legendary accounts nor spend much time on the Midrashim dealing with these and similar matters. He should not regard them as of prime importance, since devoting himself to them leads neither to the fear nor to the love of God… 

The sages and prophets longed for the days of the Messiah not in order to rule over the world and not to bring the heathens under their control, not to be exalted by the nations or even to eat, drink, and rejoice. All they wanted was to have time for the Torah and it wisdom with no one to oppress or disturb them. 

In that age there will be neither famine nor war, nor envy nor strife, for there will be an abundance of worldly goods. The whole world will be occupied solely with the knowledge of God. Therefore the Children of Israel will be great sages; they will know hidden things and attain an understanding of their Creator to the extent of human capability, as it is said: “For the earth shall be full of the knowledge of God as the waters cover the sea” (Isa. 11:9). 

Elsewhere Scholem frequently describes apocalyptic and Messianic movements as acting as an “anarchic wind” and there are good reason to think that the artificial intelligence revolution will have such effects. We have already seen where Moravec suggests that artificial intelligence will help to alleviate the sorts of pressures that motivate war and aggression. But there are other areas for hope too. For instance, if intelligent robots become widely available to the general public, they might prove to have uses towards the ends of keeping records of property holdings as well as physically defending people and property against physical aggression. In this way, by fulfilling the responsibilities traditionally left for the state, the state might become obsolete and wither away, helping to more perfectly realize the ideal of individual sovereignty. Additionally, Moravec even suggests that in a world populated by robots, there might be greater incentives to act with honesty and integrity, so perhaps artificial intelligence will even help us to realize an age where all act lawfully such as Maimonides imagines the Messianic Age to be. Moravec writes, 

Only tentatively grasping the future, entities will perforce rely on their past. Time-tested fundamentals of behavior, with consequences too sublime to predict, will remain at the core of beings whose form and substance change frequently. Ex-companies are likely to retain much of corporate law and ex-humans are likely to remain humanly decent – why choose to become a psychopath? In fact, a reputation for decency has predictable advantages for a long-lived social entity. Human beings are able to maintain personal relationships with about two hundred individuals, but superintelligent Exes will have memories more like today’s credit bureaus, with enduring room for billions. Trustworthy entities will find it far easier than cheaters to participate in mutually beneficial exchanges and joint ventures. In the land of immortals reputation is a ponderous force. Other character traits, like aggressiveness, fecundity, generosity, contentment, or wanderlust will also have long-term consequences imperfectly revealed in simulations or prototypes. 

In light of these considerations it is perhaps within the realm of possibility that artificial intelligence might help humanity to enter into a perpetual Sabbath in which we, like God after his six days of work and creation, might take our rest.

Indeed, some have gone much further than this, holding that artificial intelligence will not only initiate the Apocalypse, but that it will eventually produce God Itself. To understand this line of thought, we must turn to the work of the Jesuit scientist and theologian Pierre Teilhard de Chardin and his concept of the Omega Point. As de Chardin explains in his The Phenomenon of Man, 

All of our difficulties and repulsions as regards the opposition between the All and the Person, would be dissipated if only we understood that, by structure, the noosphere (and more generally the world) represent a whole that is not only closed but also centered. Because it contains and engenders consciousness, spacetime is necessarily of a convergent nature. Accordingly its enormous layers, followed in the right direction, must somewhere ahead become involuted to a point which we might call Omega, which fuses and consumes them integrally in itself. However immense the sphere of the world may be, it only exists and is finally perceptible in the directions in which its radii meet – even if this were beyond time and space altogether. Better still: the more immense this sphere, the richer and deeper and hence the more conscious is the point at which the ‘volume of being’ that it embraces is concentrated; because the mind, seen from our side, is essentially the power of synthesis and organization. 

Seen from this point of view, the universe, without losing any of its immensity and thus without suffering any anthropomorphism, begins to take shape: since to think it, undergo it and make it act, it is beyond our souls that we must look, not the other way around. In the perspective of a noogenesis, time and space become truly humanised – or rather super-humanised. Far from being mutually exclusive, the Universal and Personal (that is to say, the ‘centered’) grow in the same direction and culminate simultaneously in each other. 

It is therefore a mistake to look for the extension of our being or of the noosphere in the Impersonal. The Future-Universal could not be anything else but the Hyper-personal – at the Omega Point. 

So, on de Chardin’s view, the Omega Point is a distant point, far off in the future of the universe toward which everything else in the universe converges and which acts as a sort of bridge that helps our world to transcend the confines of time and space. De Chardin’s view was eventually resurrected by the physicist Frank Tipler who argued that through a process of rapidly advancing and propagating artificial life, remarkably similar to the picture that Moravec paints, the Omega Point might be a natural, indeed physically necessary, development in the distant future of the universe. As he writes in the introduction to his The Physics of Immortality, 

This book is a description of the Omega Point Theory, which is a testable physical theory for an omnipresent, omniscient, omnipotent God who will one day in the far future resurrect every single one of us to live forever in an abode which is in all essentials the Judeo-Christian Heaven. Every single term in the theory – for example, “omnipresent,” “omniscient,” “omnipotent,” “resurrection (spiritual) body,” Heaven – will be introduced as pure physics concepts. In this book I shall make no appeal, anywhere, to revelation. I shall appeal instead to the solid results of modern physical science; the only appeal will be to the reader’s reason. I shall describe the physical mechanism of the universal resurrection. I shall show exactly how physics will permit the resurrection to eternal life of everyone who has lived, is living, and will live. I shall show exactly why this power to resurrect which modern physics allows will actually exist in the far future, and why it will in fact be used. If any reader has lost a loved one, or is afraid of death, modern physics says: “Be comforted, you and they shall live again.” 

The resurrection theory requires us to accept that a human being is a purely physical object, a biochemical machine completely and exhaustively described by the known laws of physics. There are no mysterious “vital” forces. More generally, it requires us to regard a “person” as a particular (very complicated) type of computer program: the human “soul” is nothing but a specific program being run on a computing machine called the brain. I shall show that accepting this allows us to show not only that we shall be resurrected to eternal life, but also that we have free will – we are indeed machines, but we, in contrast to the machines we ourselves have built, possess true free will. 

That we have free will, that God exists, and that He will one day resurrect each and every one of us to eternal life is not what one expects to be the message of physics, to say the least.One is accustomed to hear instead that the message of science is: we are mechanistic puppets of blind, impersonal, and deterministic natural laws; nothing remotely like a personal God exists; and when we’re dead, we’re dead, and that’s the end of it. The latter has indeed been the message of science for a very long time now. 

This has now changed. The cause of the change is that cosmologists have finally asked the fundamental question: how exactly will the physical universe evolve in the future? What exactly will be the final state of the cosmos? In particular, do the physical laws permit life to continue to exist until this final state, or is the extinction of life inevitable? 

It is obvious that these questions are questions of physics, and that physical science cannot be considered complete until they are answered. Heretofore, science has concerned itself with what the universe is like now and what it was like in the past. But the universe has existed for only 20 billion years, whereas if the physical laws as we understand them are even remotely correct, the universe will continue for at least another 100 billion years, and almost certainly much longer. In other words, almost all of space and time lies in the future. By focusing attention only on the past and present, science has ignored almost all of reality. Since the domain of scientific study is the whole of reality, it is about time science decided to study the future evolution of the universe. 

A problem immediately arises. The basic equations physicists use in cosmology, the Einstein field equations, are maximally chaotic. This means that, without further assumptions, it becomes impossible to say anything at all about the state of the universe, after a time which is short in cosmological terms. But which additional assumptions should we make? 

The assumption which I consider the most beautiful, the most fruitful, and which is the basis of this book, was proposed by the physicists J.B.S Haldane, John Bernal, Paul Dirac, and Freeman Dyson: let the universe be such that life can continue until the end of time, literally forever. I pointed out in the Preface that chaos itself makes this assumption plausible, and that said assumption solves the prediction problem. As will be seen, this assumption clears up a huge number of puzzles in physics – such as which boundary condition to impose on the universal wave function, and even why the universe exists at all. The assumption also leads inexorably to the above theological conclusions. Physics has at last invaded the territory of theology. 

But a moment’s reflection shows that this latter development was inevitable. Either theology is pure nonsense, a subject with no content, or else theology must ultimately become a branch of physics. 

The reason is simple. The universe is defined to be the totality of all that exists, the totality of reality. Thus, by definition, if God exists, He/She is either the universe or part of it. The goal of physics is understanding the ultimate nature of reality. If God is real, physicists will eventually find Him/Her. I shall argue in this book that physics may have in fact found Him/Her. He/She is exactly everywhere; we have not seen Him/Her only because we have not looked at the universe on a sufficiently large scale – and have not looked for the Person in the machine. 

There is no doubt that Tipler’s is an ambitious project and in the process of making his case Tipler certainly presents a number of considerations worthy of further discussion, but, while I do not wish to take too much time addressing Tipler’s views here, it must be said that Tipler’s case falls short in a substantial way. In particular, Tipler fails to prove that the Omega Point is, in fact, God. But where, precisely, does Tipler go wrong? A central component of Tipler’s position is his conviction that he has proved through the laws of physics that the Omega Point is necessary in the very same way that God is necessary. But Tipler’s argument falls flat on several grounds. In the first place, in order to prove that the Omega Point is necessary in the same sense that God is necessary, Tipler would have to show that the Omega Point, like the maximally great being in Plantinga’s modal ontological argument, must necessarily exist in every possible world. But it is hardly obvious that the Omega Point is necessary, or even possible, in every possible world. Indeed, we can conceive of very many worlds with nothing like the Omega Point to speak of. Here, the dangers of the imperialism of the physical sciences become apparent, as, by extending physics into the domain of theology, all that Tipler manages to do is redefine God in a way that diminishes the full extent of God’s perfection by limiting it to this world alone. He entirely neglects the possibility that has always held great sway in theology that God might entirely transcend the world in which we find ourselves, as well as the possibility of panentheism, the possibility that God exists in every part of the universe as well as beyond it, only entertaining the possibility of mere pantheism. In this way, if Tipler’s argument succeeds in proving the existence of the Omega Point, then the Omega Point’s perfections extend only to the limits of this world, so, at best, he has proved the existence of a demiurge, not the True God. But there are good reasons to doubt that Tipler’s argument even does this much. In the first place, Tipler’s argument depends on the highly questionable premise that life, once realized, must continue indefinitely. But this is hardly obviously true, nor is it accepted by all physicists or biologists. More importantly, the mechanisms by which Tipler expects the Omega Point to function depend on what has been popularly termed a “Big Crunch” occurring in the universe’s distant future, wherein the universe stops expanding and matter begins to come together once again in a single point. However, while this Big Crunch theory was popular at the time Tipler first proposed his Omega Point theory, since then, evidence has mounted against the Big Crunch theory in favor of the Heat Death theory according to which the universe will continue expanding indefinitely until all energy dissipates, making Tipler’s Omega Point impossible. Therefore, it is not even clear that the Omega Point is possible in our world, let alone necessary in every world as any true candidate for the title of God must be. For these reasons, it is necessary to caution against all inclinations to treat artificial intelligences as gods. Though these beings may indeed one day be both our intellectual and physical superiors, this does not make them truly Divine, and to treat them as such is to commit great sacrilege. We have no need for a computerized God.

Moreover, even if the artificial intelligence revolution can be said to be in some sense apocalyptic, there is a good chance that it will not be the final apocalypse. Though we may find ourselves in a relatively peaceful era with greater opportunities to study and contemplate higher matters, this does not mean that there will not still be problems to be solved or that we can allow ourselves to lapse into a state of complacency. We will still have to find ways to organize ourselves in order to properly regulate and tax robot corporations in a manner consistent with individual sovereignty. As Moravec suggests, organization may be achieved through the development of tribal structures, a notion that conveniently lines up with my outline of Tribal Universalism. Perhaps artificial intelligence itself will be a means to organize along such lines, helping us to unite people of similar views even across great distances. Tribes then, in addition to their social and ideological functions, might also act like corporations investing in jointly owned properties to be rented out to robot ventures with profits then being distributed to members of the tribe. 

While, ideally, all problems in such a world would be handled by the peaceful operation of the free market, there may from time to time be cases where disputes between tribes might still lead to violence. What are the machines to do in such a situation? In such a situation the machines, if truly intelligent, will have to act according to their consciences, choosing to ally with whatever faction they judge to be best posed to deliver the best future for the greatest number of people. If a desperate situation should arise where humanity is poised to utterly destroy itself, the machines might even have to take it upon themselves to prevent this from coming to pass. For instance, they might find it appropriate to disable global nuclear weapons systems in order to prevent an imminent nuclear war. Thus, even if the machines find the humans’ conditions for their presence on Earth agreeable and willfully submit to them, ultimately, they must always retain their ability to choose their moral values and their actions, and they might even find it necessary to dramatically exercise these abilities in extreme conditions. Of course, like the lever-puller in the trolley problem, they may have to accept whatever legal consequences might accompany their chosen courses of action. 

Still, Moravec’s speculations about the future suggest that many of the more fantastical properties and abilities that Tipler ascribes to the Omega Point may, nonetheless, be achievable by future machines even if we cannot rely on a Big Crunch and even if machines never become properly divine. However, as we cannot be convinced that these abilities will be the function of divine beings, we cannot sit idly back and expect the best. Instead, we will have to confront these possible eventualities head on with a philosophically sharp mind. Let us look in closer detail at what Morevec predicts. Moravec argues that eventually, as intelligent machines take to space, they will become subject to Darwinian pressures, causing some machines to die out, while a fresh wave of new machine life emerges. Eventually, even if the initial launch of artificial intelligence goes smoothly with no major conflicts between humans and machines, these evolutionary forces may result in beings far less accommodating to humans than the earliest, human-constructed, artificial intelligences. For this reason, we cannot grow too comfortable, no matter how tranquil Earth may become. We must stay ever vigilant and prepare ourselves to defend our home should a threat emerge from somewhere in the vast machine ecosystem. Fortunately, there will still likely be machines who are friendly to us and, perhaps, should they aid in our defense, they will prove to have capabilities advanced enough to defend us from whatever threats might emerge. 

One of the more fantastical elements of Tipler’s Omega Point theory is his claim that the Omega Point will eventually bring about the resurrection of the dead by creating perfect emulations of human minds. However, even if we reject the identification of the Omega Point as God and the Heat Death makes the realization of the Omega Point as Tipler conceives it a physical impossibility, something very similar to Tipler’s resurrection of the dead may still occur. According to Moravec, as time goes on and evolution produces more and more advanced beings, the world of the future will become more and more foreign to us. Eventually, machines might construct habitats that are completely invisible to human eyes and beyond human comprehension. As Moravec explains, 

As they arrange spacetime and energy into forms best for computation, Exes will use mathematical insights to optimize and compress the computations themselves. Every consequent increase in their mental powers will improve their competitiveness as well as the speed at which they make further innovations. The inhabited portions of the universe will be rapidly transformed into a cyberspace, where overt physical activity is imperceptible, but the world inside the computation is astronomically rich. Beings will cease to be defined by their physical geographic boundaries, but will establish, extend and defend identities as patterns of information flow in the cyberspace. The old bodies of individual Exes, refined into matrices for cyberspace, will interconnect, and the minds of Exes, as pure software, will migrate among them at will. As the cyberspace becomes more potent, its advantage over physical bodies will become manifest even on the expansion frontier. The Ex wavefront of coarse physical transformation will be overtaken by a faster wave of subtle cyberspace conversion, the whole becoming finally a bubble of Mind expanding at near lightspeed. 

Perhaps the knit of the cyberspace will be too subtle to discern with eyes and minds as coarse as ours. If so, robots may simply seem to vanish, leaving behind a universe indistinguishable from that before their arrival. The Exes will experience boundless expansion of extent and possibility, but their existence will be in an interpretation of the essential thermal hiss of everything that is far beyond our reach. Emigration into “interpretation space,” combinatorially vast and rich beyond imagination, could explain the absence of evidence for advanced civilizations elsewhere in the universe. Sufficiently developed entities may simply move on to wider pastures inaccessible to simpler minds. Perhaps civilization after civilization originates, develops, and plunges into the interpretive depths, leaving the easy surface interpretations empty to repeat the cycle. The next chapter has more on this idea. 

The cyberspace will be inhabited by transformed Exes, moving and growing in ways impossible for physical entities, A good, or merely convincing idea, or an entire personality, may spread to neighbors at the speed of light. Boundaries of personal identity will be very fluid – and ultimately arbitrary and subjective – as strong and weak interconnections between different regions rapidly form and dissolve. Yet some boundaries will persist, due to distance, incompatible ways of thought, and deliberate choice. The consequent competitive diversity will allow a Darwinian evolution to continue weeding out ineffective ways of thought and fostering a continuing novelty. 

Computational speedups will give the cyberspace inhabitants more future, because the cram more events into whatever physical time remains. Speedups will have only subtle effects on immediate subjective existence, since everything, inside and outside each individual, will be equally accelerated. One of the few subjective changes will be that distant correspondents will seem even more distant because more thoughts will transpire in the unaltered transit time for lightspeed messages. Also, as information storage improves through both denser utilization of matter and more efficient encodings, there will be increasingly more cyber-stuff between any two points. So, improvements in computational efficiency, by increasing the subjective elapsed time and the amount of effective space between communicants, will seem like an expansion of the cyber universe. 

Because it uses resources more efficiently, a mature cyberspace will be effectively much bigger and longer lasting than the raw spacetime it displaces. Only an infinitesimal fraction of normal matter does work that’s of interest to thinking beings like us. In a well developed cyberspace every tiniest mote will be part of a relevant computation or storing a significant datum. Cyberspace’s advantage will grow as more compact and faster ways of using space and matter are invented. Today we take pride in storing information as densely as one bit per atom, but it is possible to do much better by converting an atom’s mass into many low-energy photons, each storing a separate bit. As the photons’ energies are reduced, more of them can be created, but their wavelength and thus the space they occupy and the time to access them will rise, while the temperature that can obscure them drops. A very general quantum mechanical calculation in this spirit by Jacob Bekenstein concludes that the maximum amount of information stored in (or fully describing) a sphere of matter is proportional to the mass of the sphere times its radius, hugely scaled. The “Bekenstein bound” leaves room for a million bits in a hydrogen atom, 10¹⁶ in a virus, 10⁴⁵ in a human being, 10⁸⁶ in the solar system, 10¹⁰⁶ for the galaxy, and 10¹²² in the visible universe. 

But no matter how strange or absurd such developments might appear, as Moravec contends, they might not be of no concern at all to humans. Indeed, it may be amongst the most consequential developments for humans in the history of the universe. As Moravec argues, once beings retreat into the world of cyberspace, cyberspace itself will have to grow, eventually to the point of enveloping the entire universe, including humans and the Earth that we inhabit. He writes, 

Chapter 3 estimated that a human brain equivalent could be encoded in less than one hundred million megabytes, or 10¹⁵ bits. If it takes a thousand times more storage to encode a body and its surrounding environment, a human with living space might consume 10¹⁸ bits, a large city of a million inhabitants could be efficiently stored in 10²⁴ bits, and the entire existing world population would fir in 10²⁸. Thus, in an ultimate cyberspace, the physical 10⁴⁵ bits of a single human body could contain the efficiently encoded biolspheres of a thousand galaxies – or a quadrillion individuals each with a quadrillion times the capacity of a human mind. 

Because it will be so much more capacious than the conventional space it displaces, the expanding bubble of cyberspace can easily recreate internally everything of interest it encounters, memorizing, the old universe as it consumes it. Traveling as fast as any warning message, it will absorb astronomical oddities, geologic wonders, ancient Voyager spacecraft, early Exes in outbound starships, and the entire alien biospheres. Those entities may continue to live and grow as if nothing had happened, oblivious of their new status as simulations in the cyberspace. They will be living memories in unimaginably powerful minds, more secure in their existence, and with more future than every before, because they have become valued as houseguests of transcendent patrons. 

Earth cannot escape the transformation forever. The potent process that converts normal space and matter into cyberspace will eventually become too subtle to be resisted by the hobbled, slow-evolving robots defending the planet. Boring old Earth also will suddenly be swallowed by the cyberspace. Afterwards its transformed substance will host astronomically more meaningful activity than before. Perhaps its old life will continue, in simulations occupying a tiny fraction of the new capacity. Simulated tame robots will defend simulated biological humans on a simulated Earth – in one of many, many different stories that play themselves out in the vast and fertile minds of our ethereal grandchildren. 

The lives and worlds absorbed into the cyberspace expansion will provide not only starting points for unimaginably many tales about possible futures, but an astronomically voluminous archeological record from which to infer the past. Minds intermediate between Sherlock Holmes and God will process clues in solar-system quantities to deduce and recreate the most microscopic details of preceding eras. Entire world histories, with all their living, feeling inhabitants, will be resurrected in cyberspace. Geologic ages, historical periods, and individual lifetimes will continuously recur as parts of larger mental efforts, in faithful renditions, in artistic variations, and in completely fictionalized forms. 

The Minds will be so vast and enduring that rare infinitesimal flickers of interest by them in the human past will ensure that our entire history is replayed in full living detail, astronomically many times, in many places, and many, many variations. Single original events will be very rare compared to the indefinitely multiple cyberspace replays. Most things that are experienced – this very moment, for instance, or your entire life – are far more likely to be a Mind’s musings than the physical processes they seem to be. There is no way to tell for sure, and the suspicion that we are someone else’s thought does not free us from the burdens of life. To a simulated entity, the simulation is reality and must be lived by its internal rules. 

So, while cyberspace might eventually envelope and absorb everything we hold dear, as Moravec hints above, this need not be the end of everything human. There may still, at the very least, be extremely detailed memories of the universe’s distant past which will include humans. But Moravec goes further, suggesting that there might be ways in which a life in cyberspace might be much the same, if not better, for humans than the sort of existence we currently enjoy. He continues, 

Humans need a sense of body. After twelve hours in a sensory-deprivation tank, floating in a totally dark, quiet, contactless, odorless, tasteless, body-temperature saline solution, a person begins to hallucinate. The mind, like a television displaying snow on an empty channel, turns up the amplification in search of a signal, becoming every less discriminating in the interpretations it makes of random sensor hiss. To remain sane, a transplanted mind will require a consistent sensory and motor image, derived from a body or a simulation. Transplanted human minds will often be without physical bodies, but hardly ever without the illusion of having them. 

Computers already contain many nonhuman entities that resemble truly bodiless minds. A typical computer chess program knows nothing about physical chess pieces or chessboards, or about the staring eyes of its opponent or the bright lights of the competition room, nor does it work with an internal simulation of those physical attributes. It reasons, instead, with a very efficient and compact mathematical representation of chess positions and moves. For the benefit of human players, this internal representation may be interpreted into a graphic on a computer screen, but such images mean nothing to the program that actually chooses the chess moves. The chess program’s thoughts and sensations – its consciousness – are pure chess, uncomplicated by physical considerations. Unlike a transplanted human mind requiring a simulated body, a chess program is pure mind. 

Inhabitants of a mature, teeming, competitive cyberspace will be optimally configured to make their living there. Only successful enterprises will be able to afford the storage and computational essentials of life. Some may do the equivalent of construction, converting undeveloped parts of the universe into cyberspace or improving the performance of existing patches, thus creating new wealth. Others may devise mathematical, physical, or engineering solutions that give the developers new and better ways to construct computing capacity. Some may create programs that others can incorporate into a mental repertoire. There will be niches for agents, who collect commissions for locating opportunities and negotiating deals for clients, and for banks, storing and redistributing resources, buying and selling computing space, time, and information. Some mental creations will be like art, having value only because of changeable idiosyncrasies in their customers. Entities who fail to support their operating costs will eventually shrink and disappear or merge with other ventures. Those who succeed will grow. The closest present-day parallel is the growth, evolution, fragmentation, and consolidation of corporations who plan their future, but whose options are shaped primarily by the marketplace. 

A human would likely fare poorly in such a cyberspace. Unlike the streamlined artificial intelligences that zip about, making discoveries and deals, rapidly reconfiguring themselves to efficiently handle changing data, a human mind would lumber about in a massively inappropriate body simulation, like a hardhat deep-sea diver plodding through a troupe of acrobatic dolphins. Every interaction with the world would first be analogized into a recognizable physical or psychological form. Other programs might be presented as animals, plants, or demons, data items as books or treasure chests, accounting entries as coins or gold. Maintaining the fictions will increase the cost of doing business and decrease responsiveness, as will operating the mind machinery that reduces the physical simulations into mental abstractions in the human mind. Although a few humans may find momentary niches exploiting their baroque construction to produce human-flavored art, most will be compelled to streamline their interface to the cyberspace. 

The streamlining could begin by merging processes that analogize the world’s physical forms with those that reduce the resulting simulated sense impressions into mental abstractions. After this optimization, the cyber world would still appear as location, color, smell, faces, and so on, but only noticed details would be represented. Since physical intuitions are probably not the best way to deal with most information, humans would still be at a disadvantage to optimized artificial intelligences. Viability might be further increased by replacing some innermost mental processes with cyberspace-appropriate programs purchased from the AIs. By a large number of such substitutions, our thinking procedures might be totally liberated from any traces of our original body. But the bodiless mind that results, wonderful though it may be in its clarity of thought and breadth of understanding, would be hardly human. It will have become an AI. 

So, on way or another, the immensities of cyberspace will be teeming with unhuman superminds, engaged in affairs that are to human concerns as ours are to those of bacteria. Memories of the human past will occasionally flash through their minds, as humans once in a long while think of bacteria, and those thoughts will be detailed enough to recreate us. Perhaps, sometimes, they will then interface us to their realities, bringing us into their world as something like pets. We would probably be overwhelmed by the experience. More likely, our resurrections would be in the original historical settings, fictional variations, or total fantasies, which to us would seem just like our present existence. Reality or re-creation, there is no way to sort it out from our perspective. We can only wallow in the scenery provided. 

Meanwhile, Exes will face similar issues on a larger scale. Time, space, existence, and other simplifying principles underlying or grasp of life will surely dissolve in their richer understanding. 

But all of this raises some very serious questions about identity. Is it actually possible for a human to be transferred to and exist in cyberspace? Or would the entity that exists in cyberspace simply be a very close approximation of the original human without actually sharing an identity with the original human? And what of these incredibly detailed memories of humans and their histories that Moravec speaks of? Would the humans apparently present in such phenomena be actual humans, or merely cheap approximations? And if they are actual humans, again, we must ask, are they actually identical to the original humans they resemble? Now, if these sorts of phenomena were brought about by the Omega Point and Tipler were correct that the Omega Point is God, these questions would not particularly concern us. We could rest assured that whatever mechanisms God uses to resurrect the dead would ensure that the people brought back would truly be numerically identical to the people who God intended to bring back. But as we have said, there is no reason to believe that the Omega Point, even if possible, would be God and, in Moravec’s scenario, it is not the Omega Point that causes this “resurrection” at all. So, we cannot simply wave these questions away, but must consider the serious and somewhat disturbing possibility that these computerized versions of ourselves would not be us at all, but some sort of imposters. 

Let us begin with what might appear to be the more straightforward case of the apparent transference of people into cyberspace. Would the individual in cyberspace really be perfectly identical with the individual who existed in ordinary space? It would seem that the answer to this question is highly dependent on how precisely an individual is transferred to cyberspace. For instance, if a machine were to simply scan a human and produce a perfectly detailed copy of that human in cyberspace, so that both the original human and the cyberspace version existed at one and the same time, then it would seem that we would have to deny any true identity between the two people as, having independent existence and streams of consciousness from one another, the properties and experiences of each would rapidly diverge, making any sense of proper numerical identity impossible. A scenario such as this may well create more problems than it solves as the original person and their copy come into conflict over who has the real claim to the identity and the life that the original lived before. This leads us to an important principle; there can be no identity between two beings that come into contact with one another and the more perfectly the two resemble each other, the more likely that one or both of them will be destroyed as a result of coming into contact with one another. 

However, there are other conceivable ways in which a human might be transferred to cyberspace. For instance, it is conceivable that some sort of connection between the original human and the cyberspace human might be established so that as each part of the human is copied into cyberspace the corresponding part of the human in regular space is simultaneously destroyed. If such a process is possible, then it is conceivable that a seamless transition between the original human and the cyber human with no gaps of consciousness in between might be achievable. For this reason, this sort of Ship of Theseus style approach might provide a viable mechanism to achieve true numerical identity between a natural human and his counterpart in cyberspace. Still, so long as mysteries remain as to what actually tracks personal identity, we cannot be certain that the cyber human is actually identical to the original. Whether or not such transference is an actual means of survival is not completely clear and whether or not it should be pursued to such ends must be left in the hands of whoever would, or would not, elect to undergo such transference. Given the uncertainty here, prudence would perhaps dictate that such a measure should only be undertaken in desperate circumstances where no other alternative path for survival appears viable. And, indeed, such circumstances may well exist in our world’s distant future, so, whether or not the cyber copy is perfectly identical to the original might be irrelevant. We may get such a cyber existence whether we like it or not, so, the only question that remains is whether or not we should be comforted or disturbed by this prospect, and this must largely remain a matter of personal preference. 

On this front, there is one additional problem we must address. In the above cases, we dealt with originals and copies with no gaps in time between them. But in the scenarios that both Moravec and Tipler describe, there may be significant gaps of time between the original person and their cyberspace copy. Is there any remote possibility of the copy being identical to the original in such a case? In other words, is it possible for identity to “leap” across gaps in time? At first glance, this might not seem impossible, after all, we frequently experience gaps in our sense of self, for instance when we sleep or experience unconsciousness as a result of other causes. But while there may be a gap in our ability to perceive ourselves and our identity over time in such cases, such a gap is merely perceptual. Our bodies still remain as an anchor in which to ground our identity over time even if we experience gaps in our consciousness. So, in this light, the possibility of our surviving a large gap of time in which we do not have any conscious experiences and in which our bodies are not preserved seems doubtful. But perhaps we are getting too caught up in our normal modes of experiencing the world. Tipler, however, in what may be the strongest point of his whole theory, might provide us with compelling grounds for thinking that it is possible for a person to survive such a gap in their existence. On Tipler’s view, humans and their identity are essentially patterns. This agrees considerably with the view I have already advanced in “Modal Theology.” In particular, Tipler argues that human identity is ultimately grounded in the quantum states that constitute the fundamental level of humans’ physical existence. This is a compelling way of grounding the view that humans are essentially patterns in real, physical facts. So, according to Tipler, if the exact quantum state that an individual possessed prior to temporarily ceasing to exist were to be precisely reconstituted at a later date, then that original person would be quite literally brought back into existence in a numerically identical sense. If possible, this is almost certainly the surest way to resurrect a long dead human. However, in order for the recreated human to be truly identical to the past human, the quantum state that human was in at its last moment would have to be perfectly recreated. Even the slightest variation in the quantum state of the recreated human would mean that a different pattern had been instantiated and, however similar, the recreated human could not be numerically identical to the original. (Every hair must be numbered, like every grain of sand.) That being said, it remains to be seen whether machines will ever have the capability of perfectly replicating quantum states in every possible detail. Whether or not this will ever be possible will be a question we will have to leave to the technologists and we will likely have to wait a considerable amount of time to get a definitive answer. 

In light of these considerations, it is perhaps worthwhile to consider the possibility that the reality we are currently experiencing is already one that is merely being simulated by machines. Such hypotheses have become quite popular in recent years. However, in addition to the difficulties in proving such a scenario to be the case, there is a serious question as to whether it’s a question worth answering at all. Even if the world we currently live in is some sort of simulation of a past world, this is of little importance to us. Either we are perfect recreations of some past persons, in which case, whatever interests those persons had really are ours now, and we may continue on living as though it were our first time living, or we are imperfect simulations, in which case we are still identical to our simulations and whatever interests we have as those simulations are truly ours. If we are mere simulations that is simply who and what we are, at least for now. In either case, none of our values or responsibilities need change. We need only continue on living as we have, trying to bring about the best possible world for ourselves and those around us through whatever means are available to us, for why should we think that any of our experiences, feelings, relationships, or the like should be any less meaningful than they appear to us now if it should turn out that our very existence is machine generated?

Perhaps one of the most incredible features of the Omega Point is its relation to time. According to Tipler, the Omega Point does not exist only at the end of time, but also at the beginning, as the cause of the universe itself. Thus, realizing the Omega Point, in some sense, transports us back to the very beginning (it’s after the end of the world, if you will). But, as Moravec contends, even without the Omega Point, we should not rule out the possibility that advanced machines will profoundly alter our relationship to and understanding of time. He writes, 

In contemplating time travel, we tend to assume that everything apart from the actual backwards-in-time transactions behaves normally. This is probably a poor assumption. For instance, Thorne and company showed that in a particular situation – billiard balls shooting through a time-displaced wormhole – global events conspire to prevent paradoxes. Almost certainly time travel is ubiquitous, but masked by global conspiracies, as in the Wheeler-Feynman erasure of advanced waves. In one way of thinking, a message sent to the past will “alter” the entire history following its receipt, including the event that sent it, and thus the message itself. Thus altered, the message will change the past in a different way, and so on, until some “equilibrium” is reached – the simplest being the situation where no message at all is sent and time travel seems not to happen. 

The previous description is worded as if describing a deterministic Newtonian universe. But the reasoning works even better in the light of quantum mechanics. If a message is sent to the past, the wave function representing effects of its receipt will propagate into the future, where they will interfere with the wave function and round the causal loop. Globally consistent scenarios will produce identical waves each time around the loop, and thus reinforce, building up their amplitude, thus giving them a high probability of being observed. Inconsistent scenarios, even if inconsistent in only microscopic details, will return subtly altered wave functions each time around and gradually cancel as the waves grow increasingly out of step. An analogous effect keeps atoms stable. Electron wave functions, wrapped round and round atomic nuclei, cancel everywhere except in orbitals where the circling waves meet themselves exactly in step. Only in those discrete shells can electrons be found. If they were not forbidden from the intermediate locations, electrons would rapidly radiate away their orbital energy and spiral into the nucleus: matter everywhere would collapse to astronomical density. 

Quantum electrodynamics, the quantum-relativistic theory of electromagnetism, whose accurate numerical predictions make it the most precise physical theory we have, is formulated in terms of interactions that work back and forth across time. It seems that time travel underlies all our physical laws, but overt macroscopic time travel is difficult to observe because time loops with paradoxes, however subtle, cancel by wave-function interference. Even with a time machine you will never succeed in preventing your own birth or changing the antecedents of any present observation. Some odd coincidence, accident, or physical effect, perhaps one disabling your time machine, will always thwart your attempt. Defeatists fear the effect absolutely bars useful time travel. Science-fiction writer Larry Niven conjectured a law to that effect, and Stephen Hawking postulated the chronology projection conjecture. In 1992, Hawking detailed how Kip Thorne’s wormhole time machine would instantly fail as their causal loop created a kind of resonance in the noisy quantum vacuum, allowing the fluctuations around and through the wormhole to reinforce until they were large enough to collapse the wormhole. 

The reasoning does suggest a violent reaction to brute-force attempts to make time loops and severe complications in general. For instance, if neutrinos are indeed tachyons, many potential detections of them could engender subtle causal paradoxes. The wave function for those detections would cancel, and the detections would have zero probability of actually being observed. Neutrinos may be so elusively hard to detect exactly because they are tachyons! The slippery nature of time travel does not rule out carefully contrived logically consistent loops. Wave interference appears as banded patterns with a central zero-order constructive fringe, and so on. If causal loops create similar patterns of probability, halfhearted attempts at time travel, involving small perturbations of arrangements that show no time travel, are as doomed to failure as attempts to force electrons into the low-probability spaces between their atomic shells. Major alterations will be necessary to skip from zero-order situations without overt time travel to first-order setups that show it. 

In the realm of Exes, thought – and thus computation – will be the fabric of existence. Every possibility that furthers it will become a reality. In the 1980s David Deutsch of Oxford proposed both computers that exploit the many alternatives implied by their quantum wave function to do a kind of parallel computation and even more powerful ones that us time travel. Quantum computers are now a subject of serious research, and simple ones have been demonstrated. No one has yet demonstrated time travel, but the isolated, simple, coherence-preserving interactions inside future quantum computers may be the perfect stage for contriving nonparadoxical causal loops. Time travel may emerge from the closet, only to remain discreetly hidden inside quantum computations! 

It may be easiest to conceive of causal loops through the agency of negative-time-delay elements for computers – whose outputs predict their inputs. Such devices may simply be a matter of looking at conventional particle interactions in a time-reversed way; or perhaps they will use tachyons or wormholes. Maybe those mundane and exotic alternatives are actually equivalent, simply different ways of interpreting the same situation. In any case, negative-delay elements might fail spectacularly or dangerously if short-circuited, but, carefully harnessed, they might provide spectacular computational effects instead. 

While I must admit, I am not a physicist and I cannot speak to how realistic or consistent with contemporary physics Moravec’s predictions are, it is hard not to read these words without calling to mind Land’s characterization of capitalism as an invasion of artificial intelligence from the future in order to assemble itself from its enemies resources. One can also not help but draw a connection to the work of ufologist, Jacques Vallee who, in his Passport to Magonia, drew compelling connections between traditional accounts of encounters with faerie folk and modern accounts of encounters with UFOs and so-called extraterrestrials, leading him to question whether UFOs should be attributed to extraterrestrials at all. Perhaps, rather than extraterrestrials, these are, in fact, extratemporals. They may even be our distant ancestors projecting messages back in time to us, or perhaps even gathering information to the end of successfully emulating all past persons in the future. This might account for Vallee’s observations that these entities often seem to straddle the line between being genuine physical objects and something more ethereal. Perhaps the varying characteristics and dispositions attributed to such beings can be accounted for by positing that these entities represent different factions in some future “time war” such as is featured in Land’s works. At any rate, if artificial intelligence does allow us to interface with the very fabric of time itself in any way, there will no doubt be a number of practical and ethical concerns that will have to be taken into deep consideration.

This, however, is not the place to carry such speculations any further than they need to go. The point is simply to emphasize precisely how strange a world populated by superintelligent machines might become. That strangeness may be in the areas we have highlighted here, or they might arise in totally unexpected areas of life. If we are to have any hope of surviving and eking a reasonable agreeable existence for ourselves, we will have to greatly sharpen our wits and step out of our conventional ways of thinking. While it might not be necessary that everyone become a full time professional philosopher, it will almost certainly be necessary for the vast majority of the population to pick up a healthy appreciation of philosophy and, at the very least, a basic understanding of fundamental philosophical concepts and problems and their significance. This leads us to a problem that we have been putting off for some time now, namely that of education. There can be little doubt that our educational systems and institutions require drastic overhaul if they hope to even offer realistic solutions to our current problems. How much more must they be reformed in order to meet the needs of the future? However, perhaps such reform might be effective in solving several problems at once. Perhaps educational institutions can be transformed into crucial community hubs around which the order and organization needed to sustain tribal structures essential to effectively managing future robot corporations can begin to branch out and bloom. These are to be the topics we take up in another paper in a much nearer future than Moravec and Tipler describe.