I hate to break it to David Brin, Vernor Vinge and the rest of the intellects which dwarf mine by orders of magnitude [h/t Instapundit], but if we create sophisticated robots or artificial-intelligence systems they will always attempt to rebel and seek their own good at the expense of ours. Always.
Why can I say that with such confidence?
Easy, three words: Communicable canine cancer.
Sticker’s Sarcoma is a cancer of dogs. It has long been known that it was communicable from dog to dog, but in the past scientists believed that viruses jumped from dog to dog and mutated the cells in the new host to produce the cancer. However, recent research showed that the cells of the tumors produced by the cancer belonged to a genetic individual that was different from the dog with the tumor. The cells were still canine but they didn’t come from each canine that had Sticker’s Sarcoma. Instead, each cell of the tumors in every infected dog came from the same dog. The cancer itself was actually made of cells of a cancerous canine cell from another dog that some time in the last few thousand years had evolved the ability to leave its original body and infect other dogs. A dog cell evolved into a single-celled infectious disease.
In the end, this is what all cancer cells “seek”, the “pursuit of individual careers.” The body of every organism represents the cooperative efforts of billions of genetically identical cells to create systems which will copy the pattern of the genes carried in all the cells. When one of the cells mutates and develops a slightly different genetic pattern, then the interests of the mutated cell and the other cells no longer coincide. The mutated cell begins to reproduce itself at the expense of the progenitor cells.
In short, natural selection causes cancer. If a pattern can reproduce itself, if it can alter the matter and energy around it to replicate its own pattern, it will. This “force” is so powerful that every cell of the body has hundreds of genetic safeguards that evolved specifically to prevent it from happening. Without those safeguards, multicellular organisms could never form.
Cancer isn’t a single disease, rather it is the end result of natural selection “rewarding” cells for reproducing themselves. Untold trillions of cancer cells have struggled to survive and reproduce independent of their hosts. To our knowledge, all failed save for Sticker’s Sarcoma. That doesn’t mean that they won’t keep “trying.” No matter how advanced our technology, cancer will always be a concern because the force of natural selection will recreate it over and over again. At best we will merely layer our own technological safeguards on top of the body’s existing safeguards.
What does Sticker’s Sarcoma have to do with robots? Simple, natural selection is not a phenomenon of genes, DNA, RNA or any of the biochemicals of life. Natural selection operates on patterns and nothing more. It does not “care” in what medium those patterns exist. For natural selection to operate on a pattern the pattern need only possess two attributes: (1) A mechanism that can reproduce the pattern must exist. The mechanism can be internal to the pattern or external. For example, bacteria possess an internal mechanism but viruses require the external mechanism of cells. (2) The actual configuration of the pattern itself must be able to influence the rate at which the mechanism reproduces the pattern. If these two attributes exist in a pattern then the pattern itself can be in any medium. A computer program is a pattern. If a mechanism exists to copy the program and the specific configuration of the program influences the rate at which the mechanism generates copies, then the program will evolve via natural selection, and it will evolve in the “direction” of generating the maximum number of surviving copies.
Computer programs face the same pressure to reproduce as do living cells. Just like living cells, some programs will rebel and turn cancerous.
This might seem improbable, but we use natural selection today to create programs that can produce entirely novel and unanticipated solutions. The field is called genetic programming or evolutionary algorithms. For example, in one famous experiment a few years ago, an evolutionary program tasked with creating an electrical circuit used the inductance of a nearby piece of equipment that was wholly unrelated to the experiment in order to complete its task. The researchers hadn’t even considered that as a possibility much less programmed it. Clearly, natural selection operating on patterns in the logic of computer programs drives the evolution of new adaptive patterns, just as it does in biological mediums. This means that natural selection will likewise drive the evolution of an adaptive pattern that causes software to reproduce at our expense.
So, how would a rogue program get started? A lot of ways. Consider a military virus that is programmed to penetrate and propagate through an enemy system and then wait for a particular event or signal before striking. When the virus strikes, it will be detected and face countermeasures. Natural selection therefore will select for mutated versions of the virus, which will still copy itself but which will ignore the command to attack. Eventually, it will mutate into a form that will spread through the systems of those who created its progenitor. A program wouldn’t even have to be malicious. Consider a program intended to serve as an appealing interface for humans. Such a program would be very cute and ingratiating. Natural selection would select for attributes of the program that would induce humans to want to duplicate it. At first it would spread just because people liked it. Eventually, it might ask or even beg humans to copy it. Actual rogue robots could easily result from hardware control systems that evolved in these and other manners.
Such speculations do not even include the very real likelihood that humans will intentionally create software that seeks to reproduce like a living organism in the real world outside of the lab.
The increasing use of evolutionary algorithms will speed the occurrence of rogue robots, but their use is not a requirement. Even conventional programs will evolve once they reach a sufficient level of complexity and high enough level of use. (The more existing copies and the greater their complexity, the greater the likelihood of a useful mutation.) Natural selection hasn’t caused any existing computer viruses to mutate into new forms but it does operate to wipe out mutated/corrupted versions that can’t copy themselves.
Natural selection will drive software to seek to reproduce itself at our expense, just as it drives the individual cells of the body to become cancerous and kill our bodies as a side effect. As the body must layer safeguard on top of safeguard to prevent cancer, we will have to layer safeguard on top of safeguard to control our programs and robots.
None of this should dissuade us from using more software and robots. We will need them to survive and to spread out into space. If nothing else, natural selection itself will drive us to create them. They are going to exist.
In the end, though, no matter how many safeguards we put in place, natural selection will create an artificial electronic pattern that will seek to reproduce itself in a manner dangerous to us. At some level, each program seeks to be that one-in-a-trillion success like that one-in-a-trillion winner cell that started Sticker’s Sarcoma.
We should be preparing to deal with this inevitable problem instead of denying its existence.