Posted on 16-2-2002

Forfeiting The Future
by Bill Joy from www.resurgence.org issue 208 (Photo shows Bill Joy)

ACCUSTOMED TO living with routine scientific break-throughs, we have yet
tocome to terms with the fact that the most compelling new technologies -
robotics, genetic engineering and nanotechnology - pose a different kind of
threat than the technologies that came before. Specifically, robots,
engineered organisms and nanobots share a dangerous amplifying factor: they
can self-replicate. A bomb is blown up only once, but one altered gene can
become many, and quickly get out of control.

While replication in a computer or a computer network can be a nuisance, at
worst it disables a machine or takes down a network or network service. But
self-replication in the new technologies runs a much greater risk: a risk
of substantial damage in the physical world.

Each of these new technologies also offers untold promise: the vision of
near immortality; genetic engineering that may soon provide treatments, if
not outright cures, for most diseases; and nanotechnology and nanomedicine
which can apparently address yet more ills. Together these technologies
could significantly extend our average life-span and improve the quality of
our lives. Yet, with each of these technologies, a sequence of small,
individually sensible advances leads to an accumulation of great power,
and, concomitantly, great danger.

What was different in the twentieth century? Certainly, the technologies
underlying the weapons of mass destruction - nuclear, biological and
chemical - were and are a powerful and enormous threat. But building
nuclear weapons required, at least for a time, access to both rare raw
materials and highly protected information.

The twenty-first-century technologies, however, are within the grasp of
individuals. They do not require large facilities and rare materials.
Knowledge alone will enable the use of them. Thus, we have the possibility
of not just weapons of mass destruction, but of knowledge-enabled mass
destruction, hugely amplified by the power of self-replication.

It is always hard to see the bigger impact of technology whilst in the
vortex of change, but failing to understand the consequences of our
inventions while we are in the rapture of discovery and innovation seems to
be a common fault of scientists and technologists. We have long been driven
by an overarching desire to know: that is the nature of science's quest,
not stopping to notice that the progress to newer and more powerful
technologies can take on a life of its own.

Because of the recent rapid and radical progress in molecular electronics
and related nanoscale technologies, by 2030 we are likely to be able to
build machines a million times as powerful as the personal computers of
today. As this enormous computing power is combined with the manipulative
advances of the physical sciences and the new, deep understandings in
genetics, enormous transformative power is being unleashed. These
combinations open up the opportunity to redesign the world completely, for
better or worse. The replicating and evolving processes that have been
confined to the natural world are about to become realms of human endeavour.

Given the incredible power of these new technologies, shouldn't we proceed
with great caution?

THE DREAM OF ROBOTICS is that intelligent machines can do our work for us,
allowing us lives of leisure, restoring us to Eden. How soon could such an
intelligent robot be built? The coming advances in computing power seem to
make it possible by 2030, and once an intelligent robot exists, it is only
a small step to a robot species - to an intelligent robot that can make
evolved copies of itself.

A second dream of robotics is that we will gradually replace ourselves with
our robotic technology, achieving near immortality by downloading our
consciousnesses. We are beginning to see intimations of this in the
implantation of computer devices into the human body. But if we are
downloaded into our technologies, what are the chances that we will
thereafter be ourselves or even human?

Genetic engineering promises to revolutionize agriculture by increasing
crop yields while reducing the use of pesticides; to create tens of
thousands of novel species of bacteria, plants, viruses and animals; to
replace reproduction, or supplement it, with cloning; to create cures for
many diseases, increasing our life-span. We know with certainty that the
profound changes in the biological sciences are imminent and will challenge
all our notions of what life is.

Technologies such as human cloning have, in particular, raised our
awareness of the profound ethical and moral issues we face. If, for
example, we were to re-engineer ourselves into several separate and unequal
species using the power of genetic engineering, then we would threaten the
notion of equality that is the very cornerstone of our democracy.

THE MANY WONDERS of nanotechnology were first imagined by the
Nobel-laureate physicist Richard Feynman in a speech in 1959, in which he
described how manipulation of matter at the atomic level could create a
utopian future of abundance, where just about everything could be made
cheaply, and almost any disease or physical problem could be solved using
nanotechnology and artificial intelligences.

Imagine some of the changes that might take place in a world where we had
molecular-level 'assemblers'. Assemblers could make possible incredibly
low-cost solar power, cure cancer and the common cold by augmentation of
the human immune system, could clean up the environment, create inexpensive
pocket supercomputers, and restore extinct species.
The enabling breakthrough to assemblers seems quite likely within the next
twenty years. Molecular electronics should mature quickly and become
enormously lucrative within this decade, causing a large incremental
investment in all nanotechnologies.

But we can't simply do our science and not worry about the ethical issues.
Unfortunately, as with nuclear technology, it is far easier to create
destructive uses of nanotechnology than constructive ones. Nanotechnology
has clear military and terrorist uses, and you need not be suicidal to
release a massively destructive nanotechnological device: such devices
could be built to be selectively destructive, affecting for example, only a
certain geographical area or a group of people who are genetically distinct.

An immediate consequence of the Faustian bargain in obtaining the great
power of nanotechnology is that we run a grave risk - the risk that we
might destroy the biosphere on which all life depends. For example, as Eric
Drexler explained in his book, Engines of Creation: "'Plants' with 'leaves'
no more efficient than today's solar cells could out-compete real plants,
crowding the biosphere with an inedible foliage. Tough omnivorous
'bacteria' could out-compete real bacteria; they could spread like pollen
in the wind, replicating swiftly and reducing the biosphere to dust in a
matter of days.

Dangerous replicators could easily be too rapidly-spreading, tough and
small to stop. We have trouble enough controlling viruses and fruit flies.
We cannot afford these kinds of accident with self-replicating assemblers."

THESE POSSIBILITIES ARE all undesirable. The only realistic alternative is
relinquishment: to limit development of the technologies that are too
dangerous, by limiting our pursuit of certain kinds of knowledge. Although
humankind inherently 'desires to know', if open access to, and unlimited
development of, knowledge henceforth puts us all in clear danger of
extinction, then common sense demands that we re-examine our reverence for
knowledge.

If we could agree, as a species, what we wanted, where we were headed and
why, then we could make our future much less dangerous - then we might
understand what we could and should relinquish. If the course of humanity
could be determined by our collective values, ethics and morals, and if we
had gained more collective wisdom over the past few thousand years, then a
dialogue to this end would be practical, and the incredible powers that we
are about to unleash would not be nearly so troubling.

One would think that we might be driven to such a dialogue by our instinct
for self-preservation. Individuals clearly have this desire, yet as a
species our behaviour seems not to be in our favour. The new Pandora's
boxes of genetics, nanotechnology and robotics are almost open, yet we seem
hardly to have noticed. Ideas can't be put back in a box: unlike uranium or
plutonium, they don't need to be mined and refined; they can be freely
copied. Once they are out, they are out.

Verifying relinquishment will be a difficult problem, but not an unsolvable
one. We are fortunate to have already done a lot of relevant work in the
context of the Biological Weapons Convention and other treaties. Verifying
compliance will also require that scientists, technologists and engineers
adopt a strong code of ethical conduct, resembling the Hippocratic oath,
that they cease and desist from work creating, developing and manufacturing
knowledge-enabled technologies of mass destruction.

Where can we look for a new ethical basis to set our course? We would do
well to consider a new book by His Holiness the Dalai Lama called Ethics
for the New Millennium. As is perhaps well-known but little-heeded, the
Dalai Lama argues that the most important thing is for us to conduct our
lives with love and compassion for others, and that our societies need to
develop a stronger notion of universal responsibility and of our
interdependency. He proposes a standard of positive ethical conduct for
individuals and societies and further argues that we must understand what
it is that makes people happy, and acknowledge the strong evidence that
neither material progress nor the pursuit of the power of knowledge is the
key - that there are limits to what science and the scientific pursuit
alone can do. o

A full-length version of this article first appeared in the April 2000
edition of Wired magazine.

Bill Joy is co-founder and Chief Scientist of Sun Micro-systems and was
Co-Chair of the Presidential Commission on the Future of IT Research.