ABlack Marble is a technology that by default destroys the civilization that invents it. It's one type of Existential Risk. AGI may be such an invention, but isn't the only one.
A technology that by default destroys the civilization that invents it. It's one type of Existential Risk. AGI may be such an invention., but isn't the only one.
As a hypothetical example, Bostrom asked what would happen to civilization if a weapon of mass destruction comparable to an atomic bomb was much easier to make, on the level of cooking sand in a microwave or something like that. The natural answer being, once knowledge of the technique spreads, any random psychopath can do it, and some do so, and soon thereafter bomb us back into the stone age. Civilization then can't rebuild past the point of microwaves without getting destroyed again, as long as the knowledge persists. But this scenario is path dependent. One could imagine a different civilization with different capabilities that could survive such knowledge. Perhaps one with a world government (no wars) and a screening for psychopathy. Forpsychopathy, etc. Perhaps a dystopian world panopticon could prevent use. Or, for a space-faring civilization that mostly lives in small independent orbital colonies, everybody already has (and civilization is somehow currently surviving) similarly destructive kinetic attack capabilities so maybe the sand-nukes don't change much.
As a hypothetical example, Bostrom asked what would happen to civilization if a weapon of mass destruction comparable to an atomic bomb was much easier to make, on the level of cooking sand in a microwave or something like that. The natural answer being, once knowledge of the technique spreads, any random psychopath can do it, and some do so, and soon thereafter bomb us back into the stone age. Civilization then can't rebuild past the point of microwaves without getting destroyed again, as long as the knowledge persists. But this scenario is path dependent. One could imagine a different civilization with different capabilities that could survive such knowledge. Perhaps one with a world government (no wars) and a screening for psychopathy. For a space-faring civilization that mostly lives in small independent orbital colonies, everybody already has (and is somehow currently surviving) similarly destructive kinetic attack capabilities so maybe the sand-nukes don't change much.
The name comes from a thoughtexperiment by Nick Bostrom,Bostrom, where he described inventions as pulling marbles out of an urn. Most are white (beneficial), some are dangerous or a mixed blessing (usually described as gray or red balls), and some are black (fatal).
A technology that by default destroys the civilization that invents it. It's one type of existential risk. Existential Risk. AGI may be such an invention.invention.
The name comes from a thought experiment by Nick Bostrom, where he described inventions as pulling marbles out of an urn. Most are white (beneficial), some are dangerous or a mixed blessing,blessing (usually described as gray or red balls), and some are black (fatal).
As a hypothetical example, Bostrom asked what would happen to civilization if a weapon of mass destruction comparable to an atomic bomb was much easier to make, on the level of cooking sand in a microwave or something like that. The natural answer being, once knowledge of the technique spreads, any random psychopath can do it, and some do so, and soon thereafter bomb us back into the stone age. Civilization then can't rebuild past the point of microwaves without getting destroyed again, as long as the knowledge
persists.persists (making black marbles an extreme type of information hazard). But this scenario is path dependent. One could imagine a different civilization with different capabilities that could survive such knowledge. Perhaps one with a world government (no wars) and a screening for psychopathy, etc. Perhaps a dystopian world panopticon could prevent use. Or, for a space-faring civilization that mostly lives in small independent orbital colonies, everybody already has (and civilization is somehow currently surviving) similarly destructive kinetic attack capabilities so maybe the sand-nukes don't change much.