Intelligent Agent Foundations Forumsign up / log in
Acausal trade: being unusual
discussion post by Stuart Armstrong 67 days ago | discuss

A putative new idea for AI control; index here.

Other posts in the series: Introduction, Double decrease, Pre-existence deals, Full decision algorithms, Breaking acausal trade, Trade in different types of utility functions, Being unusual, and Summary.

It’s important to remember that, in acausal trade, the other agents truly know nothing about you. They have some statistical image about the initial conditions throughout the universe, the probability of life evolving, they’ve maybe run some simulations: but they truly have nothing beyond that. Nor do you have any idea about them.

What that means is that if you are a particularly unusual entity, then no causally disconnected agent knows that. You can’t update them on the fact that you exist, and that they should therefore trade more with you.

(Or maybe you’re an agent of type \(A_1\) and agents of type \(A_1\) are either almost infinitely rare (with high probability) or somewhat more common, independently of other agent types. Based on the fact that you exist, you can update towards the fact that \(A_1\) are more common - but only \(A_1\)-type agents will be able to do this update, so those are the only agents you’ll have much trade with.)

So unusual agents are less likely to benefit from acausal trade, but they’re also less pressured to follow it. Suppose that there are \(\Omega\) humans, all maximising the same \(u\), and one will be designated utility-dictator.

By shear coincidence, your decision theory is disconnected from that of the rest of the humans, and you’ve been designated utility dictator.

Any causally separated agent doesn’t know this, and so you can reach your decision while making an \(1/\Omega\) difference to how they deal with humanity. In many cases, it can be rational to simply defect from acausal trade entirely.

If you have a different utility function from the other humans, all the more reason to defect, because you get almost no benefit from cooperation.

Of course, a sensible human community will resolve its internal differences before designating a utility-dictator. And the community can’t plan to benefit from this loophole, since that would mean you being designated is not so unlikely after all.

Things might be different if you think you might be in a simulation, but unusual agents aren’t very informative in simulations either.

Seeing all the other potential barriers and difficulties with acausal trade, it possible, if every existing agent is unusual enough and have distinct enough utilities, that no acausal trade will happen at all.



NEW LINKS

NEW POSTS

NEW DISCUSSION POSTS

RECENT COMMENTS

A few thoughts: I agree
by Sam Eisenstat on Some Criticisms of the Logical Induction paper | 0 likes

Thanks, so to paraphrase your
by Wei Dai on Current thoughts on Paul Christano's research agen... | 0 likes

> Why does Paul think that
by Paul Christiano on Current thoughts on Paul Christano's research agen... | 0 likes

Given that ALBA was not meant
by Wei Dai on Current thoughts on Paul Christano's research agen... | 0 likes

Thank you for writing this.
by Wei Dai on Current thoughts on Paul Christano's research agen... | 1 like

I mostly agree with this
by Paul Christiano on Current thoughts on Paul Christano's research agen... | 2 likes

>From my perspective, I don’t
by Johannes Treutlein on Smoking Lesion Steelman | 2 likes

Replying to Rob. I don't
by Vadim Kosoy on Some Criticisms of the Logical Induction paper | 0 likes

Replying to Rob. Actually,
by Vadim Kosoy on Some Criticisms of the Logical Induction paper | 0 likes

Replying to 240 (I can't
by Vadim Kosoy on Some Criticisms of the Logical Induction paper | 0 likes

Yeah, you're right. This
by Vadim Kosoy on Smoking Lesion Steelman | 1 like

The non-smoke-loving agents
by Abram Demski on Smoking Lesion Steelman | 1 like

Replying to "240" First,
by Vadim Kosoy on Some Criticisms of the Logical Induction paper | 0 likes

Clarification: I'm not the
by Tarn Somervell Fletcher on Some Criticisms of the Logical Induction paper | 0 likes

Alex, the difference between
by Vadim Kosoy on Some Criticisms of the Logical Induction paper | 1 like

RSS

Privacy & Terms