Intelligent Agent Foundations Forumsign up / log in
Transitive negotiations with counterfactual agents
discussion post by Scott Garrabrant 1005 days ago | Jessica Taylor, Patrick LaVictoire and Tsvi Benson-Tilsen like this | discuss

The main purpose of this post is to make the following minor observation: If an agents only acausally trade with other agents that they believe actually possibly exist, they can still end up effectively trading with agents that they know to be counterfactual. This is possible because the agent can believe there exists a middle man who believes that the counterfactual agent might exist. This may have consequences bargaining and decision theory, especially in cases like the counterfactual mugging.

Consider a counterfactual mugging problem. Agent \(A\) knows that the millionth digit of pi is a odd, and is in a counterfactual mugging problem. If Omega predicts that agent \(A\) will pay 1 dollar if the millionth digit of pi is odd, then agent \(B\) will receive 1000 dollars if the millionth digit of pi is even. I (nonstandardly) think of \(A\) and \(B\) as two different agents. \(A\) contains in memory a proof that the millionth digit of pi is odd, while \(B\) contains in memory a proof that the millionth digit of pi is even. Note that agent \(B\) does not exist, and is not even simulated by Omega, since the digit is in fact odd.

Agent \(B\) would really like agent \(A\) to pay the dollar, and may be willing to engage in acausal trade with agent \(A\). However agent \(A\) knows that agent \(B\) does not exist, so there is not much that agent \(B\) can offer \(A\).

Now, consider a third agent \(C\) that is uncertain about the millionth digit of pi. This agent believes that both of \(A\) and \(B\) have a 50% chance to exist, and both \(A\) and \(B\) know that \(C\) exists. \(B\) can acausally offer a trade to \(C\) in which \(C\) agrees to represent \(B\) in negotiations with \(A\) (for a price). Then \(C\) can represent \(B\) by trading with \(A\) to get \(A\) to pay a dollar.

(In one special case \(C\) is an earlier agent that becomes either agent \(A\) or \(B\) when it computes the digit of pi. In this case, perhaps negotiations are easier, since the three agents have the same goals.)



NEW LINKS

NEW POSTS

NEW DISCUSSION POSTS

RECENT COMMENTS

[Note: This comment is three
by Ryan Carey on A brief note on factoring out certain variables | 0 likes

There should be a chat icon
by Alex Mennen on Meta: IAFF vs LessWrong | 0 likes

Apparently "You must be
by Jessica Taylor on Meta: IAFF vs LessWrong | 1 like

There is a replacement for
by Alex Mennen on Meta: IAFF vs LessWrong | 1 like

Regarding the physical
by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think that we should expect
by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think I understand your
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

This seems like a hack. The
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

After thinking some more,
by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yes, I think that we're
by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

My intuition is that it must
by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

To first approximation, a
by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Actually, I *am* including
by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yeah, when I went back and
by Alex Appel on Optimal and Causal Counterfactual Worlds | 0 likes

> Well, we could give up on
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

RSS

Privacy & Terms