Transitive negotiations with counterfactual agents discussion post by Scott Garrabrant 1005 days ago | Jessica Taylor, Patrick LaVictoire and Tsvi Benson-Tilsen like this | discuss The main purpose of this post is to make the following minor observation: If an agents only acausally trade with other agents that they believe actually possibly exist, they can still end up effectively trading with agents that they know to be counterfactual. This is possible because the agent can believe there exists a middle man who believes that the counterfactual agent might exist. This may have consequences bargaining and decision theory, especially in cases like the counterfactual mugging. Consider a counterfactual mugging problem. Agent $$A$$ knows that the millionth digit of pi is a odd, and is in a counterfactual mugging problem. If Omega predicts that agent $$A$$ will pay 1 dollar if the millionth digit of pi is odd, then agent $$B$$ will receive 1000 dollars if the millionth digit of pi is even. I (nonstandardly) think of $$A$$ and $$B$$ as two different agents. $$A$$ contains in memory a proof that the millionth digit of pi is odd, while $$B$$ contains in memory a proof that the millionth digit of pi is even. Note that agent $$B$$ does not exist, and is not even simulated by Omega, since the digit is in fact odd. Agent $$B$$ would really like agent $$A$$ to pay the dollar, and may be willing to engage in acausal trade with agent $$A$$. However agent $$A$$ knows that agent $$B$$ does not exist, so there is not much that agent $$B$$ can offer $$A$$. Now, consider a third agent $$C$$ that is uncertain about the millionth digit of pi. This agent believes that both of $$A$$ and $$B$$ have a 50% chance to exist, and both $$A$$ and $$B$$ know that $$C$$ exists. $$B$$ can acausally offer a trade to $$C$$ in which $$C$$ agrees to represent $$B$$ in negotiations with $$A$$ (for a price). Then $$C$$ can represent $$B$$ by trading with $$A$$ to get $$A$$ to pay a dollar. (In one special case $$C$$ is an earlier agent that becomes either agent $$A$$ or $$B$$ when it computes the digit of pi. In this case, perhaps negotiations are easier, since the three agents have the same goals.)

### NEW DISCUSSION POSTS

[Note: This comment is three
 by Ryan Carey on A brief note on factoring out certain variables | 0 likes

There should be a chat icon
 by Alex Mennen on Meta: IAFF vs LessWrong | 0 likes

Apparently "You must be
 by Jessica Taylor on Meta: IAFF vs LessWrong | 1 like

There is a replacement for
 by Alex Mennen on Meta: IAFF vs LessWrong | 1 like

Regarding the physical
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think that we should expect
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think I understand your
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

This seems like a hack. The
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

After thinking some more,
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yes, I think that we're
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

My intuition is that it must
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

To first approximation, a
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Actually, I *am* including
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yeah, when I went back and
 by Alex Appel on Optimal and Causal Counterfactual Worlds | 0 likes

> Well, we could give up on
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes