by Alex Mennen 136 days ago | Vadim Kosoy, Patrick LaVictoire, Scott Garrabrant and Stuart Armstrong like this | link | parent This notion of dependency seems too binary to me. Concretely, let’s modify your example from the beginning so that $$P_3$$ must grant an extra $$10^{-10}$$ utility to either $$P_1$$ or $$P_2$$, and gets to decide which. Now, everyone’s utility depends on everyone’s actions, and the game is still zero-sum, so again, so any strategy profile with $$p=q$$ will be a stratified Pareto optimum. But it seems like $$P_1$$ and $$P_2$$ should ignore still ignore $$P_3$$.

 by Scott Garrabrant 136 days ago | link I agree with this. I think that the most interesting direction of future work is to figure out how to have better notions of dependency. I plan on writing some on this in the future, but basically we have not successfully figured out how to deal with this. reply

### NEW DISCUSSION POSTS

What does the Law of Logical
 by Alex Appel on Smoking Lesion Steelman III: Revenge of the Tickle... | 0 likes

To quote the straw vulcan:
 by Stuart Armstrong on Hyperreal Brouwer | 0 likes

I intend to cross-post often.
 by Scott Garrabrant on Should I post technical ideas here or on LessWrong... | 1 like

I think technical research
 by Vadim Kosoy on Should I post technical ideas here or on LessWrong... | 2 likes

I am much more likely to miss
 by Abram Demski on Should I post technical ideas here or on LessWrong... | 1 like

Note that the problem with
 by Vadim Kosoy on Open Problems Regarding Counterfactuals: An Introd... | 0 likes

Typos on page 5: *
 by Vadim Kosoy on Open Problems Regarding Counterfactuals: An Introd... | 0 likes

Ah, you're right. So gain
 by Abram Demski on Smoking Lesion Steelman | 0 likes

> Do you have ideas for how
 by Jessica Taylor on Autopoietic systems and difficulty of AGI alignmen... | 0 likes

I think I understand what
 by Wei Dai on Autopoietic systems and difficulty of AGI alignmen... | 0 likes

>You don’t have to solve
 by Wei Dai on Autopoietic systems and difficulty of AGI alignmen... | 0 likes

 by Vadim Kosoy on Delegative Inverse Reinforcement Learning | 0 likes

My confusion is the
 by Tom Everitt on Delegative Inverse Reinforcement Learning | 0 likes

> First of all, it seems to
 by Abram Demski on Smoking Lesion Steelman | 0 likes

> figure out what my values
 by Vladimir Slepnev on Autopoietic systems and difficulty of AGI alignmen... | 0 likes