by Alex Mennen 74 days ago | Vadim Kosoy, Patrick LaVictoire, Scott Garrabrant and Stuart Armstrong like this | link | parent This notion of dependency seems too binary to me. Concretely, let’s modify your example from the beginning so that $$P_3$$ must grant an extra $$10^{-10}$$ utility to either $$P_1$$ or $$P_2$$, and gets to decide which. Now, everyone’s utility depends on everyone’s actions, and the game is still zero-sum, so again, so any strategy profile with $$p=q$$ will be a stratified Pareto optimum. But it seems like $$P_1$$ and $$P_2$$ should ignore still ignore $$P_3$$.

 by Scott Garrabrant 74 days ago | link I agree with this. I think that the most interesting direction of future work is to figure out how to have better notions of dependency. I plan on writing some on this in the future, but basically we have not successfully figured out how to deal with this. reply

### NEW DISCUSSION POSTS

I have stopped working on
 by Scott Garrabrant on Cooperative Oracles: Introduction | 0 likes

 by Vadim Kosoy on Delegative Inverse Reinforcement Learning | 0 likes

So this requires the agent's
 by Tom Everitt on Delegative Inverse Reinforcement Learning | 0 likes

If the agent always delegates
 by Vadim Kosoy on Delegative Inverse Reinforcement Learning | 0 likes

 by Tom Everitt on Delegative Inverse Reinforcement Learning | 0 likes

Hi Tom! There is a
 by Vadim Kosoy on Delegative Inverse Reinforcement Learning | 0 likes

Hi Alex! I agree that the
 by Vadim Kosoy on Cooperative Oracles: Stratified Pareto Optima and ... | 0 likes

That is a good question. I
 by Tom Everitt on CIRL Wireheading | 0 likes

 by Tom Everitt on CIRL Wireheading | 0 likes

"The use of an advisor allows
 by Tom Everitt on Delegative Inverse Reinforcement Learning | 0 likes

 by Wei Dai on Current thoughts on Paul Christano's research agen... | 0 likes

Suppose that I, Paul, use a
 by Paul Christiano on Current thoughts on Paul Christano's research agen... | 0 likes

When you wrote "suppose I use
 by Wei Dai on Current thoughts on Paul Christano's research agen... | 0 likes

> but that kind of white-box
 by Paul Christiano on Current thoughts on Paul Christano's research agen... | 0 likes

>Competence can be an
 by Wei Dai on Current thoughts on Paul Christano's research agen... | 0 likes