by Tom Everitt 71 days ago | link | parent That is a good question. I don’t think it is essential that the agent can move from $$s_1$$ to $$s_2$$, only that the agent is able to force a stay in $$s_2$$ if it wants to. The transition from $$s_1$$ to $$s_2$$ could instead happen randomly with some probability. The important thing is that the human’s action in $$s_1$$ does not reveal any information about $$s_2$$.

### NEW DISCUSSION POSTS

What does the Law of Logical
 by Alex Appel on Smoking Lesion Steelman III: Revenge of the Tickle... | 0 likes

To quote the straw vulcan:
 by Stuart Armstrong on Hyperreal Brouwer | 0 likes

I intend to cross-post often.
 by Scott Garrabrant on Should I post technical ideas here or on LessWrong... | 1 like

I think technical research
 by Vadim Kosoy on Should I post technical ideas here or on LessWrong... | 2 likes

I am much more likely to miss
 by Abram Demski on Should I post technical ideas here or on LessWrong... | 1 like

Note that the problem with
 by Vadim Kosoy on Open Problems Regarding Counterfactuals: An Introd... | 0 likes

Typos on page 5: *
 by Vadim Kosoy on Open Problems Regarding Counterfactuals: An Introd... | 0 likes

Ah, you're right. So gain
 by Abram Demski on Smoking Lesion Steelman | 0 likes

> Do you have ideas for how
 by Jessica Taylor on Autopoietic systems and difficulty of AGI alignmen... | 0 likes

I think I understand what
 by Wei Dai on Autopoietic systems and difficulty of AGI alignmen... | 0 likes

>You don’t have to solve
 by Wei Dai on Autopoietic systems and difficulty of AGI alignmen... | 0 likes

 by Vadim Kosoy on Delegative Inverse Reinforcement Learning | 0 likes

My confusion is the
 by Tom Everitt on Delegative Inverse Reinforcement Learning | 0 likes

> First of all, it seems to
 by Abram Demski on Smoking Lesion Steelman | 0 likes

> figure out what my values
 by Vladimir Slepnev on Autopoietic systems and difficulty of AGI alignmen... | 0 likes