by Tom Everitt 9 days ago | link | parent That is a good question. I don’t think it is essential that the agent can move from $$s_1$$ to $$s_2$$, only that the agent is able to force a stay in $$s_2$$ if it wants to. The transition from $$s_1$$ to $$s_2$$ could instead happen randomly with some probability. The important thing is that the human’s action in $$s_1$$ does not reveal any information about $$s_2$$.

### NEW DISCUSSION POSTS

I have stopped working on
 by Scott Garrabrant on Cooperative Oracles: Introduction | 0 likes

 by Vadim Kosoy on Delegative Inverse Reinforcement Learning | 0 likes

So this requires the agent's
 by Tom Everitt on Delegative Inverse Reinforcement Learning | 0 likes

If the agent always delegates
 by Vadim Kosoy on Delegative Inverse Reinforcement Learning | 0 likes

 by Tom Everitt on Delegative Inverse Reinforcement Learning | 0 likes

Hi Tom! There is a
 by Vadim Kosoy on Delegative Inverse Reinforcement Learning | 0 likes

Hi Alex! I agree that the
 by Vadim Kosoy on Cooperative Oracles: Stratified Pareto Optima and ... | 0 likes

That is a good question. I
 by Tom Everitt on CIRL Wireheading | 0 likes

 by Tom Everitt on CIRL Wireheading | 0 likes

"The use of an advisor allows
 by Tom Everitt on Delegative Inverse Reinforcement Learning | 0 likes

 by Wei Dai on Current thoughts on Paul Christano's research agen... | 0 likes

Suppose that I, Paul, use a
 by Paul Christiano on Current thoughts on Paul Christano's research agen... | 0 likes

When you wrote "suppose I use
 by Wei Dai on Current thoughts on Paul Christano's research agen... | 0 likes

> but that kind of white-box
 by Paul Christiano on Current thoughts on Paul Christano's research agen... | 0 likes

>Competence can be an
 by Wei Dai on Current thoughts on Paul Christano's research agen... | 0 likes