by Stuart Armstrong 457 days ago | link | parent Suppose the humans have already decided whether to press the shutdown or order the AI to maximise paperclips. If $$o_s$$ is the observation of the shutdown command and $$o_p$$ the observation of the paperclip maximising command, and $$u_s$$ and $$u_p$$ the relevant utilities, then $$P$$ can be defined as $$P(u_s|h_{m-1}o_s)=1$$ and $$P(u_p|h_{m-1}o_p)=1$$, for all histories $$h_{m-1}$$. Then define $$\widehat{P}$$ as the probability of $$o_s$$ versus $$o_p$$, conditional on the fact that the agent follows a particular deterministic policy $$\pi^0$$. If the agent does indeed follow $$\pi^0$$, then $$\widehat{P}=\widehat{P}'$$. If it varies from this policy, then $$\widehat{P}'$$ is altered in proportion to the expected change in $$\widehat{P}$$ caused by choosing a different action.

### NEW DISCUSSION POSTS

If you drop the
 by Alex Appel on Distributed Cooperation | 1 like

Cool! I'm happy to see this
 by Abram Demski on Distributed Cooperation | 0 likes

Caveat: The version of EDT
 by 258 on In memoryless Cartesian environments, every UDT po... | 2 likes

[Delegative Reinforcement
 by Vadim Kosoy on Stable Pointers to Value II: Environmental Goals | 1 like

Intermediate update: The
 by Alex Appel on Further Progress on a Bayesian Version of Logical ... | 0 likes

Since Briggs [1] shows that
 by 258 on In memoryless Cartesian environments, every UDT po... | 2 likes

This doesn't quite work. The
 by Nisan Stiennon on Logical counterfactuals and differential privacy | 0 likes

I at first didn't understand
 by Sam Eisenstat on An Untrollable Mathematician | 1 like

This is somewhat related to
 by Vadim Kosoy on The set of Logical Inductors is not Convex | 0 likes

This uses logical inductors
 by Abram Demski on The set of Logical Inductors is not Convex | 0 likes

Nice writeup. Is one-boxing
 by Tom Everitt on Smoking Lesion Steelman II | 0 likes

Hi Alex! The definition of
 by Vadim Kosoy on Delegative Inverse Reinforcement Learning | 0 likes

A summary that might be
 by Alex Appel on Delegative Inverse Reinforcement Learning | 1 like

I don't believe that
 by Alex Appel on Delegative Inverse Reinforcement Learning | 0 likes

This is exactly the sort of
 by Stuart Armstrong on Being legible to other agents by committing to usi... | 0 likes