Intelligent Agent Foundations Forumsign up / log in
by Jessica Taylor 20 days ago | link | parent

Counterfactual mugging doesn’t require spoofing. Consider the following problem:

Suppose no one, given \(10^{5}\) steps of computation, is able to compute any information about the parity of the \(10^{10}\)th digit of \(\pi\), and everyone, given \(10^{100}\) steps of computation, is able to compute the \(10^{10}\)th digit of \(\pi\). Suppose that at time \(t\), everyone has \(10^5\) steps of computation, and at a later time \(t'\), everyone has \(10^{100}\) steps of computation. At the initial time \(t\), Omega selects a probability \(p\) equal to the conditional probability Omega assigns to the agent paying $1 at time \(t'\) conditional on the digit being odd. (This could be because Omega is a logical inductor, or because Omega is a CDT agent whose utility function is such that selecting this value of \(p\) is optimal). At time \(t'\), if the digit is even, a fair coin with probability \(p\) of coming up heads is flipped, and if it comes up heads, Omega pays the agent $10. If instead the digit is odd, then the agent has the option of paying Omega $1.

This contains no spoofing, and the optimal policy for the agent is to pay up if asked.





Note: I currently think that
by Jessica Taylor on Predicting HCH using expert advice | 0 likes

Counterfactual mugging
by Jessica Taylor on Doubts about Updatelessness | 0 likes

What do you mean by "in full
by David Krueger on Doubts about Updatelessness | 0 likes

It seems relatively plausible
by Paul Christiano on Maximally efficient agents will probably have an a... | 1 like

I think that in that case,
by Alex Appel on Smoking Lesion Steelman | 1 like

Two minor comments. First,
by Sam Eisenstat on No Constant Distribution Can be a Logical Inductor | 1 like

A: While that is a really
by Alex Appel on Musings on Exploration | 0 likes

> The true reason to do
by Jessica Taylor on Musings on Exploration | 0 likes

A few comments. Traps are
by Vadim Kosoy on Musings on Exploration | 1 like

I'm not convinced exploration
by Abram Demski on Musings on Exploration | 0 likes

Update: This isn't really an
by Alex Appel on A Difficulty With Density-Zero Exploration | 0 likes

If you drop the
by Alex Appel on Distributed Cooperation | 1 like

Cool! I'm happy to see this
by Abram Demski on Distributed Cooperation | 0 likes

Caveat: The version of EDT
by 258 on In memoryless Cartesian environments, every UDT po... | 2 likes

[Delegative Reinforcement
by Vadim Kosoy on Stable Pointers to Value II: Environmental Goals | 1 like


Privacy & Terms