Intelligent Agent Foundations Forumsign up / log in
An impossibility result for doing without good priors
discussion post by Jessica Taylor 640 days ago | Stuart Armstrong likes this | discuss

(edit: this post is subsumed by this one)

My previous post suggested a strategy for making bets on uncertain propositions (e.g. the color of the sun) without having a good prior. Here I present an impossibility result for doing this in general.


Say there are \(n\) agents, and \(n-1\) possible colors of the sun. Before observing the sun’s color, each agent is only able to pick a single color of solar panel to buy (each color solar panel only produces power for a single possible sun color). The sun is revealed, and then the universe is split among the agents who bought the right color solar panel; if no one bought the right color then it is split evenly. Assume all agents have utility linear in the share of the universe they control.

Each agent from 1 to \(n-1\) believes with certainty that the sun is a color equal to the agent’s index (e.g. agent 2 believes the sun is color 2). Obviously, each of these agents the right color solar panel for the sun color they predict.

The \(n\)th agent is agnostic about what color the sun will be, and would like to ensure that in expectation they get a significant fraction of the universe regardless of the color of the sun.

Unfortunately, agent \(n\) cannot guarantee more than a \(\frac{1}{2(n-1)}\) fraction of the universe in expectation across possible sun colors (whereas they “should” be able to get a \(\frac{1}{n}\) fraction according to the logic in my previous post). This is because, whenever this agent picks the right color solar panel, they will only get half of the universe (sharing with the agent whose index corresponds to the sun’s color). So the best agent \(n\) can do is to pick a uniformly random color solar panel, yielding an expected \(\frac{1}{2(n-1)}\) fraction of the universe for each possible sun color.

I’m pretty sure that allowing agents to make enforceable contracts with each other doesn’t fix the problem. The issue is that agent \(n\) starts out from a pretty bad bargaining position, having very little to offer the other agents.

If the agent could buy a mixture of different solar panel colors, and the universe were divided proportionally to power generation, then the agent could spend \(\frac{1}{n-1}\) of its resources on each color solar panel, yielding a \(\frac{1}{n}\) fraction of the universe in all possible worlds. But IT does not seem realistic to assume that actions like this are always available.

I’m not sure what to do about this. It might be a deal-breaker for this whole approach, or it might turn out that there is some realistic assumption under which an agnostic agent can guarantee a \(\frac{1}{n}\) expected fraction of the universe.

One thing to note about this example is that the other agents have anticorrelated beliefs. If the other agents’ beliefs were instead i.i.d. from some distribution over possible beliefs (e.g. some Dirichlet distribution), then the agnostic agent can successfully guarantee a \(\frac{1}{n}\) expected fraction of the universe by sampling beliefs from this distribution and then buying the color solar panel that an agent with these beliefs would buy (by symmetry). It is unclear how realistic this kind of assumption is.



NEW LINKS

NEW POSTS

NEW DISCUSSION POSTS

RECENT COMMENTS

[Note: This comment is three
by Ryan Carey on A brief note on factoring out certain variables | 0 likes

There should be a chat icon
by Alex Mennen on Meta: IAFF vs LessWrong | 0 likes

Apparently "You must be
by Jessica Taylor on Meta: IAFF vs LessWrong | 1 like

There is a replacement for
by Alex Mennen on Meta: IAFF vs LessWrong | 1 like

Regarding the physical
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think that we should expect
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think I understand your
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

This seems like a hack. The
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

After thinking some more,
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yes, I think that we're
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

My intuition is that it must
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

To first approximation, a
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Actually, I *am* including
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yeah, when I went back and
by Alex Appel on Optimal and Causal Counterfactual Worlds | 0 likes

> Well, we could give up on
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

RSS

Privacy & Terms