Intelligent Agent Foundations Forumsign up / log in
Logical Updatelessness as a Robust Delegation Problem
post by Scott Garrabrant 295 days ago | discuss

(Cross-posted an Less Wrong)

There is a cluster of problems in understanding naturalized agency which I call Robust Delegation (EDIT: When I first wrote this, I called it “subagent alignment”). It refers to the way in which a limited agent might direct a more powerful agent to do its bidding. The limit agent starts out in control because it gets to design the more powerful agent, but it is not easy to retain that power. In most applications the limited agent is a human and the powerful agent is an AI.

Example of problems in this subfield include corrigibility, value learning, and informed oversight.

Logical Updatelessness is one of the central open problems in decision theory. An updateless agent is one that does not update on its observations, but instead chooses what action it wants itself to output upon being input those observations. It can use this to get counterfactually mugged (which is a good thing). This is basically the only way we know how to make a reflectively stable agent.

The problem is that we do not know how to translate updatelessness into a logical setting. In the toy models in which we can define updatelessness, the agent is logically omniscient and just makes empirical observations. Thus, starts out just as powerful as it will ever be. In fact, we can view it as though the only agent is the agent existing at the beginning of time. That agent chooses a policy, a function from inputs to outputs, and all the future agents can just blindly follow that policy without thinking.

In the logical setting, the future agents are smarter than the past agents. They learn more logic in addition to making empirical observations. The agent at the beginning time does not have the power to contain a policy from all the logical observations to actions. This agent does not choose a simple policy to follow in the future, but instead chooses a complicated program to run in the future.

I claim that logical updatelessness is hard while empirical updatelessness is easy because an empirically updateless agent is deferring control to a simple policy that it understands, while a logically updateless agent has to defer control to a more powerful agent that it does not understand. Thus, logical updatelessness can be viewed as a robust delegation problem.

If we think about this just in terms of reflective stability, a reflectively stable agent has to never have a conflict between what its future self wants and what its past self wants. You can only achieve this by having a single agent at the center of control across all time. This center of control has to be simple enough that the agent can follow it in the early time steps, but the agent has to keep following it at the late time steps too. Thus, the agent in the late time steps has to be a powerful agent optimizing on behalf of this simple center of control.



NEW LINKS

NEW POSTS

NEW DISCUSSION POSTS

RECENT COMMENTS

There should be a chat icon
by Alex Mennen on Meta: IAFF vs LessWrong | 0 likes

Apparently "You must be
by Jessica Taylor on Meta: IAFF vs LessWrong | 1 like

There is a replacement for
by Alex Mennen on Meta: IAFF vs LessWrong | 1 like

Regarding the physical
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think that we should expect
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think I understand your
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

This seems like a hack. The
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

After thinking some more,
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yes, I think that we're
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

My intuition is that it must
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

To first approximation, a
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Actually, I *am* including
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yeah, when I went back and
by Alex Appel on Optimal and Causal Counterfactual Worlds | 0 likes

> Well, we could give up on
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

> For another thing, consider
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

RSS

Privacy & Terms