Intelligent Agent Foundations Forumsign up / log in
Logical Updatelessness as a Robust Delegation Problem
post by Scott Garrabrant 212 days ago | discuss

(Cross-posted an Less Wrong)

There is a cluster of problems in understanding naturalized agency which I call Robust Delegation (EDIT: When I first wrote this, I called it “subagent alignment”). It refers to the way in which a limited agent might direct a more powerful agent to do its bidding. The limit agent starts out in control because it gets to design the more powerful agent, but it is not easy to retain that power. In most applications the limited agent is a human and the powerful agent is an AI.

Example of problems in this subfield include corrigibility, value learning, and informed oversight.

Logical Updatelessness is one of the central open problems in decision theory. An updateless agent is one that does not update on its observations, but instead chooses what action it wants itself to output upon being input those observations. It can use this to get counterfactually mugged (which is a good thing). This is basically the only way we know how to make a reflectively stable agent.

The problem is that we do not know how to translate updatelessness into a logical setting. In the toy models in which we can define updatelessness, the agent is logically omniscient and just makes empirical observations. Thus, starts out just as powerful as it will ever be. In fact, we can view it as though the only agent is the agent existing at the beginning of time. That agent chooses a policy, a function from inputs to outputs, and all the future agents can just blindly follow that policy without thinking.

In the logical setting, the future agents are smarter than the past agents. They learn more logic in addition to making empirical observations. The agent at the beginning time does not have the power to contain a policy from all the logical observations to actions. This agent does not choose a simple policy to follow in the future, but instead chooses a complicated program to run in the future.

I claim that logical updatelessness is hard while empirical updatelessness is easy because an empirically updateless agent is deferring control to a simple policy that it understands, while a logically updateless agent has to defer control to a more powerful agent that it does not understand. Thus, logical updatelessness can be viewed as a robust delegation problem.

If we think about this just in terms of reflective stability, a reflectively stable agent has to never have a conflict between what its future self wants and what its past self wants. You can only achieve this by having a single agent at the center of control across all time. This center of control has to be simple enough that the agent can follow it in the early time steps, but the agent has to keep following it at the late time steps too. Thus, the agent in the late time steps has to be a powerful agent optimizing on behalf of this simple center of control.





Note: I currently think that
by Jessica Taylor on Predicting HCH using expert advice | 0 likes

Counterfactual mugging
by Jessica Taylor on Doubts about Updatelessness | 0 likes

What do you mean by "in full
by David Krueger on Doubts about Updatelessness | 0 likes

It seems relatively plausible
by Paul Christiano on Maximally efficient agents will probably have an a... | 1 like

I think that in that case,
by Alex Appel on Smoking Lesion Steelman | 1 like

Two minor comments. First,
by Sam Eisenstat on No Constant Distribution Can be a Logical Inductor | 1 like

A: While that is a really
by Alex Appel on Musings on Exploration | 0 likes

> The true reason to do
by Jessica Taylor on Musings on Exploration | 0 likes

A few comments. Traps are
by Vadim Kosoy on Musings on Exploration | 1 like

I'm not convinced exploration
by Abram Demski on Musings on Exploration | 0 likes

Update: This isn't really an
by Alex Appel on A Difficulty With Density-Zero Exploration | 0 likes

If you drop the
by Alex Appel on Distributed Cooperation | 1 like

Cool! I'm happy to see this
by Abram Demski on Distributed Cooperation | 0 likes

Caveat: The version of EDT
by 258 on In memoryless Cartesian environments, every UDT po... | 2 likes

[Delegative Reinforcement
by Vadim Kosoy on Stable Pointers to Value II: Environmental Goals | 1 like


Privacy & Terms