Intelligent Agent Foundations Forumsign up / log in
Stable agent, subagent-unstable
discussion post by Stuart Armstrong 325 days ago | discuss

A putative new idea for AI control; index here.

A reflectively consistent agent is one that is ok with creating copies with the same motivation as itself, or continuing to have the same motivation under self-modification.

A reflectively stable agent is one that would only create agents with the same motivation as itself (unless rewarded for doing otherwise), and would actively want to preserve its motivation under self-modification.

Here is a design that is reflectively stable for its own self-modification, but not even reflectively consistent for copies and subagents.

Subtracting the value function

Let \(h_t\) be the history of the agent’s interaction with the environment up to time \(t\).

Let the agent \(A\) have policy \(\pi_A\). Let \(V(R,\pi, h_t)\) be the expected value of (future) reward \(R\) under policy \(\pi\), given history \(h_t\). Let \(\pi_R\) the policy that maximises this value the most.

Initially, the agent has no reward. Then, at time \(t\), the AI gets reward function \(R\), and gets the penalty:

  • \(-c V(R, \pi_A, h_t)\).

For \(c<1\), this change of reward is reflexively stable (the agent will choose \(\pi_A=\pi_R\) to get maximal expected reward \((1-c)V(R,\pi_R,h_t)\)), for \(c=1\), it is reflexively consistent (any policy \(\pi_A\) gives the same expected reward of \(0\)), and for \(c>1\), it is neither.

But instead, imagine the agent gets the penalty:

  • \(-c V(R, \pi_R, h_t)\).

In this case, the agent is reflexively stable for all values of \(c\)! That’s because the penalty term is independent of its policy choice, and it will therefore choose \(\pi_A=\pi_R\) after \(t\) to maximise the reward it does get from \(R\).

No subagent stability.

Using \(\pi_R\) in the reward may make the agent itself reflexively stable, but if \(c>1\), it will not copy the same goal into its subagent.

This is easy to see, because the agent’s expected reward is \((1-c)V(R, \pi_R, h_t)\), so its best policy is to minimise \(V(R, \pi_R, h_t)\). It will do this by acting so that \(h_t\) is one the minimises that value.

For instance, it might create subagents whose task is to actively restrain the agent after \(t\). Then it will struggle against them after \(t\) (because it now wants to maximise the reward \(R\)), but will set them up to be as powerful as possible before that, because before \(t\), it wants its own future struggles to fail.

Clearly, though this is technically reflectively stable, this is not a good sort of stability.

Not ‘real’ stability

It should be noted that the agent’s “stability” is an artefact of the fact that \(\pi_R\) is defined to be “the best policy that the agent can follow to maximise \(R\)”.

“The agent” is not an ontologically fundamental object, so this stability is only as good as our definition of the agent (just as we could make the whole setup subagent-stable, if only we could define subagents - which we can’t really do).



NEW LINKS

NEW POSTS

NEW DISCUSSION POSTS

RECENT COMMENTS

[Note: This comment is three
by Ryan Carey on A brief note on factoring out certain variables | 0 likes

There should be a chat icon
by Alex Mennen on Meta: IAFF vs LessWrong | 0 likes

Apparently "You must be
by Jessica Taylor on Meta: IAFF vs LessWrong | 1 like

There is a replacement for
by Alex Mennen on Meta: IAFF vs LessWrong | 1 like

Regarding the physical
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think that we should expect
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think I understand your
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

This seems like a hack. The
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

After thinking some more,
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yes, I think that we're
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

My intuition is that it must
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

To first approximation, a
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Actually, I *am* including
by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yeah, when I went back and
by Alex Appel on Optimal and Causal Counterfactual Worlds | 0 likes

> Well, we could give up on
by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

RSS

Privacy & Terms