A sketch of a value-learning sovereign
post by Jessica Taylor 969 days ago | Patrick LaVictoire likes this | 15 comments

In the previous post, I discussed three preference frameworks for goal-directed agents. In this post, I will discuss the value-learning sovereign in more detail.

From the Arbital article on genies:

Eliezer Yudkowsky has suggested that people only confront many important problems in value alignment when they are thinking about Sovereigns, but that at the same time, Sovereigns may be impossibly hard in practice. Yudkowsky advocates that people think about Sovereigns first and list out all the associated issues before stepping down their thinking to Genies, because thinking about Genies may result in premature pruning, while thinking about Sovereigns is more likely to generate a complete list of problems that can then be checked against particular Genie approaches to see if they have become any easier.

To this end, I think it is quite useful to discuss how to create a value-learning sovereign, even if it is not a good idea to actually create one. I should be explicit about the fact that the concrete models in this post are almost certainly wrong (even conditioning on the fact that we have to build a value-learning sovereign); they’re meant to represent the best concrete illustration of value learning that I can currently write down.

## Values and ontologies

We want the AI to learn human values from human behavior. Usually, values are represented as a utility function. If the type of the world history is $$\Omega$$, then a utility function over $$\Omega$$ is of type $$\Omega \rightarrow [0, 1]$$. To learn $$U$$, we must first have some $$\Omega$$ in mind – but what could this $$\Omega$$ be? There are 2 plausible candidates:

1. The human’s ontology, $$\Omega_H$$. I have some way of mentally representing world states. My ontology contains concepts such as “human” and “happiness”. I can express values, such as caring about human happiness, in this ontology. If the AI has a representation of $$\Omega_H$$, then it may be able to learn the human utility function $$U_H : \Omega_H \rightarrow [0, 1]$$.
2. The AI’s ontology, $$\Omega_{AI}$$. The AI will also model the world somehow. Probably, its model will be at least partially learned by induction. It will probably make different predictions from me, due to the fact that it might be able to discover physics that I don’t know about (or otherwise model the world differently). Despite the differences between the AI’s world model and my own, it is quite likely that my terminal values could be specified well enough in the ontology of a strongly superintelligent AI, since this ontology is likely to be finer than my own.

How might we more formally represent the ontology? A simple environment model for talking about ontologies is the partially observable Markov decision process (POMDP). A POMDP consists of a number of iterations. In each iteration, the agent first takes an action (which causes the state to change stochastically), and then receives an observation of the next state. First we must define the set of actions $$\mathcal{A}$$ and observations $$\mathcal{O}$$. These sets apply both to the human and AI. Unlike in a standard POMDP, here the agent’s utility function is over the world history rather than the observed reward.

Now let’s formally define an ontology. An ontology consists of:

1. a type of world states, $$\mathcal{S}$$
2. the distribution over the initial state, $$s_0 : \Delta \mathcal{S}$$
3. the stochastic state transition function, $$s_t : (\mathcal{S}, \mathcal{A}) \rightarrow \Delta \mathcal{S}$$, which specifies what distribution of states results starting from a given state if the agent takes a certain action.
4. the stochastic observation function, $$o : \mathcal{S} \rightarrow \Delta \mathcal{O}$$, which specifies what distribution of observations the agent receives in a given state.

By abuse of notation, let $$\Omega$$ stand both for the type of world histories (list of $$S$$ values), and the ontology itself. Note that this model is Cartesian in much the same way as AIXI is, and therefore faces similar problems. See the paper on realistic world models for more details. It is also unrealistic in that it has no explicit “multi-level” structure; we would expect human and AI concepts to have something like this. The analysis in the rest of the post will be limited by these problems, but I think it will still be useful to analyze an incorrect concrete model.

Each stochastic function in the ontology could be represented by a probabilistic program. For example, consider the following ontology, modelled after the vacuum cleaner example in Artificial Intelligence: A Modern Approach:

# action set A = ['left', 'right', 'suck']
# observation set O = ['clean', 'dirty']

# S consists of (vacuum cleaner location, cleanliness) tuples
# where location is between 0 and 9, and cleanliness is a list of 10 booleans
# indicating whether each square is clean.

def s0():
# start at a random location.  Each room is clean with 50% probability.
return (random.randrange(10), [random.random() < 0.5 for i in range(10)])

def st(s, a):
loc = s[0]
cleanliness = s[1][:]
if a == 'left':
# move left
loc = max(0, loc - 1)
if a == 'right':
# move right
loc = min(9, loc + 1)
if a == 'suck':
# probably suck dirt from current square
if random.random() < 0.9:
cleanliness[loc] = True
return (loc, cleanliness)

def o(s):
# observe cleanliness of current square
if s[1][s[0]]:
return 'clean'
else:
return 'dirty'

With the ontology in place, we can consider some utility function over it (in this case, discounted cleanliness over time):

def U(state_seq):
util = 0
discount = 1.0
for s in state_seq:
clean = 0
for c in s[1]:
if c:
clean += 1
util += discount * clean
discount *= 0.99

Since $$U$$ discounts exponentially, it can easily be extended to a utility function over infinite state sequences.

## Planning using an ontology

If the AI already has some ontology $$\Omega$$ and some utility function over the ontology $$U$$, then it is possible for it to search for utility-maximizing policies. A policy could be represented as a stochastic function $$\pi : (\mathcal{A} \times \mathcal{O}) \rightarrow \Delta \mathcal{A}$$, which specifies what action the agent takes given an action/observation sequence. Essentially, a policy is a stochastic infinite decision tree. $$\pi$$ could be chosen to maximize $$\mathbb{E}[U(\omega) | \pi]$$, where $$\omega : \Omega$$ is the state sequence and the expectation is with respect to the distribution defined by the ontology $$\Omega$$.

## Learning setups

With the ontology machinery in place, we have the tools to look at a couple proposals for how to learn the human’s values. We assume that the AI has access to the observations and actions of the human. For example, in the vacuum cleaner example, perhaps the AI sees the sequence ['suck', 'clean', 'right', 'clean', 'right', 'dirty', 'suck', 'clean'], meaning that the human controlling the vacuum cleaner first decided to suck dirt, observed that the square was now clean, went right, observed that this square was also clean, etc. From a long enough sequence like this, the AI should approximately learn the human’s values.

In practice, it might be necessary for the AI to locate humans in the environment. This adds some additional complexity, but for now I will ignore this.

## Learn the human’s utility function expressed in the human ontology, from human behavior

The AI could assume that the human’s values are expressed over the (unknown) human ontology $$\Omega_H$$. The AI has a joint prior over $$\Omega_H$$ and a utility function $$U_H : \Omega_H \rightarrow [0, 1]$$. Additionally, the AI needs to predict how the human behaves given their ontology and utility function. One approach, common in economics, is to assume that the human maximizes expected utility. However, this model is quite unrealistic, and alternative models have been explored in the field of cognitive science.

We could represent an alternative behavioral model as a policy $$\pi : \text{List}(\mathcal{A} \times \mathcal{O}) \rightarrow \Delta \mathcal{A}$$, similar to a planning model. The behavioral model should depend on $$\Omega_H$$ and $$U_H$$. While it is possible to set the behavioral model to maximize expected utility, this is psychologically unrealistic, so the behavioral model should allow the human to sometimes take suboptimal actions.

# The AI’s planning, once it has the human ontology and utility function

Suppose the AI has inferred the human ontology $$\Omega_H$$, utility function $$U_H$$, and behavior model. How could it make plans? There are 3 immediate candidates for planning algorithms:

1. Mimic the human using the behavioral model.
2. Select a plan that achieves high expected utility according to $$\Omega_H$$ and $$U_H$$.
3. Select a plan that achieves high expected utility according to $$\Omega_{AI}$$ and $$U_{AI}$$, where $$U_{AI} : \Omega_{AI} \rightarrow [0, 1]$$ is a version of $$U_H$$ that has been translated to be over $$\Omega_{AI}$$ rather than $$\Omega_H$$

Planning algorithm 1 is not too interesting by itself; it is probably more useful as an input to other AI control methods, such as approval-directed agents.

Planning algorithm 2 is more interesting. It selects a plan that looks good according to the human’s ontology. This does not take advantage of the AI’s ability to make better predictions than the human, but it does take advantage of the AI’s better ability to search for plans. For example, if I were trying to solve a boolean satisfiability problem, an AI using this algorithm could suggest a solution, because my ontology predicts that this solution works, even though I can’t find the solution myself. In this way, an agent using this planning algorithm is similar to an approval-directed agent. The main difference is that it selects a plan (i.e. an infinite stochastic decision tree) that maximizes how good the human expects the results to be, rather than an action. Otherwise, it is quite similar.

Planning algorithm 3 uses the full power of the AI, including the AI’s ability to make better predictions than the human. It requires deriving a utility function $$U_{AI}$$ from the inferred human utility function $$U_H$$. If we have $$U_{H} : \Omega_{H} \rightarrow [0, 1]$$, and we want to create $$U_{AI} : \Omega_{AI} \rightarrow [0, 1]$$, one way to get this is to create a probabilistic ontology mapping function $$\phi : \Omega_{AI} \rightarrow \Delta \Omega_{H}$$, and then define $$U_{AI}(\omega_{AI}) = \mathbb{E}[U_{H}(\phi(\omega_{AI}))]$$. The ontology mapping function is meant to say which histories in $$\Omega_{H}$$ best represent some history in $$\Omega_{AI}$$. Probably, it is more intuitive to map states rather than world histories, but it shouldn’t matter much. The paper Ontological Crises in Artificial Agents’ Value Systems discusses an ontology mapping method.

While it would be interesting to look more closely at the relation between planning algorithm 2 and approval-directed agents at some point, I’ll focus on planning algorithm 3 for the rest of the post. Planning algorithm 3 has multiple problems:

1. Unless we have a good theory of cognitive science, it is likely that the true human ontology and utility function will have a very low or zero prior probability.
2. Human values seem underdetermined by the observation/action data. For a given observation/action sequence, there may be many triples of (ontology, utility function, behavior model) leading to this behavior. The AI must have some way of acting appropriately under this uncertainty.
3. The ontology mapping seems difficult. I’ll say more about just how hard the ontology mapping problem is in a bit.

## Learn the human’s utility function expressed in the AI’s ontology, from human behavior

As an alternative to learning the human’s ontology and utility function expressed in this ontology, the AI could assume that the human’s values are expressed over the (known) AI ontology $$\Omega_{AI}$$. The AI has a joint prior over the utility function $$U_{AI} : \Omega_{AI} \rightarrow [0, 1]$$ and the behavior model as before. The assumption is that the human plans using the AI’s ontology, rather than a different human ontology. Current value-learning algorithms, such as inverse reinforcement learning and inverse planning, work this way because they do not distinguish between the AI’s ontology and the human’s ontology.

Unfortunately, this model is psychologically implausible. We do not think of the human’s preferences about the AI’s ontology being a cause of human behavior; rather, it is the human’s preferences about the human’s ontology that is a cause of human behavior. One place where this shows up is when the human takes an action that would be irrational if the human were using the AI’s ontology (for example, the human calculates something incorrectly because they do not know about quantum physics, which the AI knows about). The AI has no choice but to either believe that the human’s utility function considers making this incorrect calculation to be optimal, or to explain it as an error according to the behavior model. For the second option to produce reasonable results, the behavior model must be quite complex; it will probably talk about the human’s ontology and how the human’s goals in this ontology relate to $$U_{AI}$$, much like proposal 1.

Overally, I do not find this proposal promising. Either the behavioral model suffices to explain correlated human errors due to the human having an incorrect ontology (in which case the behavioral model contains all the complexity of proposal 1), or it does not (in which case the AI will learn the wrong values). Therefore, I will talk about the first proposal in the rest of this post.

## Instrumental or terminal goals?

Paul Christiano has previously written about the distinction between learning terminal and instrumental goals in his post, Ambitious vs. narrow value learning. It is possible to explore this distinction in proposal 1. Since the human’s utility function is relative to the human’s ontology, it is not possible for it to express truly terminal goals. Determining humans’ reflective preferences about states of the universe requires some kind of philosophical extrapolation process, in which humans clarify their concepts and develop preferences about their new concepts.

However, by varying the behavioral model, it is possible to learn either higher-level instrumental goals (for example, getting a job), or lower-level instrumental goals (for example, filling out a particular job application). If the behavior model states that the human behaves by finding subgoals of $$U_H$$ and then optimizing for them (as we would expect if $$U_H$$ were a high-level goal), then it is more likely to detect high-level goals. On the other hand, if the behavior model states that the human optimizes for $$U_H$$ more directly (as we would expect a human to optimize for a low-level goal), then it is more likely to detect low-level goals.

Note that, since instrumental goals change over time, we would also need to have $$U_H$$ change over time. This is a simple modification to the original model. Obviously, the AI’s goal should be set so it has no incentive to change the human’s goals to make them easier to optimize. Perhaps its goal at time $$t$$ is to maximize expected utility of whatever $$U_H$$ is at time $$t$$.

Naively, if the AI’s utility function changes over time, then it will be dynamically inconsistent. The AI at an earlier time will have an incentive to lock the current value of $$U_H$$ in place, so that future versions of the AI will optimize for this $$U_H$$ instead of whatever $$U_H$$ is estimated to be in the future. This would lead to a system that determines what my instrumental preferences are, and then continues to optimize for these even as my instrumental preferences change.

# An instrumental preference for autonomy

It seems that a system that locks my “object-level” instrumental goals (such as filling out a job application) in place would be acting against some of my other instrumental goals: specifically, my instrumental goal of preserving my autonomy. Paul discusses this preference in his post:

Humans have many clear instrumental goals like “remaining in effective control of the AI systems I deploy,” “acquiring resources and other influence in the world,” or “better understanding the world and what I want.” A value learner may able to learn robust preferences like these and pursue those instrumental goals using all of its ingenuity.

In general, I will prefer plans that maximize my autonomy, so I could consider autonomy-maximization to be one of my instrumental goals. This preference could explain my desire to study moral philosophy, even when this might cause my moral opinions to change (and therefore be bad according to my current object-level moral views). By caring about my autonomy, I can mostly preserve dynamic consistency even as my goals change over time.

More concretely, suppose the state $$\mathcal{S}_H$$ in the human’s ontology contains a field for “autonomy”, indicating how much autonomy I have in this state. We would hope that state sequences in $$\Omega_{AI}$$ in which the human has low autonomy get mapped to sequences of states in $$\mathcal{S}_H$$ that have a low autonomy field. For example, state sequences in which the AI manipulates the human should be mapped to states with a low autonomy field.

Of course, it would be imprudent to assume that proposal 1 will correctly do all this. “Human autonomy” seems to be a complex concept, so it would be difficult to learn. To reduce confusion, it would be a good idea to create more explicit models of this instrumental preference for autonomy. This seems related to the hard problem of corrigibility: the human’s desire for AIs to be corrigible is really a reflection of the human’s preference for autonomy. This seems somewhat related to hierarchical planning, so maybe I will have better models of this preference after understanding hierarchical planning better.

If a model like this works, then we can ground human values in something other than terminal goals: specifically, systems of instrumental goals at each time step that chain together in a tiling fashion, with each instrumental goal system trusting the next under normal circumstances. I think this is a promising alternative way to look at human values, though I still lack concrete models for this.

## Acting under uncertainty

The system should have uncertainty about the correct values. In both proposals, the human utility function is underdetermined by the data. In proposal 1, the human ontology is underdetermined by the data, and additionally any uncertainty about the correct ontology mapping method propagates into uncertainty about the correct utility function.

Under uncertainty about the correct utility function, it is not straightforward to simply maximize expected utility. This is because the “loudness” of different possible preferences matters. Given this, there are 2 clear ways to act under uncertainty:

1. The system can use a voting system to select actions, with each possible human utility function gaining votes proportional to its posterior probability. Unfortunately, this leads to undesirable results when the majority of the posterior probability mass is on the wrong preferences. Roughly, we should only expect this to work when the posterior distribution over preferences is “centered around” an acceptable preference to optimize.
2. The system can use minimax to select a policy that does decently according to all possible utility functions. In particular, the policy should be at least as good as shutting down according to all possible utility functions. This method of handling uncertainty has problems; see the “Combining minimax with value learning” section for details.

I think it’s plausible that some variant of minimax works for conservatively optimizing values under uncertainty, so more research in this area could be useful.

## The necessity of overpowered ontology mapping

I claim that, for proposal 1 to work, the ontology mapper needs to be very powerful and reliable. This is because:

1. It needs to correctly map abstract concepts. For example, state sequences in $$\Omega_{AI}$$ in which humans have lost autonomy should get mapped to state sequences in $$\Omega_H$$ that have the “autonomy” field set to a low number. This seems far less straightforward than, say, recognizing diamonds in an ontology. This is made even more difficult by the fact that some important human concepts (including autonomy) are value-laden and might not correspond to useful predictive concepts.
2. Since the AI is optimizing over state sequences in $$\Omega_{AI}$$, the ontology mapper must work correctly across nearly all state sequences in $$\Omega_{AI}$$. Even if there is just one state sequence in $$\Omega_{AI}$$ that humans would consider bad upon reflection, but which gets mapped to a good-looking state sequence in $$\Omega_H$$, this may be sufficient for the AI to select a plan leading to this state sequence.

These problems make me quite pessimistic about this proposal. More research into ontology identification might yield insights about just how hard these problems are to solve.

## Human understanding of plans

Suppose the AI has created a plan $$\pi$$. Humans could examine this plan by seeing what state sequences $$\Omega_H$$ result from this plan (assuming humans understand the $$\Omega_H$$ ontology). There are 2 obvious ways to do this:

1. Use the human ontology $$\Omega_H$$ to predict the state sequence resulting from $$\pi$$. This may fail to predict important consequences of the AI’s plan. For example, if the AI used nanotechnology to solve some problem, and $$\Omega_H$$ does not predict this nanotechnology to do anything, then it will predict that the AI’s plan will not do much.
2. Use the AI’s ontology $$\Omega_{AI}$$ to predict the state sequence resulting from $$\pi$$, and then map this state sequence back to $$\Omega_H$$ using ontology mapping. This will likely predict the consequences of $$\pi$$ more accurately than $$\Omega_H$$. Possibly, this could help to catch errors that result when the AI accurately infers $$\Omega_H$$ and maps between the ontologies correctly, but incorrectly infers $$U_H$$. This does not seem like the most likely form failure to me; errors in ontology mapping seem more likely.

## Conclusion

I don’t think any of the models I have described will do anything useful with superhuman intelligence. The most potentially powerful models require essentially solving cognitive science (to get the behavioral model), and creating an overpowered ontology mapper. Still, I have identified concrete areas for further research, which might turn up results useful for both value-learning sovereigns and other agents. Specifically, further research into value-learning sovereigns could look at:

1. Clarifying what the instrumental preference for autonomy looks like. I would like to see a concrete example (either in the mathematical framework described in this post, or a different mathematical framework) of an AI representing (and perhaps also learning) the human’s instrumental preference for autonomy.
2. Developing a better understanding of ontology identification. I think that framing ontology identification as mapping states between ontologies (as in the paper on ontological crises) has some theoretical problems, which I hope to discuss in a future post.
3. Looking more closely at the spectrum between mimicking humans and learning and optimizing for humans’ terminal goals. Many of the proposals in this post fall somewhere in the middle of these two possibilities, but I don’t think I have exhausted all the options.
4. Studying ways of conservatively maximizing under uncertainty about the right values, similar to minimax.

 by Jessica Taylor 968 days ago | link Yeah, I think just having coarse-grained facts would not be enough. I’m referring to a more general idea when I say “multi-level models”: something that can represent concepts at different levels of abstraction, probably not with the high-level facts being a function of the low-level facts. My goal would be to have at least some concrete model for a multi-level model that, for example, preserves the “diamond” concept as it learns new physics. I think (a) and (b) are both reasons why I want to do this; I want to know if it’s possible to create an AI with goals related to concrete physical things (which requires something like multi-level models, but maybe not much more?), and I also want to have a better understanding of more abstract concepts to see if it’s possible to have an AI do anything useful with them. Could our disagreement be stated as: I think it is plausible that, with a few years of work, a small number of researchers could make useful models for things like diamond-maximization; whereas you don’t think it is plausible? reply
 by Paul Christiano 968 days ago | Jessica Taylor likes this | link In this post, it seems like ontology identification is acting as a partial solution to the easy goal inference problem, by explaining how the human’s behavior would change if their model of the world changed. I would be more interested in seeing any credible strategy for the easy goal inference problem. I don’t really see how we are going to solve that problem in a way that doesn’t incidentally address ontology identification, and I don’t really see how a solution to ontology identification is going to help with that problem. This may reflect a more general methodological difference. My impression is that the MIRI view is something like “obviously any solution will have to address X, so X is a reasonable way to approach the big problem,” for X = ontology identification, tiling, “realistic world models,” etc. My impression is more like “obviously any solution will have to address the ontology identification problem, so it doesn’t matter whether we find some special-case solution to that.” Both situations come up a lot in computer science, and I do both of them a ton in my own research. I don’t really have a clean story about when one or the other is a more appropriate response, but in these particular cases I have pretty strong intuitions. reply
 by Jessica Taylor 968 days ago | link Good point. In this post I was trying to solve something like the easy goal inference problem by factoring it into some different problems, including ontology identification, but it’s not clear whether this is the right factoring. It seems like your intuition is something like: “any way of correctly modelling human mistakes when the human and AI share an ontology will also correctly handle mistakes arising from the human and AI having a different ontology”. I think I mostly agree with this intuition. My motivation for working on ontology identification despite this intuition is some combination of (a) “easy” versions of ontology identification seem useful outside the domain of value learning (e.g. making a genie for concrete physical tasks), and (b) I don’t see many promising approaches for directly attacking the easy goal inference problem. But after writing this, I think I have updated towards looking for approaches to the easy goal inference problem that avoid ontology identification. The most promising thing I can think of right now seems to be some variation on planning algorithm 2, but with an adjustment so that the planning can take into account the AI’s different predictions (but not the AI’s internal representation). It does seem plausible that something in this space would work without directly solving the ontology identification problem. reply
 by Paul Christiano 968 days ago | Jessica Taylor likes this | link I don’t see many promising approaches for directly attacking the easy goal inference problem. I would agree with that. But I don’t see how this situation will change no matter what you learn about ontology identification. It looks to me like the easy goal inference problem is probably just ill-posed/incoherent, and we should avoid any approach that rests on us solving it. The kind of insight that would be required to change this view looks very unlike the kind required to solve ontology identification, and also unlike the kind required to make conventional progress in AI, so to the extent that there are workable approaches to the easy goal inference problem it seems like we could work towards them now. If we can’t see how to attack the problem, then by the same token that leads me to be pessimistic. On that perspective we might ask: how we are avoiding the problem? The dodges I know would also dodge ontology identification, by cashing everything out in terms of human behavior. It’s harder for me to know what the situation is like for solutions to the goal inference problem–because I don’t yet see any plausible solution strategies–but I would guess that the situation will turn out to be similar. reply
 by Vadim Kosoy 966 days ago | link Regarding the methodological difference. My perspective is that head-on attempts to solve AI safety are not very promising since we lack the tools to answer basic questions in the general area of the problem such as “what is intelligence?”, “what is the computational resource cost of constructing an agent of given intelligence?” or “what is the growth curve of self-improving agents?” Therefore, what we should be doing is constructing a general theory capable of answering questions of this type (I would call it abstract intelligence theory). Thinking about problems such naturalized induction and Vingean reflection seems to me a useful way to approach this, not because they are subproblems of the AI safety problem but because they are handles to getting a mathematical grip on the entire area. reply
 by Paul Christiano 968 days ago | link Note that “non-deterministic” usually means something different than “stochastic,” especially in the context of transition functions. reply
 by Jessica Taylor 968 days ago | link Good point, fixed. reply
 by Paul Christiano 968 days ago | link It’s not really clear to me what one would want out of “more explicit models of this instrumental preference for autonomy.” It’s a complex and messy preference that is tied up with other similarly complex and messy preferences. It probably doesn’t have a simple or natural definition in any reasonable ontology. What concrete questions about this preference would you hope to answer? To the extent this preference causes a system to have good behavior, it will be because it affects humans’ behavior, e.g. a human would predictably and systematically decline actions that significantly reduce their own autonomy. So we need to set up our system so that these effects on human behavior lead to it also avoiding actions that significantly reduce the user’s autonomy. You seem to have in mind a particular version, where the agent infers some latent structure which can then be used to correctly evaluate situations unlike those that appear in the training data (and in particular, to compare the plans put forth by an AI rather than those put forth by a human). So maybe you want to know something about what kind of concepts can robustly transfer from one domain to another quite different domain. It feels to me like you are only going to find bad news here, unless you first make some significant conceptual contributions in AI. So it seems like the first step would be to look for some good news anywhere and see what kind of good news you have to work with. (Or to wait until the AI community produces some good news on its own, and work on other problems in the meantime.) A somewhat different angle: The “instrumental goal pursuer” is no more or less dynamically inconsistent than the human. The human wouldn’t lock the current goal in place, and so obviously any preferences that successfully explain the human’s short-term behavior won’t lock the current goal in place. This is a simple observation that already appears in the training set. This doesn’t require learning a complex concept of autonomy. It just requires learning a model of human preferences that roughly reproduces human behavior. If you don’t get this kind of thing right, then it seems pretty clear that you aren’t going to get useful behavior out of the system in general. Now you may take this as a general argument against value learning, or that value learning will be difficult, but it doesn’t seem like we should consider these kinds of preferences as any different from normal preferences. reply
 by Jessica Taylor 968 days ago | link A model I might want to make would be something like a hierarchical planning algorithm. It would have some supergoal, and then find subgoals of the supergoal. If the system just naively optimized for the subgoals, then it might do silly things like lock its subgoals in place. Instead, this algorithm should prefer plans that maximize the agent’s autonomy (in case the agent changes subgoals). If this model works, then maybe we can use it to derive a partial solution to the hard problem of corrigibility. So the real question I want to answer is something like “what kind of AI would a agent with a preference for autonomy choose to build”; I suspect that this AI design will be corrigible in some way. I think this is even useful if the agent is much simpler than a human. You seem to have in mind a particular version, where the agent infers some latent structure which can then be used to correctly evaluate situations unlike those that appear in the training data (and in particular, to compare the plans put forth by an AI rather than those put forth by a human). So maybe you want to know something about what kind of concepts can robustly transfer from one domain to another quite different domain. Yeah, this seems accurate. I think this goes back to you being slightly more pessimistic than me about making progress on ontology identification (though I’m still somewhat pessimistic). This doesn’t require learning a complex concept of autonomy. It just requires learning a model of human preferences that roughly reproduces human behavior. Right, a good supervised learner should learn this. This is more of a problem if we’re using the model’s internal representation, not just its predictions. reply
 by Paul Christiano 968 days ago | link I don’t think the difference is pessimism about ontology identification per se. You overall approach (if successful) seems like it would do zero-shot transfer learning. My perspective would be something like: OK, let’s try and understand when we can do zero-shot transfer learning at all, and what assumptions we need to rely on (incidentally, I am also pessimistic about this). You are instead focusing on a different simplification of the problem, one which (I feel) is less likely to be connected to the most important underlying difficulties, and less likely to quickly provide information about whether the overall approach can work. reply
 by Paul Christiano 968 days ago | link This is more of a problem if we’re using the model’s internal representation, not just its predictions. But you aren’t directly using the model’s internal representation, are you? You are using it only to make predictions about the human’s preferences in some novel domain (e.g. over the consequences of novel kinds of plans). It seems like it would be cleaner to discuss the whole thing in the context of transfer to a new domain, rather than talking about directly using the learned representation, unless I am missing some advantage of this framing. Are you hoping to do transfer learning for human preferences in a way that depends on having a detailed understanding of those preferences (e.g. that depends in particular on a detailed understanding of the human preference for autonomy)? I would be very surprised by that. It seems like if you succeed you must be able to robustly transfer lots of human judgments to unfamiliar situations. And for that kind of solution, it’s not clear how an understanding of particular aspects of human preferences really helps. reply
 by Jessica Taylor 968 days ago | link It seems like it would be cleaner to discuss the whole thing in the context of transfer to a new domain, rather than talking about directly using the learned representation, unless I am missing some advantage of this framing. I agree with this. Problems with learning this preference should cause the system to make bad predictions (I think I was confused when I wrote that this problem only shows up with internal representations). Now that I think about it, it seems like you’re right that a system that correctly learns abstract human preferences would also learn the preference for autonomy. So this is really a special case of zero-shot transfer learning of abstract preferences. My main motivation for specifically studying the preference for autonomy is that maybe you can turn a simple version of it into a model for corrigibility. Are you hoping to do transfer learning for human preferences in a way that depends on having a detailed understanding of those preferences (e.g. that depends in particular on a detailed understanding of the human preference for autonomy)? I think I mostly want some story for why the preference for autonomy is even in the model’s hypothesis space. It seems that if we’re already confident that the system can learn abstract preferences, then we could also be confident that the system can learn the preference for autonomy; but maybe it’s more of a problem if we aren’t confident of this (e.g. the system is only supposed to learn and optimize for fairly concrete preferences). reply
 by Paul Christiano 968 days ago | link Both 1. “mimic the human” and 2. “maximize according to the human’s ontology” will only work well if the human can actually develop a world model (and in the case of 1 also plans) as well as the AI can. If you can do this, then we are probably set on value learning (at least at the level of detail in this post). Moreover, if we can produce world models as good as the AI then we can probably also produce plans as good as the AI, so probably we can just focus on [1]. I’m obviously much more optimistic about this than about approach [3]. Note: I think that the only reason to be interested in approval-directed agents rather than straightforward imitation learners is that it may be harder to effectively imitate behavior than to solve the same task in a very different way. So it seems wrong to say that imitation is most useful as an input into approval-directed agents. reply
 by Vadim Kosoy 966 days ago | link Regarding acting under uncertainty, there is another natural approach namely applying the Nash bargaining solution to an imagery collection of agents with utility functions sampled from the given ensemble. reply

### NEW DISCUSSION POSTS

There should be a chat icon
 by Alex Mennen on Meta: IAFF vs LessWrong | 0 likes

Apparently "You must be
 by Jessica Taylor on Meta: IAFF vs LessWrong | 1 like

There is a replacement for
 by Alex Mennen on Meta: IAFF vs LessWrong | 1 like

Regarding the physical
 by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think that we should expect
 by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think I understand your
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

This seems like a hack. The
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

After thinking some more,
 by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yes, I think that we're
 by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

My intuition is that it must
 by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

To first approximation, a
 by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Actually, I *am* including
 by Vadim Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yeah, when I went back and
 by Alex Appel on Optimal and Causal Counterfactual Worlds | 0 likes

> Well, we could give up on
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

> For another thing, consider
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes