Intelligent Agent Foundations Forumsign up / log in
Smoking Lesion Steelman
post by Abram Demski 167 days ago | Tom Everitt, Sam Eisenstat, Vadim Kosoy, Paul Christiano and Scott Garrabrant like this | 8 comments

It seems plausible to me that any example I’ve seen so far which seems to require causal/counterfactual reasoning is more properly solved by taking the right updateless perspective, and taking the action or policy which achieves maximum expected utility from that perspective. If this were the right view, then the aim would be to construct something like updateless EDT.

I give a variant of the smoking lesion problem which overcomes an objection to the classic smoking lesion, and which is solved correctly by CDT, but which is not solved by updateless EDT.

UDT as originally described involved a “mathematical intuition module” which would take some sort of logical counterfactual. However, I’ll be using the term “updateless” purely to describe the decision theory you get by asking another decision theory to choose a policy as soon as it is born, rather than using that decision theory all along. Hence, updateless CDT is what you get when you ask a CDT agent to choose a policy; updateless EDT is what you get when you ask an EDT agent to choose a policy.

I’ll also be treating “counterfactual” as synonymous with “causal”. There are cases where physical causal reasoning seem to give the wrong counterfactual structure, like Newcomb’s problem. I won’t be trying to solve that problem here; I’m more trying to ask whether there are any cases where causal/counterfactual reasoning looks like what we really want at all.

The “common wisdom”, as I have observed it, is that we should be aiming to construct something like an updateless CDT which works well with logical uncertainty. I’m not sure whether that would be the dominant opinion right now, but certainly TDT set things in this direction early on. From my perspective, I don’t think it’s been adequately established that we should prefer updateless CDT to updateless EDT; providing some evidence on that is the implicit aim of this post. Explicitly, I’ll mostly be contrasting updateful CDT with updateful EDT.

It might be impossible to construct an appropriate logically-updateless perspective, in which case we need logical counterfactuals to compute the effects of actions/policies; but, in some sense this would only be because we couldn’t make a sufficiently ignorant prior. (I think of my own attempt at logically updateless decisions this way.) However, that would be unfortunate; it should be easier to construct good counterfactuals if we have a stronger justification for counterfactual reasoning being what we really want. Hence, another aim of this post is to help provide constraints on what good counterfactual reasoning should look like, by digging into reasons to want counterfactuals.

Thanks go to Alex Mennen, Evan Lloyd, and Daniel Demski for conversations sharpening these ideas.

Why did anyone think CDT was a good idea?

The original reasons for preferring CDT to EDT are largely suspect, falling to the “Why Ain’cha Rich?” objection. From the LessWrong/MIRI perspective, it’s quite surprising that Newcomb’s Problem was the original motivation for CDT, when we now use it as a point against. This is not the only example. The SEP article on CDT gives Prisoner’s Dilemma as the first example of CDT’s importance, pointing out that EDT cooperates with a copy of itself in PD because unlike CDT it fails to take into account that cooperating is “auspicious but not efficacious”.

The Smoking Lesion problem isn’t vulnerable to “Why Ain’cha Rich?”, and so has been a more popular justification for CDT in the LessWrong-sphere. However, I don’t think it provides a good justification for CDT at all. The problem is ill-posed: it is assumed that those with a smoking lesion are more likely to smoke, but this is inconsistent with their being EDT agents (who are not inclined to smoke given the problem setup). (Cheating Death in Damascus) points out than Murder Lesion is ill-posed due to similar problems.)

So, for some time, I have thought that the main effective argument against EDT and for CDT was XOR blackmail. However, XOR blackmail is also solved by updateless EDT. We want to go updateless either way, so this doesn’t give us reason to favor CDT over EDT.

Regardless, I’m quite sympathetic to the intuition behind CDT, namely that it’s important to consider the counterfactual consequences of your actions rather than just the conditional expected utilities. Furthermore, the following idea (which I think I got from an academic paper on CDT, but haven’t been able to track down which one) seems at least plausible to me:

If we were dealing with ideal decision agents, who could condition on all the inputs to their decision process, CDT would equal EDT. However, imperfect agents often cannot condition on all their inputs. In this situation, EDT will get things wrong. CDT corrects this error by cutting the relationships which would be screened off if only we could condition on those inputs.

This intuition might seem odd given all the cases where CDT doesn’t do so well. If CDT fails Newcomb’s problem and EDT doesn’t, it seems CDT is at best a hack which repairs some cases fitting the above description at the expense of damaging performance in other cases. Perhaps this is the right perspective. But, we could also think of Newcomblike decision problems as cases where the classical causal structure is just wrong. With the right causal structure, we might postulate, CDT would always be as good as or better than EDT. This is part of what TDT/FDT is.

I’ll charitably assume that we can find appropriate causal structures. The question is, even given that, can we make any sense of the argument for CDT? Smoking Lesion was supposed to be an example of this, but the problem was ill-posed. So, can we repair it?

A Smoking Lesion Steelman

Agents who Don’t Know their Utility Function

I’ll be assuming a very particular form of utility function ignorance. Suppose that agents do not have access to their own source code. Furthermore, whether CDT or EDT, the agents have an “epistemic module” which holds the world-model: either just the probability distribution (for EDT) or the probability distribution plus causal beliefs. This epistemic module is ignorant of the utility function. However, a “decision module” uses what the epistemic module knows, together with the utility function, to calculate the value of each possible action (in the CDT or EDT sense).

These agents also lack introspective capabilities of any kind. The epistemic module cannot watch the decision module calculate a utility and get information about the utility function that way. (This blocks things like the tickle defense).

These agents therefore have full access to their own utility functions for the sake of doing the usual decision-theoretic calculations. Nonetheless, they lack knowledge of their own utility function in the epistemic sense. They can only infer what their utility function might be from their actions. This may be difficult, because, as we shall see, EDT agents are sometimes motivated to act in a way which avoids giving information about the utility function.

While I admit this is simply a bad agent design, I don’t think it’s unrealistic as an extrapolation of current AI systems, or particularly bad as a model of (one aspect of) human ignorance about our own values.

More importantly, this is just supposed to be a toy example to illustrate what happens when an agent is epistemically ignorant of an important input to its decision process. It would be nice to have an example which doesn’t arise from an obviously bad agent design, but I don’t have one.

Smoking Robots

Now suppose that there are two types of robots which have been produced in equal quantities: robots who like smoking, and robots who are indifferent toward smoking. I’ll call these “smoke-lovers” and “non-smoke-lovers”. Smoke-lovers ascribe smoking +10 utility. Non-smoke-lovers assign smoking -1 due to the expense of obtaining something to smoke. Also, no robot wants to be destroyed; all robots ascribe this -100 utility.

There is a hunter who systematically destroys all smoke-loving robots, whether they choose to smoke or not. We can imagine that the robots have serial numbers, which they cannot remove or obscure. The hunter has a list of smoke-lover serial numbers, and so, can destroy all and only the smoke-lovers. The robots don’t have access to the list, so their own serial numbers tell them nothing.

So, the payoffs look like this:


  • Smokes:
    • Killed: -90
    • Not killed: +10
  • Doesn’t smoke:
    • Killed: -100
    • Not killed: 0


  • Smokes
    • Killed: -101
    • Not killed: -1
  • Doesn’t smoke:
    • Killed: -100
    • Not killed: 0

All robots know all of this. They just don’t know their own utility function (epistemically).

If we suppose that the robots are EDT agents with epsilon-exploration, what happens?

Non-smoke-lovers have no reason to ever smoke, so they’ll only smoke with probability epsilon. The smoke lovers are more complicated.

The expected utility for different actions depends on the frequency of those actions in the population of smoke-lovers and non-smoke-lovers, so there’s a Nash-equilibrium type solution. It couldn’t be that all agents choose not to smoke except epsilon often; then, smoking would provide no evidence, so the smoke-lovers would happily decide to smoke. However, it also can’t be that smoke-lovers smoke and non-smoke-lovers don’t, because then the conditional probability of being killed given that you smoke would be too high.

The equilibrium will be for smoke-lovers to smoke just a little more frequently than epsilon, in such a way as to equalize the EDT smoke-lover’s expected utility for smoking and not smoking. (We can imagine a very small amount of noise in agent’s utility calculations to explain how this mixed-strategy equilibrium is actually achieved.)

As with the original smoking lesion problem, this looks like a mistake on the part of EDT. Smoking does not increase a robot’s odds of being hunted down and killed. CDT smoke-lovers would choose to smoke.

Furthermore, this isn’t changed at all by trying updateless reasoning. There’s not really any more-ignorant position for an updateless agent to back off to, at least not one which would be helpful. So, it seems we really need CDT for this one.

What should we think of this?

I think the main question here is how this generalizes to other types of lack of self-knowledge. It’s quite plausible that any conclusions from this example depend on the details of my utility-ignorance model, which would mean we can fix things by avoiding utility-ignorance (rather than adopting CDT).

On the other hand, maybe there are less easily avoidable forms of self-ignorance which lead to similar conclusions. Perhaps the argument that CDT outperforms EDT in cases where EDT isn’t able to condition on all its inputs can be formalized. If so, it might even provide an argument which is persuasive to an EDT agent, which would be really interesting.

From my perspective, I don’t think it’s been adequately established that we should prefer updateless CDT to updateless EDT

I agree with this.

It would be nice to have an example which doesn’t arise from an obviously bad agent design, but I don’t have one.

I’d also be interested in finding such a problem.

I am not sure whether your smoking lesion steelman actually makes a decisive case against evidential decision theory. If an agent knows about their utility function on some level, but not on the epistemic level, then this can just as well be made into a counter-example to causal decision theory. For example, consider a decision problem with the following payoff matrix:


  • Smokes:
    • Killed: 10
    • Not killed: -90
  • Doesn’t smoke:
    • Killed: 0
    • Not killed: 0


  • Smokes:
    • Killed: -100
    • Not killed: -100
  • Doesn’t smoke:
    • Killed: 0
    • Not killed: 0

For some reason, the agent doesn’t care whether they live or die. Also, let’s say that smoking makes a smoke-lover happy, but afterwards, they get terribly sick and lose 100 utilons. So they would only smoke if they knew they were going to be killed afterwards. The non-smoke-lover doesn’t want to smoke in any case.

Now, smoke-loving evidential decision theorists rightly choose smoking: they know that robots with a non-smoke-loving utility function would never have any reason to smoke, no matter which probabilities they assign. So if they end up smoking, then this means they are certainly smoke-lovers. It follows that they will be killed, and conditional on that state, smoking gives 10 more utility than not smoking.

Causal decision theory, on the other hand, seems to recommend a suboptimal action. Let \(a_1\) be smoking, \(a_2\) not smoking, \(S_1\) being a smoke-lover, and \(S_2\) being a non-smoke-lover. Moreover, say the prior probability \(P(S_1)\) is \(0.5\). Then, for a smoke-loving CDT bot, the expected utility of smoking is just

\(\mathbb{E}[U|a_1]=P(S_1)\cdot U(S_1\wedge a_1)+P(S_2)\cdot U(S_2\wedge a_1)=0.5\cdot 10 + 0.5\cdot (-90) = -40\),

which is less then the certain \(0\) utilons for \(a_2\). Assigning a credence of around \(1\) to \(P(S_1|a_1)\), a smoke-loving EDT bot calculates

\(\mathbb{E}[U|a_1]=P(S_1|a_1)\cdot U(S_1\wedge a_1)+P(S_2|a_1)\cdot U(S_2\wedge a_1)\approx 1 \cdot 10 + 0\cdot (-90) = 10\),

which is higher than the expected utility of \(a_2\).

The reason CDT fails here doesn’t seem to lie in a mistaken causal structure. Also, I’m not sure whether the problem for EDT in the smoking lesion steelman is really that it can’t condition on all its inputs. If EDT can’t condition on something, then EDT doesn’t account for this information, but this doesn’t seem to be a problem per se.

In my opinion, the problem lies in an inconsistency in the expected utility equations. Smoke-loving EDT bots calculate the probability of being a non-smoke-lover, but then the utility they get is actually the one from being a smoke-lover. For this reason, they can get some “back-handed” information about their own utility function from their actions. The agents basically fail to condition two factors of the same product on the same knowledge.

Say we don’t know our own utility function on an epistemic level. Ordinarily, we would calculate the expected utility of an action, both as smoke-lovers and as non-smoke-lovers, as follows:

\(\mathbb{E}[U|a]=P(S_1|a)\cdot \mathbb{E}[U|S_1, a]+P(S_2|a)\cdot \mathbb{E}[U|S_2, a]\),

where, if \(U_{1}\) (\(U_{2}\)) is the utility function of a smoke-lover (non-smoke-lover), \(\mathbb{E}[U|S_i, a]\) is equal to \(\mathbb{E}[U_{i}|a]\). In this case, we don’t get any information about our utility function from our own action, and hence, no Newcomb-like problem arises.

I’m unsure whether there is any causal decision theory derivative that gets my case (or all other possible cases in this setting) right. It seems like as long as the agent isn’t certain to be a smoke-lover from the start, there are still payoffs for which CDT would (wrongly) choose not to smoke.


by Abram Demski 120 days ago | link

Excellent example.

It seems to me, intuitively, that we should be able to get both the CDT feature of not thinking we can control our utility function through our actions and the EDT feature of taking the information into account.

Here’s a somewhat contrived decision theory which I think captures both effects. It only makes sense for binary decisions.

First, for each action you compute the posterior probability of the causal parents for each decision. So, depending on details of the setup, smoking tells you that you’re likely to be a smoke-lover, and refusing to smoke tells you that you’re more likely to be a non-smoke-lover.

Then, for each action, you take the action with best “gain”: the amount better you do in comparison to the other action keeping the parent probabilities the same:

\[\texttt{Gain}(a) = \mathbb{E}(U|a) - \mathbb{E}(U|a, \texttt{do}(\bar a))\]

(\(\mathbb{E}(U|a, \texttt{do}(\bar a))\) stands for the expectation on utility which you get by first Bayes-conditioning on \(a\), then causal-conditioning on its opposite.)

The idea is that you only want to compare each action to the relevant alternative. If you were to smoke, it means that you’re probably a smoker; you will likely be killed, but the relevant alternative is one where you’re also killed. In my scenario, the gain of smoking is +10. On the other hand, if you decide not to smoke, you’re probably not a smoker. That means the relevant alternative is smoking without being killed. In my scenario, the smoke-lover computes the gain of this action as -10. Therefore, the smoke-lover smokes.

(This only really shows the consistency of an equilibrium where the smoke-lover smokes – my argument contains unjustified assumption that smoking is good evidence for being a smoke lover and refusing to smoke is good evidence for not being one, which is only justified in a circular way by the conclusion.)

In your scenario, the smoke-lover computes the gain of smoking at +10, and the gain of not smoking at 0. So, again, the smoke-lover smokes.

The solution seems too ad-hoc to really be right, but, it does appear to capture something about the kind of reasoning required to do well on both problems.


by Paul Christiano 164 days ago | Scott Garrabrant likes this | link

I like this line of inquiry; it seems like being very careful about the justification for CDT will probably give a much clearer sense of what we actually want out of “causal” structure for logical facts.


by Vadim Kosoy 163 days ago | Abram Demski likes this | link

The claim that “this isn’t changed at all by trying updateless reasoning” depends on the assumptions about updateless reasoning. If the agent chooses a policy in the form of a self-sufficient program, then you are right. On the other hand, if the agent chooses a policy in the form of a program with oracle access to the “utility estimator,” then there is an equilibrium where both smoke-lovers and non-smoke-lovers self-modify into CDT. Admittedly, there are also “bad” equilibria, e.g. non-smoke-lovers staying with EDT and smoke-lovers choosing between EDT and CDT with some probability. However, it seems arguable that the presence of bad equilibria is due to the “degenerate” property of the problem that one type of agents have incentives to move away from EDT whereas another type has exactly zero such incentives.


by Abram Demski 162 days ago | Vadim Kosoy likes this | link

The non-smoke-loving agents think of themselves as having a negative incentive to switch to CDT in that case. They think that if they build a CDT agent with oracle access to their true reward function, they may smoke (since they don’t know what their true reward function is). So I don’t think there’s an equilibrium there. The non-smoke-lovers would prefer to explicitly give a CDT successor a non-smoke-loving utility function, if they wanted to switch to CDT. But then, this action itself would give evidence of their own true utility function, likely counter-balancing any reason to switch to CDT.

I was wondering about what happens if the agents try to write a strategy for switching between using such a utility oracle and a hand-written utility function (which would in fact be the same function, since they prefer their own utility function). But this probably doesn’t do anything nice either, since a useful choice of policy their would also reveal too much information about motives.


by Vadim Kosoy 162 days ago | Abram Demski likes this | link

Yeah, you’re right. This setting is quite confusing :) In fact, if your agent doesn’t commit to a policy once and for all, things get pretty weird because it doesn’t trust its future-self.






This is exactly the sort of
by Stuart Armstrong on Being legible to other agents by committing to usi... | 0 likes

When considering an embedder
by Jack Gallagher on Where does ADT Go Wrong? | 0 likes

The differences between this
by Abram Demski on Policy Selection Solves Most Problems | 0 likes

Looking "at the very
by Abram Demski on Policy Selection Solves Most Problems | 0 likes

Without reading closely, this
by Paul Christiano on Policy Selection Solves Most Problems | 1 like

>policy selection converges
by Stuart Armstrong on Policy Selection Solves Most Problems | 0 likes

Indeed there is some kind of
by Vadim Kosoy on Catastrophe Mitigation Using DRL | 0 likes

Very nice. I wonder whether
by Vadim Kosoy on Hyperreal Brouwer | 0 likes

Freezing the reward seems
by Vadim Kosoy on Resolving human inconsistency in a simple model | 0 likes

Unfortunately, it's not just
by Vadim Kosoy on Catastrophe Mitigation Using DRL | 0 likes

>We can solve the problem in
by Wei Dai on The Happy Dance Problem | 1 like

Maybe it's just my browser,
by Gordon Worley III on Catastrophe Mitigation Using DRL | 2 likes

At present, I think the main
by Abram Demski on Looking for Recommendations RE UDT vs. bounded com... | 0 likes

In the first round I'm
by Paul Christiano on Funding opportunity for AI alignment research | 0 likes

Fine with it being shared
by Paul Christiano on Funding opportunity for AI alignment research | 0 likes


Privacy & Terms