   In memoryless Cartesian environments, every UDT policy is a CDT+SIA policy  post by Jessica Taylor 620 days ago  Vadim Kosoy and Abram Demski like this  2 comments  
Summary: I define a memoryless Cartesian environments (which can model many familiar decision problems), note the similarity to memoryless POMDPs, and define a local optimality condition for policies, which can be roughly stated as “the policy is consistent with maximizing expected utility using CDT and subjective probabilities derived from SIA”. I show that this local optimality condition is necesssary but not sufficient for global optimality (UDT).

 
   An Untrollable Mathematician  post by Abram Demski 29 days ago  Alex Appel, Sam Eisenstat, Vadim Kosoy, Jack Gallagher, Paul Christiano, Scott Garrabrant and Vladimir Slepnev like this  1 comment  
Followup to All Mathematicians are Trollable.
It is relatively easy to see that no computable Bayesian prior on logic can converge to a single coherent probability distribution as we update it on logical statements. Furthermore, the nonconvergence behavior is about as bad as could be: someone selecting the ordering of provable statements to update on can drive the Bayesian’s beliefs arbitrarily up or down, arbitrarily many times, despite only saying true things. I called this wild nonconvergence behavior “trollability”. Previously, I showed that if the Bayesian updates on the provabilily of a sentence rather than updating on the sentence itself, it is still trollable. I left open the question of whether some other side information could save us. Sam Eisenstat has closed this question, providing a simple logical prior and a way of doing a Bayesian update on it which (1) cannot be trolled, and (2) converges to a coherent distribution.

 
    Smoking Lesion Steelman II  post by Abram Demski 144 days ago  Tom Everitt and Scott Garrabrant like this  1 comment  
After Johannes Treutlein’s comment on Smoking Lesion Steelman, and a number of other considerations, I had almost entirely given up on CDT. However, there were still nagging questions about whether the kind of selfignorance needed in Smoking Lesion Steelman could arise naturally, how it should be dealt with if so, and what role counterfactuals ought to play in decision theory if CDTlike behavior is incorrect. Today I sat down to collect all the arguments which have been rolling around in my head on this and related issues, and arrived at a place much closer to CDT than I expected.

 
   Delegative Inverse Reinforcement Learning  post by Vadim Kosoy 235 days ago  Alex Appel likes this  11 comments  
We introduce a reinforcementlike learning setting we call Delegative Inverse Reinforcement Learning (DIRL). In DIRL, the agent can, at any point of time, delegate the choice of action to an “advisor”. The agent knows neither the environment nor the reward function, whereas the advisor knows both. Thus, DIRL can be regarded as a special case of CIRL. A similar setting was studied in Clouse 1997, but as far as we can tell, the relevant literature offers few theoretical results and virtually all researchers focus on the MDP case (please correct me if I’m wrong). On the other hand, we consider general environments (not necessarily MDP or even POMDP) and prove a natural performance guarantee.
The use of an advisor allows us to kill two birds with one stone: learning the reward function and safe exploration (i.e. avoiding both the Scylla of “Bayesian paranoia” and the Charybdis of falling into traps). We prove that, given certain assumption about the advisor, a Bayesian DIRL agent (whose prior is supported on some countable set of hypotheses) is guaranteed to attain most of the value in the slow falling time discount (longterm planning) limit (assuming one of the hypotheses in the prior is true). The assumption about the advisor is quite strong, but the advisor is not required to be fully optimal: a “soft maximizer” satisfies the conditions. Moreover, we allow for the existence of “corrupt states” in which the advisor stops being a relevant signal, thus demonstrating that this approach can deal with wireheading and avoid manipulating the advisor, at least in principle (the assumption about the advisor is still unrealistically strong). Finally we consider advisors that don’t know the environment but have some beliefs about the environment, and show that in this case the agent converges to Bayesoptimality w.r.t. the advisor’s beliefs, which is arguably the best we can expect.

 
  Delegative Inverse Reinforcement Learning  post by Vadim Kosoy 235 days ago  Alex Appel likes this  11 comments  
We introduce a reinforcementlike learning setting we call Delegative Inverse Reinforcement Learning (DIRL). In DIRL, the agent can, at any point of time, delegate the choice of action to an “advisor”. The agent knows neither the environment nor the reward function, whereas the advisor knows both. Thus, DIRL can be regarded as a special case of CIRL. A similar setting was studied in Clouse 1997, but as far as we can tell, the relevant literature offers few theoretical results and virtually all researchers focus on the MDP case (please correct me if I’m wrong). On the other hand, we consider general environments (not necessarily MDP or even POMDP) and prove a natural performance guarantee.
The use of an advisor allows us to kill two birds with one stone: learning the reward function and safe exploration (i.e. avoiding both the Scylla of “Bayesian paranoia” and the Charybdis of falling into traps). We prove that, given certain assumption about the advisor, a Bayesian DIRL agent (whose prior is supported on some countable set of hypotheses) is guaranteed to attain most of the value in the slow falling time discount (longterm planning) limit (assuming one of the hypotheses in the prior is true). The assumption about the advisor is quite strong, but the advisor is not required to be fully optimal: a “soft maximizer” satisfies the conditions. Moreover, we allow for the existence of “corrupt states” in which the advisor stops being a relevant signal, thus demonstrating that this approach can deal with wireheading and avoid manipulating the advisor, at least in principle (the assumption about the advisor is still unrealistically strong). Finally we consider advisors that don’t know the environment but have some beliefs about the environment, and show that in this case the agent converges to Bayesoptimality w.r.t. the advisor’s beliefs, which is arguably the best we can expect.

 
  Being legible to other agents by committing to using weaker reasoning systems  post by Alex Mennen 80 days ago  Stuart Armstrong and Vladimir Slepnev like this  1 comment  
Suppose that an agent \(A_{1}\) reasons in a sound theory \(T_{1}\), and an agent \(A_{2}\) reasons in a theory \(T_{2}\), such that \(T_{1}\) proves that \(T_{2}\) is sound. Now suppose \(A_{1}\) is trying to reason in a way that is legible to \(A_{2}\), in the sense that \(A_{2}\) can rely on \(A_{1}\) to reach correct conclusions. One way of doing this is for \(A_{1}\) to restrict itself to some weaker theory \(T_{3}\), which \(T_{2}\) proves is sound, for the purposes of any reasoning that it wants to be legible to \(A_{2}\). Of course, in order for this to work, not only would \(A_{1}\) have to restrict itself to using \(T_{3}\), but \(A_{2}\) would to trust that \(A_{1}\) had done so. A plausible way for that to happen is for \(A_{1}\) to reach the decision quickly enough that \(A_{2}\) can simulate \(A_{1}\) making the decision to restrict itself to using \(T_{3}\). 
 
     Policy Selection Solves Most Problems  post by Abram Demski 85 days ago  Alex Appel and Vladimir Slepnev like this  4 comments  
It seems like logically updateless reasoning is what we would want in order to solve many decisiontheory problems. I show that several of the problems which seem to require updateless reasoning can instead be solved by selecting a policy with a logical inductor that’s run a small amount of time. The policy specifies how to make use of knowledge from a logical inductor which is run longer. This addresses the difficulties which seem to block logically updateless decision theory in a fairly direct manner. On the other hand, it doesn’t seem to hold much promise for the kind of insights which we would want from a real solution.

 
  Policy Selection Solves Most Problems  post by Abram Demski 85 days ago  Alex Appel and Vladimir Slepnev like this  4 comments  
It seems like logically updateless reasoning is what we would want in order to solve many decisiontheory problems. I show that several of the problems which seem to require updateless reasoning can instead be solved by selecting a policy with a logical inductor that’s run a small amount of time. The policy specifies how to make use of knowledge from a logical inductor which is run longer. This addresses the difficulties which seem to block logically updateless decision theory in a fairly direct manner. On the other hand, it doesn’t seem to hold much promise for the kind of insights which we would want from a real solution.

 
   Hyperreal Brouwer  post by Scott Garrabrant 138 days ago  Vadim Kosoy and Stuart Armstrong like this  2 comments  
This post explains how to view Kakutani’s fixed point theorem as a special case of Brouwer’s fixed point theorem with hyperreal numbers. This post is just math intuitions, but I found them useful in thinking about Kakutani’s fixed point theorem and many things in agent foundations. This came out of conversations with Sam Eisenstat. 
 
  Resolving human inconsistency in a simple model  post by Stuart Armstrong 140 days ago  Abram Demski likes this  1 comment  
A putative new idea for AI control; index here.
This post will present a simple model of an inconsistent human, and ponder how to resolve their inconsistency.
Let \(\bf{H}\) be our agent, in a turnbased world. Let \(R^l\) and \(R^s\) be two simple reward functions at each turn. The reward \(R^l\) is thought of as being a ‘longterm’ reward, while \(R^s\) is a shortterm one.

 
   The Happy Dance Problem  post by Abram Demski 96 days ago  Scott Garrabrant and Stuart Armstrong like this  1 comment  
Since the invention of logical induction, people have been trying to figure out what logically updateless reasoning could be. This is motivated by the idea that, in the realm of Bayesian uncertainty (IE, empirical uncertainty), updateless decision theory is the simple solution to the problem of reflective consistency. Naturally, we’d like to import this success to logically uncertain decision theory.
At a research retreat during the summer, we realized that updateless decision theory wasn’t so easy to define even in the seemingly simple Bayesian case. A possible solution was written up in Conditioning on Conditionals. However, that didn’t end up being especially satisfying.
Here, I introduce the happy dance problem, which more clearly illustrates the difficulty in defining updateless reasoning in the Bayesian case. I also outline Scott’s current thoughts about the correct way of reasoning about this problem.

 
  Catastrophe Mitigation Using DRL  post by Vadim Kosoy 96 days ago  3 comments  
Previously we derived a regret bound for DRL which assumed the advisor is “locally sane.” Such an advisor can only take actions that don’t lose any value in the long term. In particular, if the environment contains a latent catastrophe that manifests with a certain rate (such as the possibility of an UFAI), a locally sane advisor has to take the optimal course of action to mitigate it, since every delay yields a positive probability of the catastrophe manifesting and leading to permanent loss of value. This state of affairs is unsatisfactory, since we would like to have performance guarantees for an AI that can mitigate catastrophes that the human operator cannot mitigate on their own. To address this problem, we introduce a new form of DRL where in every hypothetical environment the set of uncorrupted states is divided into “dangerous” (impending catastrophe) and “safe” (catastrophe was mitigated). The advisor is then only required to be locally sane in safe states, whereas in dangerous states certain “leaking” of longterm value is allowed. We derive a regret bound in this setting as a function of the time discount factor, the expected value of catastrophe mitigation time for the optimal policy, and the “value leak” rate (i.e. essentially the rate of catastrophe occurrence). The form of this regret bound implies that in certain asymptotic regimes, the agent attains nearoptimal expected utility (and in particular mitigates the catastrophe with probability close to 1), whereas the advisor on its own fails to mitigate the catastrophe with probability close to 1.

 
   
Older 
 NEW POSTSNEW DISCUSSION POSTS[Delegative Reinforcement
by Vadim Kosoy on Stable Pointers to Value II: Environmental Goals  1 like 
Intermediate update:
The
by Alex Appel on Further Progress on a Bayesian Version of Logical ...  0 likes 
Since Briggs [1] shows that
by 258 on In memoryless Cartesian environments, every UDT po...  2 likes 
This doesn't quite work. The
by Nisan Stiennon on Logical counterfactuals and differential privacy  0 likes 
I at first didn't understand
This is somewhat related to
by Vadim Kosoy on The set of Logical Inductors is not Convex  0 likes 
This uses logical inductors
by Abram Demski on The set of Logical Inductors is not Convex  0 likes 
Nice writeup. Is oneboxing
Hi Alex!
The definition of
by Vadim Kosoy on Delegative Inverse Reinforcement Learning  0 likes 
A summary that might be
by Alex Appel on Delegative Inverse Reinforcement Learning  1 like 
I don't believe that
by Alex Appel on Delegative Inverse Reinforcement Learning  0 likes 
This is exactly the sort of
by Stuart Armstrong on Being legible to other agents by committing to usi...  0 likes 
When considering an embedder
The differences between this
by Abram Demski on Policy Selection Solves Most Problems  1 like 
Looking "at the very
by Abram Demski on Policy Selection Solves Most Problems  0 likes 
