On the computational feasibility of forecasting using gamblers discussion post by Vadim Kosoy 4 days ago | discuss
Current thoughts on Paul Christano's research agenda
post by Jessica Taylor 6 days ago | Sam Eisenstat, Paul Christiano and Wei Dai like this | 5 comments

This post summarizes my thoughts on Paul Christiano’s agenda in general and ALBA in particular.

"Like this world, but..."
post by Stuart Armstrong 8 days ago | discuss

A putative new idea for AI control; index here.

Pick a very unsafe goal: $$G=$$“AI, make this world richer and less unequal.” What does this mean as a goal, and can we make it safe?

I’ve started to sketch out how we can codify “human understanding” in terms of human ability to answer questions.

Here I’m investigating the reverse problem, to see whether the same idea can be used to give instructions to an AI.

 Improved formalism for corruption in DIRL discussion post by Vadim Kosoy 10 days ago | discuss
Smoking Lesion Steelman
post by Abram Demski 21 days ago | Sam Eisenstat, Vadim Kosoy, Paul Christiano and Scott Garrabrant like this | 5 comments

It seems plausible to me that any example I’ve seen so far which seems to require causal/counterfactual reasoning is more properly solved by taking the right updateless perspective, and taking the action or policy which achieves maximum expected utility from that perspective. If this were the right view, then the aim would be to construct something like updateless EDT.

I give a variant of the smoking lesion problem which overcomes an objection to the classic smoking lesion, and which is solved correctly by CDT, but which is not solved by updateless EDT.

Delegative Inverse Reinforcement Learning
post by Vadim Kosoy 21 days ago | discuss

We introduce a reinforcement-like learning setting we call Delegative Inverse Reinforcement Learning (DIRL). In DIRL, the agent can, at any point of time, delegate the choice of action to an “advisor”. The agent knows neither the environment nor the reward function, whereas the advisor knows both. Thus, DIRL can be regarded as a special case of CIRL. A similar setting was studied in Clouse 1997, but as far as we can tell, the relevant literature offers few theoretical results and virtually all researchers focus on the MDP case (please correct me if I’m wrong). On the other hand, we consider general environments (not necessarily MDP or even POMDP) and prove a natural performance guarantee.

A cheating approach to the tiling agents problem
post by Vladimir Slepnev 22 days ago | Alex Mennen and Vadim Kosoy like this | 3 comments

(This post resulted from a conversation with Wei Dai.)

Formalizing the tiling agents problem is very delicate. In this post I’ll show a toy problem and a solution to it, which arguably meets all the desiderata stated before, but only by cheating in a new and unusual way.

Here’s a summary of the toy problem: we ask an agent to solve a difficult math question and also design a successor agent. Then the successor must solve another math question and design its own successor, and so on. The questions get harder each time, so they can’t all be solved in advance, and each of them requires believing in Peano arithmetic (PA). This goes on for a fixed number of rounds, and the final reward is the number of correct answers.

Moreover, we will demand that the agent must handle both subtasks (solving the math question and designing the successor) using the same logic. Finally, we will demand that the agent be able to reproduce itself on each round, not just design a custom-made successor that solves the math question with PA and reproduces itself by quining.

Loebian cooperation in the tiling agents problem
post by Vladimir Slepnev 28 days ago | Alex Mennen, Vadim Kosoy, Abram Demski, Patrick LaVictoire and Stuart Armstrong like this | 4 comments

The tiling agents problem is about formalizing how AIs can create successor AIs that are at least as smart. Here’s a toy model I came up with, which is similar to Benya’s old model but simpler. A computer program X is asked one of two questions:

• Would you like some chocolate?

• Here’s the source code of another program Y. Do you accept it as your successor?

Humans are not agents: short vs long term
post by Stuart Armstrong 43 days ago | 2 comments

A putative new idea for AI control; index here.

This is an example of humans not being (idealised) agents.

Imagine a human who has a preference to not live beyond a hundred years. However, they want to live to next year, and it’s predictable that every year they are alive, they will have the same desire to survive till the next year.

 New circumstances, new values? discussion post by Stuart Armstrong 46 days ago | discuss
Cooperative Oracles: Stratified Pareto Optima and Almost Stratified Pareto Optima
post by Scott Garrabrant 50 days ago | Vadim Kosoy, Patrick LaVictoire and Stuart Armstrong like this | 6 comments

In this post, we generalize the notions in Cooperative Oracles: Nonexploited Bargaining to deal with the possibility of introducing extra agents that have no control but have preferences. We further generalize this to infinitely many agents. (Part of the series started here.)

 Futarchy, Xrisks, and near misses discussion post by Stuart Armstrong 50 days ago | Abram Demski likes this | discuss
Futarchy Fix
post by Abram Demski 53 days ago | Scott Garrabrant and Stuart Armstrong like this | 9 comments

Robin Hanson’s Futarchy is a proposal to let prediction markets make governmental decisions. We can view an operating Futarchy as an agent, and ask if it is aligned with the interests of its constituents. I am aware of two main failures of alignment: (1) since predicting rare events is rewarded in proportion to their rareness, prediction markets heavily incentivise causing rare events to happen (I’ll call this the entropy-market problem); (2) it seems prediction markets would not be able to assign probability to existential risk, since you can’t collect on bets after everyone’s dead (I’ll call this the existential risk problem). I provide three formulations of (1) and solve two of them, and make some comments on (2). (Thanks to Scott for pointing out the second of these problems to me; I don’t remember who originally told me about the first problem, but also thanks.)

Divergent preferences and meta-preferences
post by Stuart Armstrong 54 days ago | discuss

A putative new idea for AI control; index here.

In simple graphical form, here is the problem of divergent human preferences:

 Optimisation in manipulating humans: engineered fanatics vs yes-men discussion post by Stuart Armstrong 58 days ago | discuss
 An Approach to Logically Updateless Decisions discussion post by Abram Demski 63 days ago | Sam Eisenstat, Jack Gallagher and Scott Garrabrant like this | 4 comments
AI safety: three human problems and one AI issue
post by Stuart Armstrong 64 days ago | Ryan Carey and Daniel Dewey like this | 2 comments

A putative new idea for AI control; index here.

There have been various attempts to classify the problems in AI safety research. Our old Oracle paper that classified then-theoretical methods of control, to more recent classifications that grow out of modern more concrete problems.

These all serve their purpose, but I think a more enlightening classification of the AI safety problems is to look at what the issues we are trying to solve or avoid. And most of these issues are problems about humans.

Acausal trade: conclusion: theory vs practice
post by Stuart Armstrong 67 days ago | discuss

A putative new idea for AI control; index here.

When I started this dive into acausal trade, I expected to find subtle and interesting theoretical considerations. Instead, most of the issues are practical.

 Acausal trade: being unusual discussion post by Stuart Armstrong 67 days ago | discuss
 Acausal trade: different utilities, different trades discussion post by Stuart Armstrong 67 days ago | discuss
 Acausal trade: trade barriers discussion post by Stuart Armstrong 67 days ago | discuss
 Value Learning for Irrational Toy Models discussion post by Patrick LaVictoire 68 days ago | discuss
 Acausal trade: full decision algorithms discussion post by Stuart Armstrong 68 days ago | discuss
 Acausal trade: universal utility, or selling non-existence insurance too late discussion post by Stuart Armstrong 68 days ago | discuss
Why I am not currently working on the AAMLS agenda
post by Jessica Taylor 71 days ago | Ryan Carey, Marcello Herreshoff, Sam Eisenstat, Abram Demski, Daniel Dewey, Scott Garrabrant and Stuart Armstrong like this | 2 comments

(note: this is not an official MIRI statement, this is a personal statement. I am not speaking for others who have been involved with the agenda.)

The AAMLS (Alignment for Advanced Machine Learning Systems) agenda is a project at MIRI that is about determining how to use hypothetical highly advanced machine learning systems safely. I was previously working on problems in this agenda and am currently not.

Older

### NEW DISCUSSION POSTS

A few thoughts: I agree
 by Sam Eisenstat on Some Criticisms of the Logical Induction paper | 0 likes

Thanks, so to paraphrase your
 by Wei Dai on Current thoughts on Paul Christano's research agen... | 0 likes

> Why does Paul think that
 by Paul Christiano on Current thoughts on Paul Christano's research agen... | 0 likes

Given that ALBA was not meant
 by Wei Dai on Current thoughts on Paul Christano's research agen... | 0 likes

Thank you for writing this.
 by Wei Dai on Current thoughts on Paul Christano's research agen... | 1 like

I mostly agree with this
 by Paul Christiano on Current thoughts on Paul Christano's research agen... | 2 likes

>From my perspective, I don’t
 by Johannes Treutlein on Smoking Lesion Steelman | 2 likes

 by Vadim Kosoy on Some Criticisms of the Logical Induction paper | 0 likes

 by Vadim Kosoy on Some Criticisms of the Logical Induction paper | 0 likes

 by Vadim Kosoy on Some Criticisms of the Logical Induction paper | 0 likes

Yeah, you're right. This
 by Vadim Kosoy on Smoking Lesion Steelman | 1 like

The non-smoke-loving agents
 by Abram Demski on Smoking Lesion Steelman | 1 like