Intelligent Agent Foundations Forumsign up / log in

This is somewhat related to what I wrote about here. If you consider only what I call convex gamblers/traders and fix some weighting (“prior”) over the gamblers then there is a natural convex set of dominant forecasters (for each history, it is the set of minima of some convex function on \(\Delta\mathcal{O}^\omega\).)

reply


Hi Alex!

The definition of \(h^{!k}\) makes sense for any \(h\), that is, the superscript \(!k\) in this context is a mapping from finite histories to sets of pairs as you said. In the line in question we just apply this mapping to \(x_{:n}\) where \(x\) is a bound variable coming from the expected value.

I hope this helps?

reply


Indeed there is some kind of length limit in the website. I moved Appendices B and C to a separate post.

reply

by Vadim Kosoy 57 days ago | link | parent | on: Hyperreal Brouwer

Very nice. I wonder whether this fixed point theorem also implies the various generalization of Kakutani’s fixed point theorem in the literature, such as Lassonde’s theorem about compositions of Kakutani functions. It sounds like it should because the composition of hypercontinuous functions is hypercontinuous, but I don’t see the formal argument immediately since if we have \(x \in *X,\ y \in *Y\) with standard parts \(x_\omega,\ y_\omega\) s.t. \(f(x)=y\), and and \(y' \in *Y,\ z \in *Z\) with standard parts \(y'_\omega=y_\omega,\ z_\omega\) s.t. \(g(y')=z\) then it’s not clear why there should be \(x'\in X,\ z'\in Z\) s.t. with standard parts \(x'_\omega=x_\omega,\ z'_\omega=z_\omega\) s.t. \(g(f(x'))=z'\).

reply


Freezing the reward seems like the correct answer by definition, since if I am an agent following the utility function \(R\) and I have to design a new agent now, then it is rational for me to design the new agent to follow the utility function I am following now (i.e. this action is usually rated as the best according to my current utility function).

reply


Unfortunately, it’s not just your browser. The website truncates the document for some reason. I emailed Matthew about it and ey are looking into it.

reply


I think technical research should be posted here. Moreover, I think that merging IAFF and LW is a bad idea. We should be striving to attract people from mainstream academia / AI research groups rather than making ourselves seem even more eccentric / esoteric.

reply


Note that the problem with exploration already arises in ordinary reinforcement learning, without going into “exotic” decision theories. Regarding the question of why humans don’t seem to have this problem, I think it is a combination of

  • The universe is regular (which is related to what you said about “we can’t see any plausible causal way it could happen”), so a Bayes-optimal policy with a simplicity prior has something going for it. On the other hand, sometimes you do need to experiment, so this can’t be the only explanation.

  • Any individual human has parents that teach em things, including things like “touching a hot stove is dangerous.” Later in life, ey can draw on much of the knowledge accumulated by human civilization. This tunnels the exploration into safe channels, analogously to the role of the advisor in my recent posts.

  • One may say that the previous point only passes the recursive buck, since we can consider all of humanity to be the “agent”. From this perspective, it seems that the universe just happens to be relatively safe, in the sense that it’s pretty hard for an individual human to do something that will irreparably damage all of humanity… or at least it was the case during most of human history.

  • In addition, we have some useful instincts baked in by evolution (e.g. probably some notion of existing in a three dimensional space with objects that interact mechanically). Again, you could zoom further out and say evolution works because it’s hard to create a species that will wipe out all life.

reply


Typos on page 5:

  • “random explanation” should be “random exploration”
  • “Alpa” should be “Alpha”

reply


Hi Tom!

There is a positive probability that the advisor falls into the trap, but this probability goes to \(0\) as the time discount parameter \(t\) goes to \(\infty\) (which is the limit I study here). This follows from the condition \(\beta(t)=\omega(t^{2/3})\) in the Theorem. To give a simple example, suppose that \(\mathcal{A}=\{0,1,2\}\) and the environment is s.t.:

  • When you take action 0, you fall into a trap and get reward 0 forever.

  • When you take action 1, you get reward 0 for the current round and remain in the same state.

  • When you take action 2, you get reward 1 for the current round (unless you are in the trap) and remain in the same state.

In this case, our advisor would have to take action 0 with probability \(\exp\left(-\omega\left(t^{2/3}\right)\right)\) and action 2 has to be more probable than action 1 by a factor of \(\exp\left(\omega\left(t^{-1/3}\right)\right) \approx 1 + \omega\left(t^{-1/3}\right)\).

reply

by Tom Everitt 160 days ago | link

Hi Vadim!

So basically the advisor will be increasingly careful as the cost of falling into the trap goes to infinity? Makes sense I guess.

What is the incentive for the agent not to always let the advisor choose? Is there always some probability that the advisor saves them from infinite loss, or only in certain situations that can be detected by the agent?

reply

by Vadim Kosoy 160 days ago | link

If the agent always delegates to the advisor, it loses a large fraction of the value. Returning again to the simple example above, the advisor on its own is only guaranteed to get expected utility \(1/2 + \omega(t^{-1/3})\) (because it often takes the suboptimal action 1). On the other hand, for any prior over a countable set of environments that includes this one, the corresponding DIRL agent gets expected utility \(1 - o(1)\) on this environment (because it will learn to only take action 2). You can also add an external penalty for each delegation, adjusting the proof is straightforward.

So, the agent has to exercise judgement about whether to delegate, using its prior + past observations. For example, the policy I construct in Lemma A delegates iff there is no action whose expected loss (according to current beliefs) is less than \(\beta(t)^{-1}t^{-1/3}\).

reply

by Tom Everitt 159 days ago | link

So this requires the agent’s prior to incorporate information about which states are potentially risky?

Because if there is always some probability of there being a risky action (with infinitely negative value), then regardless how small the probability is and how large the penalty is for asking, the agent will always be better off asking.

(Did you see Owain Evans recent paper about trying to teach the agent to detect risky states.)

reply

by Vadim Kosoy 159 days ago | link

The only assumptions about the prior are that it is supported on a countable set of hypotheses, and that in each hypothesis the advisor is \(\beta\)-rational (for some fixed \(\beta(t)=\omega(t^{2/3})\)).

There is no such thing as infinitely negative value in this framework. The utility function is bounded because of the geometric time discount (and because the momentary rewards are assumed to be bounded), and in fact I normalize it to lie in \([0,1]\) (see the equation defining \(\mathrm{U}\) in the beginning of the Results section).

Falling into a trap is an event associated with \(\Omega(1)\) loss (i.e. loss that remains constant as \(t\) goes to \(\infty\)). Therefore, we can risk such an event, as long as the probability is \(o(1)\) (i.e. goes to \(0\) as \(t\) goes to \(\infty\)). This means that as \(t\) grows, the agent will spend more rounds delegating to the advisor, but for any given \(t\), it will not delegate on most rounds (even on most of the important rounds, i.e. during the first \(O(t)\)-length “horizon”). In fact, you can see in the proof of Lemma A, that the policy I construct delegates on \(O(t^{2/3})\) rounds.

As a simple example, consider again the toy environment from before. Consider also the environments you get from it by applying a permutation to the set of actions \(\mathcal{A}\). Thus, you get a hypothesis class of 6 environments. Then, the corresponding DIRL agent will spend \(O(t^{2/3})\) rounds delegating, observe which action is chosen by the advisor most frequently, and perform this action forevermore. (The phenomenon that all delegations happen in the beginning is specific to this toy example, because it only has 1 non-trap state.)

If you mean this paper, I saw it?

reply

by Tom Everitt 150 days ago | link

My confusion is the following:

Premises (*) and inferences (=>):

  • The primary way for the agent to avoid traps is to delegate to a soft-maximiser.

  • Any action with boundedly negative utility, a soft-maximiser will take with positive probability.

  • Actions leading to traps do not have infinitely negative utility.

=> The agent will fall into traps with positive probability.

  • If the agent falls into a trap with positive probability, then it will have linear regret.

=> The agent will have linear regret.

So when you say in the beginning of the post “a Bayesian DIRL agent is guaranteed to attain most of the value”, you must mean that in a different sense than a regret sense?

reply

by Vadim Kosoy 150 days ago | link

Your confusion is because you are thinking about regret in an anytime setting. In an anytime setting, there is a fixed policy \(\pi\), we measure the expected reward of \(\pi\) over a time interval \(t\) and compare it to the optimal expected reward over the same time interval. If \(\pi\) has probability \(p > 0\) to walk into a trap, regret has the linear lower bound \(\Omega(pt)\).

On other hand, I am talking about policies \(\pi_t\) that explicitly depend on the parameter \(t\) (I call this a “metapolicy”). Both the advisor and the agent policies are like that. As \(t\) goes to \(\infty\), the probability \(p(t)\) to walk into a trap goes to \(0\), so \(p(t)t\) is a sublinear function.

A second difference with the usual definition of regret is that I use an infinite sum of rewards with geometric time discount \(e^{-1/t}\) instead of a step function time discount that cuts off at \(t\). However, this second difference is entirely inessential, and all the theorems work about the same with step function time discount.

reply

Older

NEW LINKS

NEW POSTS

NEW DISCUSSION POSTS

RECENT COMMENTS

This is somewhat related to
by Vadim Kosoy on The set of Logical Inductors is not Convex | 0 likes

This uses logical inductors
by Abram Demski on The set of Logical Inductors is not Convex | 0 likes

Nice writeup. Is one-boxing
by Tom Everitt on Smoking Lesion Steelman II | 0 likes

Hi Alex! The definition of
by Vadim Kosoy on Delegative Inverse Reinforcement Learning | 0 likes

A summary that might be
by Alex Appel on Delegative Inverse Reinforcement Learning | 1 like

I don't believe that
by Alex Appel on Delegative Inverse Reinforcement Learning | 0 likes

This is exactly the sort of
by Stuart Armstrong on Being legible to other agents by committing to usi... | 0 likes

When considering an embedder
by Jack Gallagher on Where does ADT Go Wrong? | 0 likes

The differences between this
by Abram Demski on Policy Selection Solves Most Problems | 1 like

Looking "at the very
by Abram Demski on Policy Selection Solves Most Problems | 0 likes

Without reading closely, this
by Paul Christiano on Policy Selection Solves Most Problems | 1 like

>policy selection converges
by Stuart Armstrong on Policy Selection Solves Most Problems | 0 likes

Indeed there is some kind of
by Vadim Kosoy on Catastrophe Mitigation Using DRL | 0 likes

Very nice. I wonder whether
by Vadim Kosoy on Hyperreal Brouwer | 0 likes

Freezing the reward seems
by Vadim Kosoy on Resolving human inconsistency in a simple model | 0 likes

RSS

Privacy & Terms