Modified Demski Prior Problem: Revision discussion post by Abram Demski 1349 days ago | Patrick LaVictoire and Scott Garrabrant like this | discuss The argument in A Problem with the Modified Demski Prior does not work as stated. The argument which I gave seems to make sense in the case where the random bits are added as axioms, but fails in the more natural case where we update on them in a Bayesian manner. To re-articulate the problem: suppose we have a boolean sequence $$S_n$$ which we would like to predict, encoded in the language of $$PA$$ plus a special predicate $$P(n)$$ which represents the value $$S_n$$. Values of $$S_n$$ are coming from a biased coin in reality. We can update the modified Demski prior on the values we observe for $$S_n$$ in two ways: by adding them as axioms, or by Bayesian updating. Adding them as axioms, we run into the problem which I described. Sampled programs are required to be consistent with the axioms so far. Programs which make guesses on every $$P(n)$$ will decrease in mass quickly as we add more $$P(n)$$, even if they guess with the true frequencies. Programs which make guesses very rarely will not. Within the set of programs which guess frequently, the next guess will quickly approach the correct frequency. Within programs which guess rarely, however, this will not happen. Despite only rarely making guesses, the overall mass from these uninformed guessers will come to dominate the probability of new $$P(n)$$, because the loss in mass of frequent guessers is exponential in $$n$$, while the mass of infrequent-guessers which guess at a given $$n$$ is roughly the algorithmic probability of $$n$$. If we do a Bayesian update, however, then infrequent-guessers are treated as enumerating some far-out $$n$$ and then filling in the remaining cases by sampling more frequent-guessers. The hypothesis represented by the infrequent-guesser is, therefore, still losing probability mass at about the rate of everything else. There’s no problem. The original problem came up when our MIRIx group was thinking about things like irreducible patterns, where the sequence actually follows from our existing axioms; so, it was reasonable to think as we did. (The actual set-up to formulate this case would have to be more in the style of asymptotic logical uncertainty, not the simpler add-$$P(n)$$-axioms which I discussed above.) However, my formulation when I wrote the forum post got it wrong.

### NEW DISCUSSION POSTS

[Note: This comment is three
 by Ryan Carey on A brief note on factoring out certain variables | 0 likes

There should be a chat icon
 by Alex Mennen on Meta: IAFF vs LessWrong | 0 likes

Apparently "You must be
 by Jessica Taylor on Meta: IAFF vs LessWrong | 1 like

There is a replacement for
 by Alex Mennen on Meta: IAFF vs LessWrong | 1 like

Regarding the physical
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think that we should expect
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think I understand your
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

This seems like a hack. The
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

After thinking some more,
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yes, I think that we're
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

My intuition is that it must
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

To first approximation, a
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Actually, I *am* including
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yeah, when I went back and
 by Alex Appel on Optimal and Causal Counterfactual Worlds | 0 likes

> Well, we could give up on
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes