 A Problem with the Modified Demski Prior post by Abram Demski 1423 days ago | Patrick LaVictoire and Scott Garrabrant like this | discuss The basic idea of the modified Demski prior is to let Turing machines enumerate sentences, using a Solomonoff-style prior. As such, we might hope that this prior would have the good properties of Solomonoff induction when applied to an empirical problem. This is not the case. The argument here comes out of discussions with Scott. (What follows is not a proof, but only a proof sketch with a hole at the end.) [Update: the argument below contains an error.] The modified Demski prior is defined as follows. Interpret a universal machine $$U$$ as enumerating sentences (giving us sentence codes with a special separator code between them, possibly forever). Define a distribution over such enumerations by feeding the machine random bits. Starting with a base theory (our axioms), keep extending the set by drawing new theories $$T$$ from the distribution, and keeping those which are consistent with the axioms and with what we’ve kept so far. The limit of this process is one random extension of the base theory. Consider the case of first-order logic with the axioms of $$PA$$, and an infinite stock of predicate symbols. (We really need only one additional predicate symbol, however.) Suppose we’ve represented the problem as sequence induction, as we would for the Solomonoff distribution: we have a sequence $$S_n$$ of 0s and 1s, which we want to predict. Encode this with a distinguished predicate not used for anything else, $$S(n)$$. Suppose that $$S_n$$ encodes a stochastic problem, so that no deterministic machine predicts the sequence with 100% accuracy. Let’s say it encodes flips of a biased coin which lands on True 30% of the time. Consider only the first draw from $$U$$, and furthermore, only consider the cases where $$U$$ enumerates sentences of the form $$S(n)$$ or $$\neg S(n)$$ exclusively. Because $$U$$ is enumerating sentences rather than deciding sentences in sequence, it can skip as many sentences as it likes. Suppose that we’ve updated on $$N-1$$ sentences so far. For the $$N$$th sentence, we make a prediction using a mixture distribution over all the machines $$T$$ which haven’t been wrong yet, weighted by their probability of coming from $$U$$. The mixture distribution consisting of machines which have made predictions for all $$S(n)$$ so far is likely to be quite good, predicting very close to .3 on True for sufficiently large $$N$$. Those which have predicted few or none, however, may be quite bad. The problem is that these will have more weight, asymptotically. The argument is as follows: It is sufficient to consider the non-normalized version of the distribution, and compare the relative mass of the different components. The volume of the machines which try all predictions will shrink based on a constant factor (on average), since a certain percent should be getting things wrong. On the other hand, consider the class of machines which hold off making predictions. Some of these machines will take the following form: “Sample from $$U$$, but discard any sentences $$S(n)$$ for \(n

### NEW DISCUSSION POSTS

[Note: This comment is three
 by Ryan Carey on A brief note on factoring out certain variables | 0 likes

There should be a chat icon
 by Alex Mennen on Meta: IAFF vs LessWrong | 0 likes

Apparently "You must be
 by Jessica Taylor on Meta: IAFF vs LessWrong | 1 like

There is a replacement for
 by Alex Mennen on Meta: IAFF vs LessWrong | 1 like

Regarding the physical
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think that we should expect
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

I think I understand your
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

This seems like a hack. The
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

After thinking some more,
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yes, I think that we're
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

My intuition is that it must
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

To first approximation, a
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Actually, I *am* including
 by Vanessa Kosoy on The Learning-Theoretic AI Alignment Research Agend... | 0 likes

Yeah, when I went back and
 by Alex Appel on Optimal and Causal Counterfactual Worlds | 0 likes

> Well, we could give up on
 by Jessica Taylor on The Learning-Theoretic AI Alignment Research Agend... | 0 likes