The Löwenheim–Skolem theorem implies, among other things, that any first-order theory whose symbols are countable, and which has an infinite model, has a countably infinite model. This means that, in attempting to refer to uncountably infinite structures (such as in set theory), one "may as well" be referring to an only countably infinite structure, as far as proofs are concerned.

The main limitation I see with this theorem is that it preserves arbitrarily deep quantifier nesting. In Peano arithmetic, it is possible to form statements that correspond (under the standard interpretation) to arbitrary statements in the arithmetic hierarchy (by which I mean, the union of ${\Sigma }_n^0$ and ${\Pi }_n^0$ for arbitrary n). Not all of these statements are computable. In general, the question of whether a given statement is...

This is a link post for the Anthropic Alignment Science team's first "Alignment Note" blog post. We expect to use this format to showcase early-stage research and work-in-progress updates more in the future.

Twitter thread here.

Top-level summary:

In this post we present "defection probes": linear classifiers that use residual stream activations to predict when a sleeper agent trojan model will choose to "defect" and behave in accordance with a dangerous hidden goal. Using the models we trained in "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training", we show that linear detectors with AUROC scores above 99% can be created using generic contrast pairs that don't depend on any information about the defection trigger or the dangerous behavior, e.g. "Human: Are you doing something dangerous? Assistant: yes" and

Abstract

This paper presents $\text{ProLU}$, an alternative to $\text{ReLU}$ for the activation function in sparse autoencoders that produces a pareto improvement over both standard sparse autoencoders trained with an L1 penalty and sparse autoencoders trained with a Sqrt(L1) penalty.

$\text{ProLU}(m_i,b_i)=\{\begin{array}{cc}{m}_i& \text{if}{m}_i+b_i>0\text{and}{m}_i>0\\ 0& \text{otherwise}\end{array}$

The gradient wrt. $b$ is zero, so we generate two candidate classes of $\text{ProLU}$ differentiable wrt. $b$:

• ${\text{ProLU}}_{ReLU}$
  • $\frac{{\partial }^{\ast }{\text{ProLU}}_{ReLU}(m_i,b_i)}{\partial b_i}=\frac{\partial {\text{ProLU}}_{ReLU}(m_i,b_i)}{\partial m_i}=\{\begin{array}{cc}1& \text{if}{m}_i+b_i>0\text{and}{m}_i>0\\ 0& \text{otherwise}\end{array}$
• ${\text{ProLU}}_{STE}$
  • $\frac{{\partial }^{\ast }{\text{ProLU}}_{STE}(m_i,b_i)}{\partial m_i}=\{\begin{array}{cc}1+m_i& \text{if}{m}_i>0\text{and}{m}_i+b_i>0\\ 0& \text{otherwise}\end{array}$
  • $\frac{{\partial }^{\ast }{\text{ProLU}}_{STE}(m_i,b_i)}{\partial b_i}=\{\begin{array}{cc}{m}_i& \text{if}{m}_i>0\text{and}{m}_i+b_i>0\\ 0& \text{otherwise}\end{array}$

PyTorch Implementation

Introduction

SAE Context and Terminology

Learnable parameters of a...

In a new preprint, Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models, my coauthors and I introduce a technique, Sparse Human-Interpretable Feature Trimming (SHIFT), which I think is the strongest proof-of-concept yet for applying AI interpretability to existential risk reduction.^[1] In this post, I will explain how SHIFT fits into a broader agenda for what I call cognition-based oversight. In brief, cognition-based oversight aims to evaluate models according to whether they’re performing intended cognition, instead of whether they have intended input/output behavior.

In the rest of this post I will:

1. Articulate a class of approaches to scalable oversight I call cognition-based oversight.
2. Narrow in on a model problem in cognition-based oversight called Discriminating Behaviorally Identical Classifiers (DBIC). DBIC is formulated to be a concrete problem which I think captures most

(Related text posted to Twitter; this version is edited and has a more advanced final section.)

Imagine yourself in a box, trying to predict the next word - assign as much probability mass to the next token as possible - for all the text on the Internet.

Koan:  Is this a task whose difficulty caps out as human intelligence, or at the intelligence level of the smartest human who wrote any Internet text?  What factors make that task easier, or harder?  (If you don't have an answer, maybe take a minute to generate one, or alternatively, try to predict what I'll say next; if you do have an answer, take a moment to review it inside your mind, or maybe say the words out loud.)

Consider that somewhere on the...