We Were Right! Real Inner Misalignment

The doomers, perhaps, but the alignment & inner alignment problems and instrumental convergence are real results. It's pretty easy to find examples of misaligned AI and convergent instrumental goals. My favorite example of a misaligned AI I've heard of is a robotic arm that was trained to pick up a ball. The results were validated by taking a picture humans would grade pass/fail (holding the ball or not holding it). But the robot arm learned to use forced perspective to make it appear as if it were holding the ball, when in fact it was not (like the way people take photos where they "hold up" the leaning tower of Pisa). I heard that story verbally so can't cite it, but here's something similar. [this video] It's not a huge leap from there to say, we cannot guarantee the safe operation of an AI actor controlling a system that actually matters (as in not a video game character). The burden of proof shifts to proving that you've engineered a system that is safe.

That's an accurate assessment of the situation, according to every AI alignment researcher I've seen talk about it, including the relatively optimistic ones. This includes people who are mainly focused on AI capabilities but have real knowledge of alignment. This part in particular caught my eye: "Other assumptions could also break down in the future, like favorable generalization properties during deployment". There have been actual experiments in which AIs appeared to successfully learn their objective in training, and then did something unexpected when released into a broader environment.[1] I've seen some leading AI researchers dismiss alignment concerns, but without actually engaging with the arguments at all. I've seen no serious rebuttals that actually address the things the alignment people are concerned about. [1] [this video]

Not really a characterization of this work (I like mechanistic interpretability even though I consider it a detour in the long run), but of the submission title. It's bad. I suspect it's informed by Rob Miles' characterization of this algorithm as "bizarre and inhuman"[1] AI safety/alignment discourse as a whole is so incredibly bad. It's autodidactic and weird in the bad sense, weird like people who think they are brave iconoclasts but just haven't the curiosity and humility to learn the basics of the field are weird. Add some money from Dustin Moskovitz to Open Philanthropy to Robert Miles' propaganda firehose on youtube, add policy lobbying and buying bankrupt crypto podcasters, and you have the worst that could come from the culture of public intellectualism. The notions they have inserted into this potentially vital field, like inner/outer misalignment and "mesaoptimization" [2] (where simple overfitting is the real and more productive explanation) are deeply misleading, technically illiterate and stoking public fears in directions we should care about the least, siphoning resources and attention from real issues informed by specifics of contemporary ML (chiefly data and curriculum engineering). They're close to putting a lid on LLM development – because they're getting spooked by some vague sense of growing "capabilities"; even though LLMs are evidently a vastly better approach to safe-yet-capable-general-AI than anything any of their thought leaders have ever come up w