also theory inspired by him https://www.lesswrong.com/posts/ZJY3eotLdfBPCLP3z/theoretical-neuroscience-for-alignment-theory ## things Steven Byrnes' comments under AGI ruin > (I think most people in AI alignment, especially “scaling hypothesis” people, are expecting early AGIs to involve truly mindboggling amounts of compute, followed by some _very long_ period where the required compute _very gradually_ decreases on account of algorithmic advances. That’s not what I expect; instead I expect the discovery of new better learning algorithms with a different scaling curve that zooms to AGI and beyond quite quickly.) > You could also say “DeepMind will just ask their AI what they should do next”. If they do that, then maybe the AI (if they’re doing really great on safety such that the AI answers honestly and helpfully) will reply: “Hey, here’s what you should do, you should let me undergo recursive-self-improvement, and then I’ll be able to think of all kinds of crazy ways to destroy the world, and then I can think about how to defend against all those things”. But if DeepMind is being methodical & careful enough that their AI hasn’t destroyed the world _already_ by this point, I’m inclined to think that they’re _also_ being methodical & careful enough that when the AI proposes to do that, DeepMind will say, “Umm, no, that’s totally nuts and super dangerous, definitely don’t do that, at least don’t do it right now.” And then DeepMind goes back to publishing nice papers on cancer and on beating ML benchmarks and so on for a few more months, and then Meta’s AI kills everyone. > I can think of at least two leaders of would-be AGI development efforts (namely [Yann LeCun](https://www.lesswrong.com/posts/WxW6Gc6f2z3mzmqKs/debate-on-instrumental-convergence-between-lecun-russell) of Meta and [Jeff Hawkins](https://www.lesswrong.com/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and#3_6_Response_to_Jeff_Hawkins_s_argument_against_AGI_accident_risk) of Numenta) who believe (what I consider to be) spectacularly stupid things about AGI x-risk, and have believed those things consistently for decades, despite extensive exposure to good counter-arguments.