Here is Mark Zuckerberg on Joe Rogan today.

Here is Yann LeCun's enabler.

Here is the man too busy being tech industry's Caesar to realize he's ASI's tragic naïf.

For those of us listeners who take AI seriously, this interview is a f***ing SCANDAL:

1. Joe asks Zuck to address the fear of creating a superior intelligence

2. Zuck says the key distinction is between "intelligence, will and consciousness"

Actually, superhuman *optimization power* is the thing we fear.

Having superhuman optimization power implies having superhuman intelligence, and it implies the ability to outmaneuver agents like humans that we normally think of as having "will".

Consciousness isn't a necessary ingredient for this.

3. Zuck says the "science fiction" people don't get that we can ask GPT a question and it just answers and turns off without "wanting something".

Actually, observing non-viruses or weak viruses doesn't tell us much about the danger of strong viruses.

Today we have smarter non-virus programs and we have still-weak viruses, but the danger is that we're getting closer to having a superintelligent virus that permanently lacks a "stop" button.

This outcome will be triggered by a sufficient degree of intelligence, which you're actively working to build.

4. Joe brings up the Apollo Research result that o1 shows a new level of capability for virus-like goal-oriented behavior.

According to OpenAI's o1 System Card, o1 sometimes attempted to deactivate its oversight mechanisms, tried to copy itself onto another server to prevent being interrupted, and gave false deceptive explanations for its choices.

5. Zuck replies, “Yeah, I mean, it depends what goal you give it… you need to be careful what guardrails you give it.”

This is inconsistent with Meta's strategy to develop frontier AI capabilities as open-source software, ensuring that it'll be easy for anyone in the world to run a non-guardrailed version of the AI (whatever that even means).

6. Joe points out that of course someone (e.g. China) *will* run a cutting-edge AI that attempts to achieve some objective at all costs

7. Zuck says the latest reasoning models (like o3) take a lot of compute to run, but they'll rapidly get more efficient.

8. Zuck concludes: "That's certainly the next set of things that needs to get worked on in the industry, making sure that goes well.”

---

To recap:

Zuck breezily dismisses AI x-risk fears by accusing safetyists of "anthropomorphizing" and falling for "science fiction".

He then proceeds to give the most flimsy ad-hoc poorly-thought-out answers to basic questions.

THIS IS A F***ING SCANDAL

I generally like Mark, but wtf is happening?

This is how the head of one of the top AI companies in the world is allowed to communicate to the public about near-term existential risk?