Reading "SITUATIONAL AWARENESS The Decade Ahead" report by Leopold Aschenbrenner, a former member of Open AI's Superalignment Team had an impact on me - in this context especially his warnings about how we need to make AI development a national security priority (I wonder if this signal will reach decision-makers responsible for National Security). You can find it here: https://situational-awareness.ai/wp-content/uploads/2024/06/situationalawareness.pdf and I think this is a good video summary: https://youtu.be/om5KAKSSpNg?si=gWoPuA7TUtWzF9t9 

Here are some long quotes from the relevant part of the report starting from p. 89:

IIIb. Lock Down the Labs: Security for AGI
The nation’s leading AI labs treat security as an afterthought.
Currently, they’re basically handing the key secrets for AGI
to the CCP on a silver platter. Securing the AGI secrets and
weights against the state-actor threat will be an immense
effort, and we’re not on track.
On the current course, the leading Chinese AGI labs
won’t be in Beijing or Shanghai—they’ll be in San Francisco
and London. In a few years, it will be clear that the AGI secrets are the United States’ most important national defense
secrets—deserving treatment on par with B-21 bomber or
Columbia-class submarine blueprints, let alone the proverbial “nuclear secrets”—but today, we are treating them the way we
would random SaaS software. At this rate, we’re basically just
handing superintelligence to the CCP.
All the trillions we will invest, the mobilization of American industrial might, the efforts of our brightest minds—none of that
matters if China or others can simply steal the model weights
(all a finished AI model is, all AGI will be, is a large file on a
computer) or key algorithmic secrets (the key technical breakthroughs necessary to build AGI).
America’s leading AI labs self-proclaim to be building AGI:
they believe that the technology they are building will, before the decade is out, be the most powerful weapon America
has ever built. But they do not treat it as such. They measure
their security efforts against “random tech startups,” not “key
national defense projects.” As the AGI race intensifies—as it
becomes clear that superintelligence will be utterly decisive in
international military competition—we will have to face the
full force of foreign espionage. Currently, labs are barely able
to defend against scriptkiddies, let alone have “North Koreaproof security,” let alone be ready to face the Chinese Ministry
of State Security bringing its full force to bear.
And this won’t just matter years in the future. Sure, who cares
if GPT-4 weights are stolen—what really matters in terms of
weight security is that we can secure the AGI weights down the
line, so we have a few years, you might say. (Though if we’re
building AGI in 2027, we really have to get moving!) But the
AI labs are developing the algorithmic secrets—the key technical
breakthroughs, the blueprints so to speak—for the AGI right
now (in particular, the RL/self-play/synthetic data/etc “next
paradigm” after LLMs to get past the data wall). AGI-level
security for algorithmic secrets is necessary years before AGIlevel security for weights. These algorithmic breakthroughs
will matter more than a 10x or 100x larger cluster in a few
years—this is a much bigger deal than export controls on compute, which the USG has been (presciently!) intensely pursuing. Right now, you needn’t even mount a dramatic espionage
operation to steal these secrets: just go to any SF party or look
through the office windows.
situational awareness 91
Our failure today will be irreversible soon: in the next 12-24
months, we will leak key AGI breakthroughs to the CCP. It will
be the national security establishment’s single greatest regret
before the decade is out.
The preservation of the free world against the authoritarian
states is on the line—and a healthy lead will be the necessary
buffer that gives us margin to get AI safety right, too. The
United States has an advantage in the AGI race. But we will
give up this lead if we don’t get serious about security very
soon. Getting on this, now, is maybe even the single most important thing we need to do today to ensure AGI goes well.
(...) 
The threat model
There are two key assets we must protect: model weights
(especially as we get close to AGI, but which takes years of
preparation and practice to get right) and algorithmic secrets
(starting yesterday).
Model weights
An AI model is just a large file of numbers on a server. This
can be stolen. All it takes an adversary to match your trillions
of dollars and your smartest minds and your decades of work
is to steal this file. (Imagine if the Nazis had gotten an exact
duplicate of every atomic bomb made in Los Alamos.)
If we can’t keep model weights secure, we’re just building AGI
for the CCP (and, given the current trajectory of AI lab security,
even North Korea).
Even besides national competition, securing model weights is
critical for preventing AI catastrophes as well. All of our handwringing and protective measures are for naught if a bad actor
(say, a terrorist or rogue state) can just steal the model and do
whatever they want with it, circumventing any safety layers.
Whatever novel WMDs superintelligence could invent would
rapidly proliferate to dozens of rogue states. Moreover, security
is the first line of defense against uncontrolled or misaligned
AI systems, too (how stupid would we feel if we failed to contain the rogue superintelligence because we didn’t build and
test it in an air-gapped cluster first?).
Securing model weights doesn’t matter that much right now:
stealing GPT-4 without the underlying recipe doesn’t do that
much for the CCP. But it will really matter in a few years, once
we have AGI, systems that are genuinely incredibly powerful.
Perhaps the single scenario that most keeps me up at night is if China
or another adversary is able to steal the automated-AI-researchermodel-weights on the cusp of an intelligence explosion. China could
immediately use these to automate AI research themselves
situational awareness 94
(even if they had previously been way behind)—and launch
their own intelligence explosion. That’d be all they need to
automate AI research, and build superintelligence. Any lead
the US had would vanish.
Moreover, this would immediately put us in an existential race;
any margin for ensuring superintelligence is safe would disappear. The CCP may well try to race through an intelligence
explosion as fast as possible—even months of lead on superintelligence could mean a decisive military advantage—in the
process skipping all the safety precautions any responsible US
AGI effort would hope to take. We would also have to race
through the intelligence explosion to avoid complete CCP dominance. Even if the US still manages to barely pull out ahead
in the end, the loss of margin would mean having to run enormous risks on AI safety.
We’re miles away for sufficient security to protect weights today. Google DeepMind (perhaps the AI lab that has the best
security of any of them, given Google infrastructure) at least
straight-up admits this. Their Frontier Safety Framework outlines security levels 0, 1, 2, 3, and 4 (~1.5 being what you’d
need to defend against well-resourced terrorist groups or cybercriminals, 3 being what you’d need to defend against the
North Koreas of the world, and 4 being what you’d need to
have even a shot of defending against priority efforts by the
most capable state actors).72 They admit to being at level 0
72 Based off of their claimed correspondence of their security levels to RAND’s
weight security report’s L1-L5
(only the most banal and basic measures). If we got AGI and
superintelligence soon, we’d literally deliver it to terrorist
groups and every crazy dictator out there!
Critically, developing the infrastructure for weight security
probably takes many years of lead times—if we think AGI in
~3-4 years is a real possibility and we need state-proof weight
security then, we need to be launching the crash effort now.
Securing weights will require innovations in hardware and
radically different cluster design; and security at this level can’t
be reached overnight, but requires cycles of iteration.
If we fail to prepare in time, our situation will be dire. We will
be on the cusp of superintelligence, but years away from the se-
situational awareness 95
curity necessary. Our choice will be to press ahead, but directly
deliver superintelligence to the CCP—with the existential race
through the intelligence explosion that implies—or wait until
the security crash program is complete, risking losing our lead.
Algorithmic secrets
While people are starting to appreciate (though not necessarily
implement) the need for weight security, arguably even more
important right now—and vastly underrated—is securing algorithmic secrets.
One way to think about this is that stealing the algorithmic
secrets will be worth having a 10x or more larger cluster to the
PRC:
• As discussed in Counting the OOMs, algorithmic progress
is probably similarly as important as scaling up compute to
AI progress. Given the baseline trend of ~0.5 OOMs of compute efficiency a year (+ additional algorithmic “unhobbling”
gains on top), we should expect multiple OOMs-worth of algorithmic secrets between now and AGI. By default, I expect
American labs to be years ahead; if they can defend their
secrets, this could easily be worth 10x-100x compute.
– (Note that we’re willing to incur American investors 100s
of billions of dollars of costs by export controlling Nvidia
chips—perhaps a 3x increase in compute cost for Chinese
labs—but we’re leaking 3x algorithmic secrets all over the
place!)
• Maybe even more importantly, we may be developing the key
paradigm breakthroughs for AGI right now. As discussed previously, simply scaling up current models will hit a wall:
the data wall. Even with way more compute, it won’t be
possible to make a better model. The frontier AI labs are furiously at work at what comes next, from RL to synthetic
data. They will probably figure out some crazy stuff—
essentially, the “AlphaGo self-play”-equivalent for general
intelligence. Their inventions will be as key as the invention of the LLM paradigm originally was a number of years
situational awareness 96
ago, and they will be the key to building systems that go far
beyond human-level. We still have an opportunity to deny
China these key algorithmic breakthroughs, without which
they’d be stuck at the data wall. But without better security
in the next 12-24 months, we may well irreversibly supply
China with these key AGI breakthroughs.
• It’s easy to underrate how important an edge algorithmic
secrets will be—because up until ~a couple years ago, everything was published. The basic idea was out there: scale
up Transformers on internet text. Many algorithmic details
and efficiencies were out there: Chinchilla scaling laws, MoE,
etc. Thus, open source models today are pretty good, and
a bunch of companies have pretty good models (mostly depending on how much $$$ they raised and how big their
clusters are). But this will likely change fairly dramatically
in the next couple years. Basically all of frontier algorithmic
progress happens at labs these days (academia is surprisingly irrelevant), and the leading labs have stopped publishing their advances. We should expect far more divergence
ahead: between labs, between countries, and between the
proprietary frontier and open source models. A few American labs will be way ahead—a moat worth 10x, 100x, or
more, way more than, say, 7nm vs. 3nm chips—unless they
instantly leak the algorithmic secrets.73 

(...)
There’s a real mental dissonance on security at the leading AI labs. They full-throatedly claim to be building AGI this
decade. They emphasize that American leadership on AGI
will be decisive for US national security. They are reportedly
planning 7T chip buildouts that only make sense if you really
believe in AGI. And indeed, when you bring up security, they
nod and acknowledge “of course, we’ll all be in a bunker” and
smirk.
And yet the reality on security could not be more divorced
from that. Whenever it comes time to make hard choices to
prioritize security, startup attitudes and commercial interests
prevail over the national interest. The national security advisor
would have a mental breakdown if he understood the level of
security at the nation’s leading AI labs.
There are secrets being developed right now, that can be used
for every training run in the future and will be the key unlocks
to AGI, that are protected by the security of a startup and will
be worth hundreds of billions of dollars to the CCP.81 The reality
is that, a) in the next 12-24 months, we will develop the key
algorithmic breakthroughs for AGI, and promptly leak them
to the CCP, and b) we are not even on track for our weights to
be secure against rogue actors like North Korea, let alone an
all-out effort by China, by the time we build AGI. “Good security for a startup” simply is not even close to good enough,
and we have very little time before the egregious damage to the
national security of the United States becomes irreversible.
We’re developing the most powerful weapon mankind has
ever created. The algorithmic secrets we are developing, right
now, are literally the nation’s most important national defense
secrets—the secrets that will be at the foundation of the US
and her allies’ economic and military predominance by the
end of the decade, the secrets that will determine whether we
have the requisite lead to get AI safety right, the secrets that
will determine the outcome of WWIII, the secrets that will
determine the future of the free world. And yet AI lab security
is probably worse than a random defense contractor making
bolts.
It’s madness.
Basically nothing else we do—on national competition, and on
AI safety—will matter if we don’t fix this, soon.