Couldn’t even be bothered to do a basic 3-2-1! Wow

How does this even make sense business wise for AWS?Is their cost per unit so low?

I made an 840TB storage server last month for $15,000.

I have almost 10% of that in my closet RAID5&#x27;d with large part of it backing up constantly to Backblaze for 10$&#x2F;month, running on 10 year old hardware, with basically only the hard drives having any value ... Used a case made of cardboard till I wanted to improve the cooling, and got a used Fractal Design case for 20€._Only_ the kind of combination of incompetence and bad politics here can lead to the kind of % of how much data has been lost here, given the policy was to only save stuff on that &quot;G-drive&quot; and avoid local copies. The &quot;G-drive&quot; they intentionally did not back up because they couldn&#x27;t figure out a solution to at least store a backup across the street ...

&gt;AWS S3 has pricing of about $0.023&#x2F;GB&#x2F;month, which means ... about $20k&#x2F;monthor outright buying hardware capable of storing 850TB for the same $20K one time payment. Gives you some perspective on how overpriced AWS is.

AWS? Linus Tech Tips has run multiple petabyte servers in their server closet just for sponsor money and for the cool of it. No need to outsource your national infrastructure to foreign governments, a moderate (in government terms) investment in a few racks across the country could&#x27;ve replicated everything for maybe half a year&#x27;s worth of Amazon subscription fees.

It&#x27;s even worse. According to other articles [1], the total data of &quot;G drive&quot; was 858 TB.It&#x27;s almost farcical to calculate, but AWS S3 has pricing of about $0.023&#x2F;GB&#x2F;month, which means the South Korean government could have reliable multi-storage backup of the whole data at about $20k&#x2F;month. Or about $900&#x2F;month if they opted for &quot;Glacier deep archive&quot; tier ($0.00099&#x2F;GB&#x2F;month).They did have backup of the data ... in the same server room that burned down [2].[1] <a href="https:&#x2F;&#x2F;www.hankyung.com&#x2F;article&#x2F;2025100115651" rel="nofollow">https:&#x2F;&#x2F;www.hankyung.com&#x2F;article&#x2F;2025100115651</a>[2] <a href="https:&#x2F;&#x2F;www.hani.co.kr&#x2F;arti&#x2F;area&#x2F;area_general&#x2F;1221873.html" rel="nofollow">https:&#x2F;&#x2F;www.hani.co.kr&#x2F;arti&#x2F;area&#x2F;area_general&#x2F;1221873.html</a>(both in Korean)

On the other hand: backups should also include a version history of some kind, or you&#x27;d be vulnerable to ransomware.

You&#x27;re assuming average worker utilized the full 30G of storage. More likely average was at like 0.3G.

&gt; However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained — meaning all data has been permanently lost.Yikes. You&#x27;d think they would at least have one redundant copy of it all.&gt; erasing work files saved individually by some 750,000 civil servants&gt; 30 gigabytes of storage per personThat&#x27;s 22,500 terabytes, about 50 Backblaze storage pods.Or even just mirrored locally.

I dont know why but cant stop laughing. And the great thing is that they will get paid again to write the same thing.

You jest, but I once had a client who&#x27;s IaC provisioning code was - you guessed it - stored on the very infrastructure which got destroyed.

Here&#x27;s a 2024 incident:&gt; &quot;The outage also hit servers that host procedures meant to overcome such an outage... Company officials had no paper copies of backup procedures, one of the people added, leaving them unable to respond until power was restored.&quot;<a href="https:&#x2F;&#x2F;www.reuters.com&#x2F;technology&#x2F;space&#x2F;power-failed-spacex-mission-control-before-september-spacewalk-by-nasa-nominee-2024-12-17&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reuters.com&#x2F;technology&#x2F;space&#x2F;power-failed-spacex...</a>

Ironically many of those documents for procedures probably lived on that drive...

Ironically, see the phrack article someone linked above

I like the definition of security = confidentiality + integrity + availability.So confidentiality was maintained but integrity and availability were not.

The data seems secure. No cyberthreat actors can access it now. Effective access control: check.

I&#x27;m sure they had dozens of process heavy cybersecurity committees producing hundreds if not thousands of powerpoints and word documents outlining procedures and best practices over the last decade.There is this weird divide between the certified class of non-technical consultants and actual overworked and pushed to corner cut techs.

That may not be a perfect answer. One issue with fire suppression systems and spinning rust drives is that the pressure change etc. from the system can also ‘suppress’ the glass platters in drives as well.

At first you think what an incompetent government would do such things, but even OVH pretty much did the same a few years ago. Destroyed some companies in the progress. A wooden floor in a datacenter with backups in the same building …<a href="https:&#x2F;&#x2F;www.datacenterdynamics.com&#x2F;en&#x2F;news&#x2F;ovhcloud-fire-report-sbg2-data-center-had-wooden-ceilings-no-extinguisher-and-no-power-cut-out&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.datacenterdynamics.com&#x2F;en&#x2F;news&#x2F;ovhcloud-fire-rep...</a>

I agree. No automated fire suppression system for critical infrastructure with no backup?

It’s accurate: <a href="https:&#x2F;&#x2F;www.chosun.com&#x2F;english&#x2F;national-en&#x2F;2025&#x2F;10&#x2F;02&#x2F;FPWGFSXMLNCFPIEGWKZF3BOQ3M&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.chosun.com&#x2F;english&#x2F;national-en&#x2F;2025&#x2F;10&#x2F;02&#x2F;FPWGFS...</a>

Arson? Sounds increasingly like espionage.

Surely there must be something that&#x27;s missing in translation?
This feels like it simply can&#x27;t be right.

It&#x27;s now derived from the word &#x27;gone&#x27;

The name G-Drive is said to be derived from the word ‘government’.

G-drive was simply the name of the storage system

&gt; all documents be stored exclusively on G-DriveDoes G-Drive mean Google Drive, or &quot;the drive you see as G:&quot;?If this is Google Drive, what they had locally were just pointers (for native Google Drive docs), or synchronized documents.If this means the letter a network disk storage system was mapped to, this is a weird way of presenting the problem (I am typing on the black keyboard and the wooden table, so that you know)

Must have been a program without much trickle down into gov tech

Saw a few days ago that the application site for the GKS, the most important scholarship for international students in Korea, went offline for multiple days, surprising to hear that they really lost all of the data though. Great opportunity to build a better website now?But yeah it&#x27;s a big problem in Korea right now, lots of important information just vanished, many are talking about it.

“Final reports and official records submitted to the government are also stored in OnNara, so this is not a total loss”.

How do you come to the conclusion that because things work without certain documents that you can start laying off workers?

Surely having human-resource backups will also help with disaster recovery

&gt;or you can fire a lot of those workersSometimes things can seem to run smoothly for years when neglected... until they suddenly no longer run smoothly!

Long term damage, and risk are two things that don&#x27;t show up with a test like this. Also, often why things go forward is just momentum, built from the past.

Yeah you can do the same with your car too - just gradually remove parts and see what&#x27;s really necessary. Seatbelts, horn, rear doors? Gone. Think of the efficiency!

The fire started on 26th September and news about it reached HN only now. I think this is telling how disruptive for South Korea daily life this accident really was.

Mindblowing. Took a walk. All I can say is that if business continues &quot;as usual&quot; and the economy and public services continue largely unaffected then either there were local copies of critical documents, or you can fire a lot of those workers; either one of those ways the &quot;stress test&quot; was a success.

It would have been better if storage was distributed.

Paper records are usually distributed both by agency and by locality.

For paper documents, you&#x27;d make at least a few copies for storage at the source, and then every receiver will get his&#x2F;her own notarized copies.Electronically, everyone just receives a link to read the document.

Would this have been any different if these documents were stored non-electronically though? I understand that the whole point of electronic data is that it can be backed up, but if the alternative were simply an analog system then it would have fared no better.

One source,<a href="https:&#x2F;&#x2F;www.nytimes.com&#x2F;2025&#x2F;09&#x2F;13&#x2F;world&#x2F;asia&#x2F;nepal-unrest-arson-burn-victims.html" rel="nofollow">https:&#x2F;&#x2F;www.nytimes.com&#x2F;2025&#x2F;09&#x2F;13&#x2F;world&#x2F;asia&#x2F;nepal-unrest-a...</a> (&quot;Many of the nation’s public records were destroyed in the arson strikes, complicating efforts to provide basic health care&quot;)

Not sure where you got that info. only physical documents were burned (intentionally by the incumbents you could argue) however the digital backups were untouched

Funny, because the same thing happened in Nepal a few weeks ago. Protestors&#x2F;rioters burned some government buildings, along with the tech infrastructure within them, so now almost all electronic data is gone.

Totally, backup disasters are a regular occurence (maybe not to the degree of negligence) but the Estonia DR is wild.

Definitely. Especially when considering that there were 95 other systems in this datacentre which do have backups and&gt; The actual number of users is about 17% of all central government officialsFar from all, and they&#x27;re not sure what&#x27;s recoverable yet (&quot;“It’s difficult to determine exactly what data has been lost.”&quot;)Which is not to say that it&#x27;s not big news (&quot;the damage to small business owners who have entered amounts to 12.6 billion Korean won.” The ‘National Happiness Card,’ used for paying childcare fees, etc., is still ‘non-functional.’&quot;), but to put it a bit in perspective and not just &quot;all was lost&quot; as the original submission basically statedQuotes from <a href="https:&#x2F;&#x2F;www.chosun.com&#x2F;english&#x2F;national-en&#x2F;2025&#x2F;10&#x2F;02&#x2F;FPWGFSXMLNCFPIEGWKZF3BOQ3M&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.chosun.com&#x2F;english&#x2F;national-en&#x2F;2025&#x2F;10&#x2F;02&#x2F;FPWGFS...</a> as linked by u&#x2F;layer8 elsewhere in this thread

This comment is in some way more interesting than the topic of the article.

That is absolutely delightful. Estonia is just _good_ at this stuff. Admirable.

This is because everything is in digital form. Essentially all government systems are digital-first, and for the citizen, often digital-only. If the data is lost, there may be no paper records to restore everything from land registry, business registry (operating agreements, ownership records), etc.Without an out-of-country backup, a reversion to previous statuses means the country is lost (Estonia has been occupied a lot). With it, much of the government can continue to function, as an expat government until freedom and independence is restored.

&quot;secured against cyberattacks or crisis situations with KSI Blockchain technology&quot;hmmmm

I wanted to update some paperwork to add my wife as a beneficiary to some accounts. I go to the bank in person and they tell me “call this number, they can add the beneficiary”. I call the number and wait on hold for 30 minutes and then the agent tells me that they will send me an email to update the beneficiary. I get an email over 24 hours later with a PDF THAT I HAVE TO PRINT OUT AND SIGN and then scan and send back to the email. I do that, but then I get another email back saying that there is another form I have to print and sign.This is the state of banking in Canada. God forbid they just put a text box on the banking web app where I can put in my beneficiary.Not to mention our entire health care system still runs on fax!It blows my mind that we have some of the smartest and well educated people in the world with some of the highest gdp per capita in the world and we cannot figure out how to get rid of paper documents. You should be issued a federal digital ID at birth which is attested through a chain of trust back to the federal government. Everything related to the government should be tied back to that ID.

&gt; Estonia follows the “once-only” principle: citizens provide their data just once, and government agencies re-use it securely. The next step is proactive services—where the government initiates service delivery based on existing data, without waiting for a citizen’s request.I wish the same concept was in Canada as well. You absolutely have to resubmit all your information every time you do a request. On top of that, federal government agencies still mail each other the information, so what usually can be done in 1 day takes a whole month to process, assuming the mail post isn&#x27;t on strike (spoiler: they are now).I think Canada is one of the worst countries in efficiency and useless bureaucracy among 1st world countries.

Meanwhile, Estonia has a &quot;data embassy&quot; in Luxembourg: <a href="https:&#x2F;&#x2F;e-estonia.com&#x2F;solutions&#x2F;e-governance&#x2F;data-embassy&#x2F;" rel="nofollow">https:&#x2F;&#x2F;e-estonia.com&#x2F;solutions&#x2F;e-governance&#x2F;data-embassy&#x2F;</a>TL;DR: Estonia operates a Tier 4 (highest security) data center in Luxembourg with diplomatic immunity. Can actively run critical government services in real-time, not just backups.

Thanks! we&#x27;ve added that link to the toptext as well

&quot;equivalent to 50 hard drives&quot; ?

&gt; The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets.This attempt at putting it in perspective makes me wonder what would put it in perspective. &quot;100M sets of harry potter novels&quot; would be one step in the right direction, but nobody can imagine 100M of anything either. Something like &quot;a million movies&quot; wouldn&#x27;t work because they are very different from text media in terms of how much information is in one, even if the bulk of the data is likely media. It&#x27;s an interesting problem even if this article&#x27;s attempt is so bad it&#x27;s almost funnyGood article otherwise though, indeed a lot more detail than the OP. It should probably replace the submission. Edit: dang was 1 minute faster than me :)

Some more details in this article: <a href="https:&#x2F;&#x2F;www.chosun.com&#x2F;english&#x2F;national-en&#x2F;2025&#x2F;10&#x2F;02&#x2F;FPWGFSXMLNCFPIEGWKZF3BOQ3M&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.chosun.com&#x2F;english&#x2F;national-en&#x2F;2025&#x2F;10&#x2F;02&#x2F;FPWGFS...</a>

Hm, care to elaborate. I kinda liked this idea even though I know that it shouldn&#x27;t make much sense but still lol, would this have any benefits over testing backups other than the excitement lol

Testing backups is boring. If you want exciting, test restores!

&gt; The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups.This is why I don&#x27;t really want to run my own cloud :)Actually testing the backups is boring.That said, ones the flames are out, they might actually be able to recover some of it.

It works fine as long as it doesn&#x27;t break, and it&#x27;s cheaper to buy than an equivalently sized system that does have back ups.

Isn&#x27;t that self-evident? Do you have two microwaves from different batches, regularly tested, solely for the eventuality that one breaks? Systems work fine until some (unlikely) risk manifests...Idk if this sounds like I&#x27;m against backups, I&#x27;m not, I&#x27;m just surprised by the question

what&#x27;s the point of a storage system with no back up?

Didn&#x27;t Elon shut that down?[0]: <a href="https:&#x2F;&#x2F;www.cnbc.com&#x2F;2025&#x2F;02&#x2F;13&#x2F;company-ripped-by-elon-musk-that-stores-government-records-in-a-mine-may-actually-get-a-boost-from-doge.html" rel="nofollow">https:&#x2F;&#x2F;www.cnbc.com&#x2F;2025&#x2F;02&#x2F;13&#x2F;company-ripped-by-elon-musk-...</a>

Why is there a &quot;still&quot; in there?

The US government still relies heavily on physical records.

South Korean IT seemed to be stuck in 2007 just not too long ago, would be surprised if it has changed much in the last few years. Do the websites still require you to use internet explorer?

Remember Log4j vulnerability? A lot of the Korea governmental sites weren&#x27;t affected because the Java version was too old :)Don&#x27;t even get me started on ActiveX.

Software and information technology in Korea just sucks.buttons are jpegs&#x2F;gifs, everything is on Java EE and on vulnerable old webservers etc... A lot of government stuff supports only Internet Explorer even though it&#x27;s long dead

&gt; The first thing that comes to mind when I think of the South Korean government is the storied tradition of physical confrontation in their parliament along with more than a few viral videos of brawls and such over the yearsYou&#x27;re thinking of Taiwan, not South Korea.

If only our politicians were young and agile enough to get into brawls.. their speed seems to be more sleeping on the job while democracy crumbles.

The first thing that comes to mind when I think of the South Korean government is the storied tradition of physical confrontation in their parliament along with more than a few viral videos of brawls and such over the years. It used to be better in the US, but with the intensity of discord in our government lately, I don&#x27;t think anyone really knows anymore.

I was going to say, Samsung anything immediately makes me assume the software is awful. With a dose of zero privacy, cloud enabled door-knob or something.

Samsung&#x27;s software is generally terrible; they&#x27;re decent at hardware, not software.

Our incompetence in the US is much more distributed. It wouldn&#x27;t surprise me if the same kind of data isn&#x27;t backed up, but at least it&#x27;s dozens of separate federal agencies not-backing up their data in different physical places.

Well, Elon has a recent copy of everything at least.

Yes. The US government requires offsite backups .They also require routine testing distaster recovery plans.I participated in so many different programs over the years with those tests.Tests that would roll over to facilities across the country

It&#x27;s hard to believe this happened. South Korea has tech giants like Samsung, and yet this is how the government runs? Is the US government any better?

&gt; The Ministry of the Interior and Safety also issued guidelines to each ministry stating, “All work materials should not be stored on office PCs but should be stored on the G-Drive.”They very well might have only been saving to this storage system. It was probably mapped as a drive or shared folder on the PC.

Do they? It&#x27;s not clear if this was two-way sync or access on-demand.Like, I use Google Drive for Desktop but it only downloads the files I access. If I don&#x27;t touch a file for a few days it&#x27;s removed from my local cache.

Theoretically, they still have the primary copies (on each individual person&#x27;s &quot;cloud-enabled&quot; device).

So, someone figured out how to do backups

Yeah, this whole thing smells.Who has the incentive to do this, though? China&#x2F;North Korea? Or someone in South Korea trying to cover up how bad they messed up? Does adding this additional mess on top mean they looked like they messed up less? (And for that to be true, how horrifically bad does the hack have to be?)

Phrack&#x27;s timeline may read like it, but it wasn&#x27;t an onsite inspection due to hacking, but a scheduled maintenance to replace the overdue UPS, hence battery-touching involved. Even the image they linked just says &quot;scheduled maintenance.&quot;

Such coincidences do happen. 20 years ago the plane which was carrying all the top brass of the Russian Black Sea Fleet as well as the Fleet’s accounting documentation for inspection to Moscow burst in flames and fell to the ground while trying to get airborne. Being loaded with fuel it immediately became one large infernal fireball. By some miracle no top brass suffered even minor burn&#x2F;injury while all the accounting documentation burned completely.

&quot;NK hackers&quot; reminds me &quot;my homework was eaten by a dog&quot;. It&#x27;s always NK hackers that steal data&#x2F;crypto and there is absolutely no possibility to do something with it or restore the data, because you know they transfer the info on a hard disk and they shoot it with an AD! Like that general!How do we know it&#x27;s NK? Because there are comments in north-korean language, duh! Why are you asking, are you russian bot or smt??

The good news is: there are still off-site backups.The bad news is: they&#x27;re in North Korea.

Woah, read the timeline at the top of this. The fire happened the very day the government ordered onsite inspection was supposed to start due to Chinese&#x2F;NK hacking.

I don&#x27;t backup my phone. The NSA does it for me!

Silver lining: it&#x27;s likely that technically there is a backup (section 1.3).It&#x27;s just in NK or china.Yikes.

It’s a zine. Been around since the 80’s. Hackers &#x2F; security industry types read and publish to it.

<a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Phrack" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Phrack</a>

It looks delightful, but definitely for and by a specific subculture.

This is the first time I see this site, who&#x2F;what is phrack? A hacker group?

Thanks for this, it gives a lot of extra info and content compared to the original article.

And with no technology! Perhaps become some kind of ascetic monk.

When you see a chronology like that, you don&#x27;t keep trying to speak truth to power.You delete your data, trash your gear, and hop on a bus, to start over in some other city, in a different line of work.

Witness A said, “It appears that the fire started when a spark flew during the process of replacing the uninterruptible power supply,” and added, “Firefighters are currently out there putting out the fire. I hope that this does not lead to any disruption to the national intelligence network, including the government’s 24 channel.”[1][1] <a href="https:&#x2F;&#x2F;mbiz.heraldcorp.com&#x2F;article&#x2F;10584693" rel="nofollow">https:&#x2F;&#x2F;mbiz.heraldcorp.com&#x2F;article&#x2F;10584693</a>

&gt; 27th of September 2025, The fire is believed to have been caused while replacing Lithium-ion batteries. The batteries were manufactured by LG, the parent company of LG Uplus (the one that got hacked by the APT).Compromised batteries or battery controllers?

Well, I think we know &quot;who&quot;dunnit it&#x27;s more of a how-dunnit &amp; are-they-still-in-dunnit

For a moment there I was wondering if “apt down” was a typo and you meant “ifdown”. ;)

“APT Down” is likely a reference to a popular Korean drinking game.<a href="https:&#x2F;&#x2F;www.thetakeout.com&#x2F;1789352&#x2F;korea-apt-drinking-game-rules&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.thetakeout.com&#x2F;1789352&#x2F;korea-apt-drinking-game-r...</a>

Ohh side note but this was the journalist group which was blocked by protonThe timing as well is very suspicious and I think that there can be a lot of discussion about thisRight now, I am wondering about the name most tbh which might seem silly but &quot;APT down - The North Korean files&quot;It seems that APT means in this case advanced persistent threat but I am not sure what they mean by Apt Down, like the fact that it got shut down by their journalism or-? I am sorry if this may seem naive and on a serious note this raises so many questions...

Though this is far from the most important points of this article, why do even the article’s authors defend Proton after having their accounts suspended, and after having seemingly a Korean intelligence official warn them that they weren’t secure? Even if they’re perfectly secure they clearly do not have the moral compass people believe they have.

&gt; KIM is heavily working on ToyBox for Android.2 HN front page articles in 1!

proton is alternative to gmail still. you replace nsa and ad networks with nsa only. it&#x27;s a win.

holy shit lol. this is naked gun level incompetence

<a href="https:&#x2F;&#x2F;phrack.org&#x2F;issues&#x2F;72&#x2F;7_md#article" rel="nofollow">https:&#x2F;&#x2F;phrack.org&#x2F;issues&#x2F;72&#x2F;7_md#article</a>

I thought most liberal governments gave up on those.

I&#x27;ve been putting off a cloud to cloud migration, but apparently it can be done in hours?

Unfortunately, the algorithm to unhash it is written in smoke signals

Technically the data is still in the cloud

Brilliant.This deserves its own HN submission. I submitted it but it was flagged due to the title.Thank you for sharing it on HN.

Copy&#x2F;paste:7 things all kids need to hear1 I love you2 I&#x27;m proud of you3 I&#x27;m sorry4 I forgive you5 I&#x27;m listening6 RAID is not backup. Make offsite backups. Verify backup. Find out restore time. Otherwise, you got what we call Schrödinger backup7 You&#x27;ve got what it takes

<a href="https:&#x2F;&#x2F;mastodon.social&#x2F;@nixCraft&#x2F;113524310004145896" rel="nofollow">https:&#x2F;&#x2F;mastodon.social&#x2F;@nixCraft&#x2F;113524310004145896</a>

This is the reason the 3, 2, 1 rule for backing up exists.

For the German enjoyers among us I recommend also this old song: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=jN5mICXIG9M" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=jN5mICXIG9M</a>

They might be singing this song now. (To the tune of &#x27;Yesterday&#x27; from the Beatles).<pre><code> Yesterday,
 All those backups seemed a waste of pay.
 Now my database has gone away.
 Oh I believe in yesterday.

 Suddenly,
 There’s not half the files there used to be,
 And there’s a deadline
 hanging over me.
 The system crashed so suddenly.

 I pushed something wrong
 What it was I could not say.
 Now my data’s gone
 and I long for yesterday-ay-ay-ay.

 Yesterday,
 The need for back-ups seemed so far away.
 Thought all my data was here to stay,
 Now I believe in yesterday.</code></pre>

Well that works out doesn’t it? Saves them from discovery.

repeat after me:multiple copies; multiple locations; multiple formats.

They are clouds of smoke to begin with. The smoke from the joints of those who believed that storing their data somewhere out of their control was a good idea!

&quot;The day the cloud went up in smoke&quot;

&gt;the G-Drive’s structure did not allow for external backups.ah the so called schrodingers drive. It&#x27;s there unless you try to copy it

Has there been a more recent event, or are you referring to Alexandria?

Watching Mr. Robot and seeing the burned batteries the same time...

No problem — I&#x27;m sure their Supremely nice Leader up north kept a backup. He&#x27;s thoughtful like that...

I don&#x27;t get it. Can you please explain the reference?

Well, now they&#x27;ll have to negotiate with North Korea to get these backups..

Hope this happens to Altman’s data centers.

Someone found the literal HCF instruction.

Too bad this can&#x27;t happen everywhere.

May be a “fast follow”? Right after launch of the “MVP”?

It got pushed a couple sprints and we&#x27;ve got it on the plan for next quarter as long as no new features come in before then.

Why, there&#x27;s nothing left to backup?

If it wasn&#x27;t it most certainly is now

I mean ... was making backups on the backlog at least? Can they at least point to the work item that was going to get done soonish?

If I worked there I&#x27;d have had a hard time believing there were really no backups. Governments can be very nebulous.

I wonder how many IT professionals were begging some incompetent upper management official to do this the right way, but were ignored daily. You&#x27;d think there would be concrete policies to prevent these things...

For sure the only error here is zero redundancy.

Why not? You can easily encrypt your data before sending it for storage on on S3, for example.

On the Microsoft side CVE-2025–55241 is still pretty recent.<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=45282497">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=45282497</a>

S3 features have saved our bacon a number of times. Perhaps your experience and usage is different. They are worth trusting with business critical data as long as you&#x27;re following their guidance. GCP though have not proven it, their data loss news is still fresh in my mind.

Rightfully did not trust these companies. Sure what happened is a disaster for them, but you cant simply trust Amazon &amp; Microsoft.

But not security. And for governmental data security is a far more important consideration.not losing data and keeping untrusted parties out of your data is a hard problem, that &quot;cloud&quot; aka &quot;stored somewhere that is accessible by agents of a foreign nation&quot; does not solve.

Using the cloud would have been the easiest way to achieve the necessary redundancy, but by far not the only one. This is just a flawed concept from the start, with no real redundancy.

For this reason, Microsoft has Azure US Government, Azure China etc

Yeah, I heard that consumer clouds are only locally redundant and there aren&#x27;t even backups. So big DC damage could result in data loss.

“The BBC understands that customers, through various backup technologies, external, were able to recover all lost data.”You backup stuff. To other regions.

...on a single-zone persistent disk: <a href="https:&#x2F;&#x2F;status.cloud.google.com&#x2F;incident&#x2F;compute&#x2F;15056#5719570367119360" rel="nofollow">https:&#x2F;&#x2F;status.cloud.google.com&#x2F;incident&#x2F;compute&#x2F;15056#57195...</a>&gt; GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters.Of course, it&#x27;s perfectly possible to have proper distributed storage without using a cloud provider. It happens to be hard to implement correctly, so apparently, the SK government team in question just decided... not to?

&gt;As a government you should not be putting your stuff in an environment under control of some other nation, period.Why? If you encrypt it yourself before transfer, the only possible control some_other_nation will have over you or your data is availability.

Because these companies never lose data, like during some lightning strikes, oh wait: <a href="https:&#x2F;&#x2F;www.bbc.com&#x2F;news&#x2F;technology-33989384" rel="nofollow">https:&#x2F;&#x2F;www.bbc.com&#x2F;news&#x2F;technology-33989384</a>As a government you should not be putting your stuff in an environment under control of some other nation, period. That is a completely different issue and does not really relate to making backups.

This reads very similar to the Turbo Encabulator video.

I know there is legit hate for VMWare&#x2F;Broadcom but there is a legit case to be made for VCF with an equivalent DR setup where you have replication enabled by Superna and Dell PowerProtect Data Domain protecting both local and remote with Thales Luna K160 KMIP for the data at rest encryption for the vSAN.To add, use F710s, H710s and then add ObjectScale storage for your Kubernetes workloads.This setup repatriates your data and gives you a Cloud like experience. Pair it with like EKS-A and you have a really good on premises Private Cloud that is resilient.

Why not? If the region is in country, encrypted, and with proven security attestations validated by third parties, a backup to a cloud storage would be incredibly wise. Otherwise we might end up reading an article about a fire burning down a single data center

Good thing Korea has cloud providers, apparently Kakao has even gone...beyond the cloud!<a href="https:&#x2F;&#x2F;kakaocloud.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;kakaocloud.com&#x2F;</a>
<a href="https:&#x2F;&#x2F;www.nhncloud.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.nhncloud.com&#x2F;</a>
<a href="https:&#x2F;&#x2F;cloud.kt.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;cloud.kt.com&#x2F;</a>To name a few.

Encrypted backups would have saved a lot of pain here

And yet here is an example where keeping critical data off public cloud storage has been significantly worse for them in the short term.Not that they should just go all in on it, but an encrypted copy on S3 or GCS would seem really useful right about now.

&gt; no government should keep critical data on foreign cloud storagePrimary? No. Back-up?These guys couldn’t provision a back-up for their on-site data. Why do you think it was competently encrypted?

If you can’t encrypt your backups such that you could store them tatooed on Putin’s ass, you need to learn about backups more.

You don’t need cloud when you have the data centre, just backups in physical locations somewhere else

It&#x27;s 2025.  Encryption is a thing now.  You can store anything you want on foreign cloud storage.  I&#x27;d give my backups to the FSB.

Especially on US cloud storage.The data is never safe thanks to the US Cloud Act.

Why not?Has there been any interruption in service?

Agree completely that it&#x27;s absolute wild to run such a system without backups. But at this point no government should keep critical data on foreign cloud storage.

You could be right, but it could also be a bad summary or bad translation.We shouldn&#x27;t rush to judgement.

It could be accurate. Let’s say, for whatever reason, it is.Ok.Then it wasn’t a workable design.The idea of “backup sites” has existed forever. The fact you use the word “cloud” to describe your personal collection of servers doesn’t suddenly mean you don’t need backups in a separate physical site.If the government mandates its use, it should have a hot site at a minimum. Even without that a physical backup in a separate physical location in case of fire&#x2F;attack&#x2F;tsunami&#x2F;large band of hungry squirrels is a total must-have.However it was decided that not having that was OK, that decision was negligence.

Silly to think this is the fault of ops engineers. More likely, the project manager or C-suite didn&#x27;t have time nor budget to allocate on disaster recovery.The project shipped, it&#x27;s done, they&#x27;ve already moved us onto the next task, no one wants to pay for maintenance anyway.This has been my experience in 99% of the companies I have worked for in my career, while the engineers that built the bloody thing groan and are well-aware of all the failure modes of the system they&#x27;ve built. No one cares, until it breaks, and hopefully they get the chance to say &quot;I **** told you this was inadequate&quot;

What a lame excuse. “The G-Drive’s structure did not allow for backups” is a blatant lie. It’s code for, “I don’t value other employees’ time and efforts enough to figure out a reliable backup system; I have better things to do.”Whoever made this excuse should be demoted to a journeyman ops engineer. Firing would be too good for them.

The U.S. Government can’t decrypt data for which it does not possess the key (assuming the encryption used is good).

&gt; The government official who insisted that commercial AWS&#x2F;GCP&#x2F;Azure couldn&#x27;t possibly be trusted with keeping the informationThey were still right though: it&#x27;s absolutely clear without an ounce of doubt that whatever you put on an US cloud is being accessible by the US government, who can also decide to sanction you and deprive you from your ability to access the data yourself.Not having backups is entirely retarded, but also completely orthogonal.

Not sure “sane backup strategy” and “park your whole government in a private company under American jurisdiction” are mutually exclusive. I feel like I can think of a bunch of things that a nation would be sad to lose, but would be even sadder to have adversaries rifling through at will. Or, for that matter, extort favors under threat of cutting off your access.At least in this case you can track down said officials in their foxholes and give them a good talking-to. Good luck holding AWS&#x2F;GCP&#x2F;Azure accountable…

Well it is just malpractise. Even when I was an first semester art student I knew about the concept of off-site backups.

He may or may not have been right, but it&#x27;s besides the point.The 3-2-1 backup rule is basic.

Nothing increases the risk of servers catching fire like government investigators showing up to investigate allegations that North Korea hacked the servers.

&gt; The issue here is not refusing to use a foreign third party. That makes sense.Encrypt before sending to a third party?

Call me a conspiracy theorist, but this kind of mismanagement is intentional by design so powerful people can hide their dirty laundry.

Backups should be far away, too. Apparently some companies lost everything on 9&#x2F;11 because their backups were in the other tower.

This. Speaking specifically from the IT side of things, an employer or customer refusing to do backups is the biggest red flag I can get, an immediate warning to run the fuck away before you get blamed for their failure, stego-tech kind of situation.That being said, I can likely guess where this ends up going:* Current IT staff and management are almost certainly scapegoated for “allowing this to happen”, despite the program in question (G-DRIVE) existing since 2017 in some capacity.* Nobody in government will question sufficiently what technical reason is&#x2F;was given to justify the lack of backups and why that was never addressed, why the system went live with such a glaring oversight, etc, because that would mean holding the actual culprits accountable for mismanagement* Everyone involved is unlikely to find work again anytime soon once names are bandied about in investigations* The major cloud providers will likely win several contracts for “temporary services” that in actuality strip the sovereignty the government had in managing its own system, even if they did so poorly* Other countries will use this to justify outsourcing their own sovereign infrastructure to private enterpriseThis whole situation sucks ass because nothing good is likely to come of this, other than maybe a handful of smart teams lead by equally competent managers using this to get better backup resources for themselves.

&gt; The issue here is not refusing to use a foreign third party. That makes sense.For anyone else who&#x27;s confused, G-Drive means Government Drive, not Google Drive.

Yeah, the whole supposed benefit of an organization using storage the cloud is to avoid stuff like this from happening. Instead, they managed to make the damage far worse by increasing the amount of data lost by centralizing it.

It does only make sense if you are competent enough to manage data, and I mean: Any part of it, forever. It&#x27;s not impossible, of course, but it is really not as trivial as the self-host crowd makes it out to be, if you absolutely need a certain amount of 9s of reliability. There is a reason  why AWS etc can exist. I am sure the cloud market is not entirely reasonable but certainly far more reasonable than relying on some mid consultant to do this for you at this scale.

The issue is without profit incentive of course it isn’t X (backed up, redundant, highly available, whatever other aspect is optimized away by accountants).Having worked a great deal inside of aws on these things aws provides literally every conceivable level of customer managed security down to customer owned and keyed datacenters operated by aws, with master key HSMs owned, purchased by the customer, with customer managed key hierarchies at all levels and detailed audit logs of everything done by everything including aws itself. The security assurance of aws is far and away beyond what even the most sophisticated state actor infrastructure does and is more modern to boot - because it’s profit incentive drives that.Most likely this was not about national security than about nationalism. They’re easily confused but that’s fallacious. And they earned the dividends of fallacious thinking.

I very seriously doubt that the US cares about South Korea&#x27;s deepest, darkest secrets that much, if at all.Not using a cloud provider is asinine. You can use layered encryption so the expected lifetime of the cryptography is beyond the value of the data...and the US government themselves store data on all 3 of them, to my knowledge.I say US because the only other major cloud providers I know of are in China, and they do have a vested interest in South Korean data, presumably for NK.

The issue here is not refusing to use a foreign third party. That makes sense.The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk.That’s gross mismanagement.

Yeah let’s fax all government data to the Trump administration.

cloud will also not back up your stuff if you configure it wrong so not sure how&#x27;s that related

The simple solution here would have been something like a bunch of netapps with snapmirrors to a secondary backup site.Or ZFS or DRBD or whatever homegrown or equivalent non-proprietart alternative is available these days and you prefer.

If only there were a second data center in South Korea where they could backup their data…

When I worked on Apple Maps infra South Korea required all  servers be in South Korea.

There is some data privacy requirement in SK where application servers and data have to remain in the country. I worked for a big global bank and we had 4 main instances of our application: Americas, EMEA, Asia and South Korea.

&gt; The government official who insisted that commercial AWS&#x2F;GCP&#x2F;Azure couldn&#x27;t possibly be trusted with keeping the information will be keeping their head low for a few days then...They absolutely cannot be trusted, especially sensitive govt. data. Can you imagine the US state department getting their hands on compromising data on Korean politicians?Its like handing over the govt. to US interests wholesale.That they did not choose to keep the backup, and then another, at different physical locations is a valuable lesson, and must lead to even better design the next time.But the solution is not to keep it in US hands.

Encryption only makes sense if &quot;the cloud&quot; is just a data storage bucket to you. If you run applications in the cloud, you can&#x27;t have all the data encrypted, especially not all the time. There are some technologies that make this possible, but none are mature enough to run even a small business, let alone a country on.It sounds technologically illiterate to you because when people say &quot;we can&#x27;t safely use a foreign cloud&quot; you think they&#x27;re saying &quot;to store data&quot; and everyone else is thinking at the very least &quot;to store and process data&quot;.Sure, they could have used a cloud provider for encrypted backups, but if they knew how to do proper backups, they wouldn&#x27;t be in this mess to begin with.

I understand data sovereignty in the case where a foreign entity might cut off access to your data, but this paranoia that storing info under your bed is the safest bet is straight up false. We have post-quantum encryption widely available already. If your fear is that a foreign entity will access your data, you&#x27;re technologically illiterate.Obviously no person in a lawmaking position will ever have the patience or foresight to learn about this, but the fact they won&#x27;t even try is all the more infuriating.

&gt; G-Drive’s structure did not allow for external backupsHa! &quot;Did not allow&quot; my ass. Let me translate:&gt; We didn&#x27;t feel like backing anything up or insisting on that functionality.

Days? That&#x27;s optimistic. It depends on what govt cloud contained. For example imagine all the car registrations. Or all the payments for the pension fund

They put everything only in one datacenter. A datacenter located elsewhere should have been setup to mirror.This has nothing to do with commercial clouds. Commercial clouds are just datacenters. They could pick one commercial cloud data center and not do much more to mirror or backup in different regions. I understand some of the services have inherent backups.

Pretty sensible to not host it on these commercial services. What is not so sensible is to not make backups.

your first criticism was they should have handed their data sovereignty over to another country?there are many failure points here, not paying Amazon&#x2F;Google&#x2F;Microsoft is hardly the main point.

Dude, the issues go wayyy beyond opting for selfhosting rather than US clouds.We use selfhosting, but we also test our fire suppression system every year, we have two different DCs, and we use S3 backups out of town.Whoever runs that IT department needs to be run out of the country.

I was once advised to measure your backup security in zip codes and time zones.You have a backup copy of your file, in the same folder? That helps for some &quot;oops&quot; moments, but nothing else.You have a whole backup DRIVE on your desktop? That&#x27;s better. Physical failure of the primary device is no longer a danger. But your house could burn down.You have an alternate backup stored at a trusted friend&#x27;s house across the street? Better! But what if a major natural disaster happens?True life, 30+ years ago when I worked for TeleCheck, data was their lifeblood. Every week a systems operator went to Denver, the alternate site, with a briefcase full of backup tapes. TeleCheck was based in Houston, so a major hurricane could&#x27;ve been a major problem.

The government official who insisted that commercial AWS&#x2F;GCP&#x2F;Azure couldn&#x27;t possibly be trusted with keeping the information will be keeping their head low for a few days then...&quot;The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups.&quot;This is absolutely wild.

<a href="https:&#x2F;&#x2F;www.chosun.com&#x2F;english&#x2F;national-en&#x2F;2025&#x2F;10&#x2F;02&#x2F;FPWGFSXMLNCFPIEGWKZF3BOQ3M&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.chosun.com&#x2F;english&#x2F;national-en&#x2F;2025&#x2F;10&#x2F;02&#x2F;FPWGFS...</a>

Fire destroys S. Korean government's cloud storage system, no backups available