Some Comments on the Current Waikato Hospital Security Incident

As I write this the hospital is entering it’s 2nd week of major disruptions caused by a malicious cyber attack. I’ve read some of the comments written on social media and thought some commentary from an IT professional might be of interest to some. I’ll write this up in the form of a Q&A;

1. How were their systems compromised?
   
   My understanding is that an infected eMail attachment was opened by someone. Most computer networks run multiple zones of security; they have “a hard exterior” (firewalls etc) to prevent the bad people from getting in in the first place, but a (relatively) “soft and vulnerable interior” because people on the inside of the security rings (staff) are largely trusted not to act maliciously. There are still layers of security on the “inside” but they’re nowhere near as comprehensive and robust as those that stop the bad guys from getting in from the outside. Inside the security layer things HAVE to be more relaxed because there needs to be a certain level of trust between computers for them to be able to function as intended (eg a mail server that says “I’m going to be the most secure mail server in the world by not accepting mail from anyone nor passing it to anyone” would be fantastic from a security/vulnerability perspective but hopeless at the actual task it was required to perform; so there always has to be a balance/compromise between security and functionality).
   
   Unfortunately (we use that word a lot!) when an infected attachment is run it’s pretty much an invitation from “someone on the inside” that says “welcome – come inside – it’s warmer in here – our defenses are down – come have a look around here – do what you like”. A bit like a school kid who brings a friend home from school and lets them wander around their parents house; the parents may have incredible security to stop burglars breaking in … but their child bypasses all of that – at which point there’s little to stop their friend doing what they like in the house once they’re invited in. Not a great analogy but hopefully you get the idea.
   
2. But what about anti-virus programs – weren’t they running one?
   
   I know nothing about their particular infrastructure, but I can say with confidence that they’ll have had extremely robust anti-virus technology installed. Unfortunately (there’s that word again!) for the most part it can only protect against threats that it already knows about; Whenever something comes along that it doesn’t find suspicious it lets it right through. It’s only when something has been let through – which is realised – sent to the antivirus vendors – who look at it – work out what it does – how it does it – write a countermeasure to it – incorporate that into their definitions – which they then publish – which users have to then download and install (mostly automatically) – then only then do others have protection against that form of attack … and that can take days (or weeks to occur).
   
   Without wishing to over-simplify anything it’s most accurate to say “anti-virus programs are mostly only effective at preventing what some other poor bugger has already been infected with”. I think COVID vaccines are probably a good analogy; a lot of people have to get infected before a need for the vaccine was realised … and even then it takes time to respond to that need.
   
3. Why can’t they just restore everything from a backup?
   
   My understanding is that the backups are also compromised. That’s the thing; the bad guys want to achieve certain objectives – they don’t do this for fun – they do it for financial gain. They don’t want to break in – announce themselves – and then promptly get shut out before they have a chance to leverage their presence. So odds are the virus has just sat there dormant for a period of time until it’s contaminated all the backups (again, I’m just speaking in general terms — I have no knowledge of any of the particulars of the hospital’s situation other than what I’ve read in the media).
   
   The other problem is that you can’t just restore information without compromises; case-in-point just imagine that this happens to a bank and they restored your account to a point where $20,000 worth of deposits didn’t show. I’m sure you see the problem.
   
   In essence they’ve had a failure in a “failure is not an option” environment … and it’s creating one hell of a lot of headaches and one hell of a mess.
   
4. Can the bad guys be found and held to account?
   
   Maybe. The usual way these things work is the bad guys demand payment in bitcoin … which is extremely open, transparent, and traceable … the only problem is that the identity of the malicious actor operating that account remains completely anonymous.
   
   So on one hand it’s difficult or impossible to know who they are … but on the other hand (and depending on how good they are) it’s possible that they’ll make a mistake and leave a clue that someone with the right skills can pick up on. The attackers are probably highly skilled … but there are some awfully highly skilled good guys too. It’s my unwavering personal hope that they good guys unmask them and that the bad guys suffer a very public, unpleasant, and long-lasting consequence that sends the strong message that doing this kind of thing is a REALLY bad idea.
   
5. What could they have done differently?
   
   Pretty safe to assume that question will be getting asked many times over in their “inner circles”. It’s a tough one to answer because the solutions still need to strike a balance between security – cost – recoverability – privacy – accessability – and many other things. From a personal perspective – to cover off the eMail angle – I’m a big fan of using cloud-based eMail solutions like Google run (the commercial version of gMail). In this scenario they run bleeding edge anti-spam and anti-virus filtering (which can be applied to eMails long after they’ve been received because Google is always the ones holding them) and most attachments aren’t ever downloaded – they’re simply handled in Google’s viewer (images, PDFs etc) (along with a similar comprehensive architecture for handling the equivalents of Word & Excel type documents).
   
   Policy would have been “to not open unexpected attachments” but unfortunately (there we go again) (a) it’s just human nature that paople will make mistakes and (b) the nature of the beast is that the attackers only need to be successful once; so a policy that’s upheld by employees 99.999% of the time will STILL open the floodgates. It’s a problem.

In closing I’d just like to say “I feel their pain”; I’ve been involved in a couple of data recovery/reconstruction jobs where clients have opened infected attachments and introduced crypto viruses – and it’s soul-destroying; recovery is time-consuming + expensive + frustrating – and some data loss is inevitable. Preventing it happening again will be expensive – and no guarantees of effectiveness. Many people will be having sleepless nights as part of the recovery operation; my thoughts are with you.