Explain IT: Season 1, Episode 6 - Dealing with an IT Incident

Playing now - Dealing with an IT Security Incident


Six steps to follow when you experience a security incident or breach on your network

In this episode, Mark Overton, Softcat's Information Security Officer, Adrian Richings, Softcat's SIEM lead and Adam Louca, Softcat's Chief Technologist for security join host Michael Bird to play out a security incident scenario. Together they follow a path through six steps to demonstrate the best way to deal with a breach or other security incident on your network, explaining the real life stumbling blocks and outlining the best way to prepare for an incident.

Michael Bird
Michael Bird Digital Marketing Manager Softcat
Adrian Richings
Adrian Richings SIEM Lead Softcat
Mark Overton
Mark Overton Information Security Officer Softcat
Adam Louca
Adam Louca Chief Security Technologist Softcat
Key takeaways
  • Preparing for a security breach is important. Having a strong and easy to follow plan, agreed with key stakeholders, can make the process much easier and straightforward, should a breach occur.
  • Having the right security culture is important – your users should know what to do and who to contact should they suspect a security incident. ‘Champion users’ within the organisation are a good way to help all users validate any suspected security incident.
  • You may consider your network to be watertight, but there will always be weak spots, such as Multi-Factor Authentication (MFA) not being applied on the ActiveSync channel. This leaves it vulnerable to phishing emails.
  • There are six steps to dealing with a security incident – discovery, verification, analysis, response, reporting and review – these are covered in detail in the podcast.
  • A post incident review is key to identifying the successes and areas for improvement. and also to recognise those who worked to rectify the problem.
  • The new GDPR guidelines require a report of any breach within 72 hours of discovery. This comes after the ‘verification’ step of the process and should include timely updates to the ICO as the incident progresses.

Michael Bird: Hello and welcome to Explain IT, brought to you by Softcat. This is a show for IT professionals that aims to simplify the complex and often over complicated bits of enterprise IT without compromising on detail. I'm host Michael Bird and over the next 20 or so minutes I'll be challenging our panel of experts to take a different area of the IT ecosystem and, of course, Explain IT. This week we're going to be looking at how an organisation might deal with a security incident and highlight the ways an organisation should, and importantly, shouldn't respond. So, to take us through how to deal with a security incident are Mark Overton, Softcat’s Information Security Officer, Adrian Richings, Softcat’s SIEM lead and Adam Louca, Softcat’s Chief Technologist for security.

Ok so we're going to try something a little bit different here. What we're going to do is we’re going to take hypothetical scenario, we’re going to put ourselves in the incident response room and we’re going to have a look at how an organisation might deal with it. So, I guess first we need to come up with a hilarious company name. Any suggestions?

Adam Louca: So as an avid fan of The IT Crowd I quite like the idea of using Reynholm Industries.

Michael Bird: Reynholm Industries it is, ok so we need to probably choose an industry, Adrian any suggestions?

Adrian Richings: Maybe an e-commerce, retail type industry?

Michael Bird: Any reason for why we’d pick that?

Adrian Richings: They typically have a large amount of credit and personal identifiable information, always good to a threat actor.

Michael Bird: Ok. Size, Mark?

Mark Overton: We’ll go around the one to two thousand user mark, is quite typical for the companies that Softcat works with.

Michael Bird: Ok so that's say, 2000 and then turnover?

Adam Louca: Let's go for one billion dollars

Michael Bird: Or pounds…

Adam Louca: Well 700 million pounds, based on your local exchange rate.

Michael Bird: Ok and the security incident… I guess let's not reveal the whole incident, let's talk about what we've discovered right now, what the first point of contact is, so Mark what's our security incident?

Mark Overton: Well right now Michael what we know is that a user inside our company has forwarded an email to the security team that is attempting to extort us. Supposedly an external party has gained access to our CRM database and they are asking for half a million pounds.

Michael Bird: Ok

Adam Louca: They missed the opportunity to ask for one million dollars, I don't think they're very good already.

Mark Overton: Well, no, they’re clearly amateurs. Certainly at this point we don't know whether this is true or not.

Michael Bird: So that's a big question mark. Ok so we've discovered this incident in our network by a user that sent us an email. Is that quite common? User sending emails, coming over to us or do we usually have fancy tool that does it?

Adrian Richings: So you can pick them up by various methods - your internal user community can send them through and there are automated SIEM tools and other monitoring tools that would pick it up and obviously you’ve also got the dreaded external notification by either threat actor or law enforcement, they can do that for you too.

Michael Bird: So the breach has been discovered. What are the steps that we then take?

Adrian Richings: So you've got discovery, is the first part. So we'll be looking at end user community, do they know what security breach is? Do they know how to report this to you? There's not much point them having this information and sitting on it. Can they communicate it to your security team? Obviously you’ve got SIEM and automated tooling which helps with your discovery activities and obviously you can also empower the end user community themselves to do that ‘weights and measures’ check with champion users. That helps you do that verification step, so they're not just picking up something and firing it, they might actually be doing some weights and measures checks beforehand to say “Actually this is, or it isn’t security checked.” It depends on how comfortable you are within your organisation with empowering your users.

Adam Louca: And then I guess, Mark, I know something you've been working very heavily on within Softcat is around getting our security culture in the right place and one of the things I know that we’ve had a discussion about is that users look to champions or look to people who are perceived to have more knowledge or more experience in the security area as guidance to what they should do with a certain piece of information, and this all happens very locally. I guess, what's your view on how important that function is, in terms of making sure an incident actually eventually gets back to the security team?

Mark Overton: Well you certainly don't want to discourage the reporting by the users. The flip side to that is many companies, probably like ourselves, will see some users, certainly recently, reporting emails to us from unknown third parties who are asking for their consent to keep their data past the enforcement of GDPR, but actually, in many cases, the users don't recognise these companies and they see it as an unknown company sending them saying we've got your data is that ok? And they’ll report this to us as a potential security incident. So it's challenging, you’re always going to get a volume, but I think most security professionals would agree, they’d rather see anything that users weren't comfortable with, but there’s certainly value if you can get some champions, maybe in local offices and so on, who users feel comfortable engaging with to validate this stuff.

Adam Louca: For sure, and I guess in this scenario, Mark you said previously, the first notification we've had that we’ve potentially got a security incident is an internal member of staff is forwarded us this email. What would typically be the additional things we might want to discover about the email, I guess Adrian, that might help us moving on to the next stage which is verification.

Adrian Richings: So you probably want a copy of the original email with headers and stuff intact because that's normally a good starting point.

Adam Louca: Ok that makes sense, and I guess you want to get that either directly from the user or the mail server. Is there any other places you might want to start to discover information from?

Adrian Richings: So assume you've got the email itself. If it’s got a URL or contact information within that you can start going off and having a look around that as well.

Michael Bird: Ok so how do we actually know that this is a genuine security incident and not just someone having a bit of a laugh or trying to just mess with us?

Adrian Richings: So you obviously going to have to try and validate that information that you've got. So you'll have the whole, traditional ‘what constitutes a security incident’. I'm quite old school in the fact that it's CIA triad, obviously everyone's heard about it – Confidentiality, Integrity, Availability - and if you've got any of those potentially influenced by this, that's a potential security incident. Confidentiality and integrity are obviously the two big hitters in that space. In the case of our extortion, you're going to be looking at the information within that and the quality of the information.

Michael Bird: So just go through that triad again, so confidentiality, integrity…

Adrian Richings: …and availability, CIA.

Michael Bird: So you look at those three things and you say ‘are they ticking any of those three boxes?’ and if they are then…

Adrian Richings: …it's a potential security incident. Availability is the one that's a little bit grey because you have operational incidents that affect availability, but definitely if it's affected confidentiality and it's affected integrity, it’s a security incident.

Mark Overton: And this is the stage now where the rubber really hits the road and we go from verification of an incident which is day to day for most organisations with a security team, to verifying that an incident has actually occurred, which, in the case of losing a large amount of data, like we're talking about here, is not something people are used to dealing with in many circumstances. So once you discover and verify that this has actually occurred, then the tone completely changes; everyone has gone from doing something they’re used to doing day in, day out to something which they were hoping was never going to happen.

Adam Louca: And I guess from that verification perspective, if we’re looking in our scenario that we've received this email from an external party who has claimed to have got a copy of our CRM, I guess we're looking for sample data to verify they actually have that copy of that database?

Adrian Richings: Yeah you’d definitely want to sample data you'd use this then as your fingerprint to go looking within your environment to say what systems hold that data and therefore what potentially is those systems that could have been compromised that could be the source of this breach. Also If you’ve got watermarking, or other types of information embedded in your systems, you can actually give me the particular line for whatever your watermark example is and that will allow you then to categorically confirm whether it is your set or whether it's one that's been amalgamated from one of the various breaches that are out there on the internet.

Adam Louca: I guess that's, interestingly, probably got harder and harder to uniquely verify that it is actually your data. Especially, as you say, now there's so many incidents - there’s billions and billions of records out there and they're just the ones that are widely known about, let alone the stuff that probably trades hands through various nefarious markets.

Mark Overton: Absolutely, I mean if the attackers are being helpful, they may well point you to a token inside your environment say, a text file on a server with information that could only have been put there by them. In those circumstances, again, stuff gets serious pretty quickly.

Adam LoucaYeah, that's probably pretty scary – ‘you've been compromised by such-and-such a group’ as a text message on the desktop is fairly upsetting.

Michael Bird: So we've discovered it, we have verified. Is it now the time that we try and fix it? Or is there something else that we do first?

Adrian Richings: So you've got to spin up the analysis bit now, how has this thing happened? What’s gone on? What’s the scope? What’s the impact? What data is at risk? Because obviously you've got a data breach, you’re going to have to inform a whole lot of regulatory bodies, so there's a whole analysis chunk that now kicks in and that will involve getting all your support teams and all those teams that you might not necessarily deal with in a day-to-day basis and get them involved, so you kind of want to take a moment to think. Having dealt with a few large incidents - Instinct is to jump straight in there and start pulling up systems and grabbing logs and setting up rooms and pulling people in and it all gets very messy and very uncontrolled quickly, so it's good to take a moment and just plan out, based on the information you've got, what steps should I start? And it can all change in the future, but right here, now, today, with information I've got, who do I need to talk to you? Who do I need to pull in? What systems should I start looking at? So take that moment just to have a pause and a think, because that generally catches people out.

Mark Overton: Yes at this point it really does show if you’ve thought about what might happen in this scenario and actually written out a playbook a runbook about how you’re going to respond to it. Many organisations haven't, but at the bare minimum, everyone should at least know if they get out of their depth, who they're going to contact and ideally have spoken to that company before. You hear about incident response retainers, they’re exactly for these scenarios, the point where you have a true validated breach, but you don't keep the resources in house to deal with it, well, at least then, having a third party who knows, has you listed as someone who will call you for assistance so you can get going much faster if you've got that planned out.

Michael Bird: That's interesting, so you pay a company just to sit there, ready and waiting for if an incident happens and then you call them all in and then they… do they come on site and take over, like the CIA with big black briefcases and clear the decks and say ‘right we’re here now, we're going to fix it’?

Adrian Richings: You’d have a retainer, so either by hours or by days or whatever, and yeah you'd say, at a certain engagement point, this is where we'll take the incident, package up the information we’ve got and hand it over to you and you’ll run it on our behalf.

To go back to the scenario, it's quite simple you will have had this notification and hopefully you'll have got your subset of data and you go ‘right which systems have this subset of data in it? And let's go and identify those and you’ll literally go off and do an analysis of each of those, either yourselves, or if you've got a retainer you'll get your forensic guys in and they'll come in, take images of the systems and then look for indications of compromise and they should confirm or rule out those systems as having your sample set in them as well. So basically say, ‘yes that's the potential system that could have been compromised to do this thing’ or ‘no, actually that data in our sample set - only 9 of the 10 records exist in that system, thus it can’t have been that system that was compromised’ or, ‘it couldn't have been wholly that system’ and basically you’ll look at trying to remove systems from the scope of could have been breached and then eventually you'll end up with a handful and that's where you’ll focus most of your attention.

Michael Bird: So is it at this point we understand what's happened?

Adrian Richings: You won't know at this point, all you'll have is an inkling of something might have occurred and obviously somebody is telling you this – you’ve got a sample of data, you're looking to make sure that actually occurred, based on the logs in your environment.

Michael Bird: So at this point we know for definite that our CRM database was compromised, but we don't know how necessarily or why.

Adam Louca: You'll probably have a very strong indication that it has been compromised, but I think you're bang on there, you probably don't have the detail. You'll have, I was going to say persons of interest, but it's probably more systems of interest really.

Michael Bird: Ok so we have all that information. We don't know necessarily how they got in, what the process is, but we’ve verified for definite that they have compromised our CRM database and they have legitimately done that. And we know that for sure. And we have a bit of an inkling about what’s happened. What do we think our inkling is?

Adam Louca: I think in our scenario, we've been given the verification that they've left a text message document on the server in question that contains the CRM database which has got the verification, and then we’ve started to perform some initial analysis on that. So we’ve looked at when that was written, who that was written by, we’ve started to look at some of the metadata.

Adrian Richings: Who was logged in at the time, what processes were running, that type of stuff.

Michael Bird: Ok.

Adam Louca: And that's started to lead us towards a particular user that is of interest, so a privileged account on the network that has access to the CRM database.

Michael Bird: You don't want to be that user do you?!

Adam Louca: You don’t want to be that user!

Adrian Richings: Normally an admin, but hey ho!

Adam Louca: And actually that will start to allow us to move back through the chain, so we are able to see actually, where was that document created from? What client systems were connected in and potentially the authentication logs left there for their time. So in our scenario we're probably now back to the point where we’ve identified a second system of interest that was likely to be their pivot point or their internal position on the network.

Michael Bird: So we've done discovery, we’ve verified it, we've done a bit of analysis, so we're now at the stage where we know for definite that our CRM database has been compromised, what is it that we do now? What is step four?

Adrian Richings: Obviously if you have decided that you want to isolate this thing, so let's go on to the remediation step, we’re going to close the proverbial door now. So we take a copy of that environment - you want to take a forensically sound copy and ideally you want to do a live memory capture, that's one of the more advantageous bits of forensic capture, if you can get that - it's not as easy, but it's good to have cos it gives you what's running as well as what’s on disk - and then obviously you'll take, if there are other systems that are implicated in this, compromised, then obviously you’d do the same thing -  go grab forensic copies of those. And then the CRM system - you need somebody that will be suitably empowered within the business and say, ‘look we've got this system compromised, here is the risk associated with it. Are you happy for us to wipe, reload it from backup or wherever it is that the recovery planned for that particular system is’. But yes, somebody's going to need to sign that off because your lowly security on the list that’s running the incident - he ain't going to be empowered to sign that off, so you're going to have to go back to the business and say, ‘this bad thing’s happened, this is what we’d like to do to recover it. Here’s the cost implications of doing that - what would you like to do, Mr Business? Mr Reynholm!’

Michael Bird: So step four, really is to try and fix the incident. Or is there another step after that which is really really fix it? Or is this just, this is the time when we try and fix it, boot everyone out, that kind of thing.

Mark Overton: There’s going to be some real tough decisions inside any business at this point. In our scenario with the CRM, you are going to be talking about turning that CRM off and disrupting business operations.

Adrian Richings: It's simple maths, generally if the attackers, threat actor is demanding half a million pounds and you’re 20 million pound a day business, you might choose to pay it.

Michael Bird: Wow.

Adam Louca: Yeah. Extortion. I mean, it’s standard extortion at the end of the day. It's a risk/reward kind of value triangle really, and you've got to make a call. I think very interestingly, any companies that operate within global reach or have to send people out to hostage environments or places, potentially, you could get picked up by a group and be held for ransom, will have plans for actually, how much they’ll pay, and in what scenarios they'll pay.

Michael Bird: So that's interesting, I guess it we're not saying here is the ultimate outcome is to clear the security incident from your network, actually one of the options could be, well, actually it's going to be much cheaper, much quicker, much more efficient for us as an organisation to just pay them.

Adam Louca: So there's two sides to this, there is ‘how quick can I get back to functioning’ which is essentially a cost avoidance scenario because that's more about avoiding lost revenues than it is about actually impact and the security breach or security incident – the security incident has happened, there's absolutely nothing you can do to stop or make that any less because in the eyes of anyone outside the organisation, even if you paid the ransom and they said, ‘don’t worry, we’ve deleted the data,’ you can never take that as true. So the data is gone, the horse has bolted, so to speak, but the next question is ‘how little can we spend to get us back making money in a known, good state?’

Adrian Richings: Yeah, and you've got fines and regulatory considerations as well in your cost model, it gets quite interesting at the top of the food chain.

Michael Bird: So going back to our scenario then, what is our response, would do we take? What do we do?

Mark Overton: Well I think we're pretty certain, we’re turning stuff off in our environment. So someone in the security team will need to go up to senior management, advise them what has happened.

Michael Bird: That's a fun conversation to have.

Mark Overton: Definitely a fun conversation, and also at that point you're likely to say, ‘we believe this person is still inside our environment, and in a manner, that they can probably do pretty much anything, so we will be recommending that we start shutting down the systems we know have been affected, and we might also be looking at some areas of external access into the organisation which maybe aren’t completely critical which we can just start turning off whilst we try and work out the extent of this compromise.’

Adam Louca: And this is where, I guess, you almost get a feedback loop really, to some extent, because you’re going to have an initial response and you’re going to report your initial findings, and then you're going to go back for a second go, because actually, in the scenario we've got, we’ve identified potentially two systems of interest, we’ve identified a privileged account of interest, so we’ve got a level of lateral movement, dropped the file on the database server, extracted that CRM and got out, but actually we don't actually know if that's the origin of the actual breach, so it might be the origin of the actual entrance to the network, that is obviously the place where they have made an active communication within our network, but as we've experienced, that typically that's not just going to happen by magic, there's going to be some previous steps and that's probably where you’re going to start looking at who has that user been communicating with? Actually if it’s a piece of malware that's been delivered to that machine to turn it into a Trojan, or into a rat, ultimately that's going to have come from somewhere, so starting to get into more analysis around where that initial pivot point has actually spoken to.

Michael Bird: So we've discovered that an incident has happened, we verified it ,we know for definite that it's not fake, we've done the analysis, we know that it's absolutely from our CRM database and the response we’ve taken – we’ve switched off our CRM system and maybe some other associated services, plus any external access that isn't required or isn't business critical potentially. Are we done now?

Adam Louca: No, so we’re going to now go back through the loop again, so we’re going to go back into that analysis loop and dig deeper into it. So I guess the way you’ve got to think about this is that ultimately, depending on the complexity and the level of the incident, there's potentially going to be several iterations for analysis and response as you take in more information and use that to build up a better picture.

Adrian Richings: It’s the spider’s web picture - you've got your CRM system in the centre that you know is compromised, right, ok, so how did it get there? So on the outer one, well in our scenario he compromised at least one client PC, he might have compromised two or three and it was just that one that managed to talk to the CRM system, and ok, so how did he compromise those PCs, well there was a phishing mechanism or a piece of malware that was used, so that's the next layer in the spider’s web. You’ll always track back, you're not just going to turn off your CRM system, you’re going to look for all those points where he's potentially touched and go, ‘right, they’re tainted, I'm not going to trust them,’ let’s reload, reimage, whatever, restore them to good. And it's not just the infrastructure either; people, processes, all that kind of good stuff needs to be looked at. Was the security posture of this asset or CRM system not particularly good? Was it not up to date? Did it not have a particularly hardened build, or it had a hardened build that hadn’t been looked at in X many years and it's moved on, it's on an old version of operating system that’s end of life? It’s not just a case of closing the proverbial door, you need to actually look around to prevent it reoccurring.

Michael Bird: So let's talk through our incident then. So Adam, can you take us up the chain?

Adam Louca: So I guess the scenario we started with is that the end result is that we've had the CRM compromised and the database has been exfiltrated. That has been accessed by a privileged user from an internal client, so they've laterally moved from a compromised internal client using a privileged account - potentially something really simple like local admin being the same on every machine. It’s one we see quite a lot, a lot of people love to change, the domain admins really look after domain accounts but then everyone's got the same 14 character admin password that’s broken with a good hashing algorithm in a few hours. So we’ve noticed that, we’ve then taken this image and we’ve seen that actually, ok there's a Trojan on that machine and that's connecting back to a known CNC address. We’ve seen that that infection has come in via by a URL link that was sent to the user from another internal user, so they’ve leveraged the fact that the user trusted another user to click on a link to download this piece of software, and then we’ve gone back to that initial point of entry so that initial user that actually sent the email to the second user, and we’ve identified that they have also received a phishing email asking for their credentials. So this is where we start to get a feeling that this was probably the start point, that actually all of this whole incident has started from something that is very simple, very basic. It was a pretty standard Office 365 or any digital online service phishing link. Now interestingly, one of the things we’ve started to actually see here at Softcat, which is something I think people should be aware of, is that a lot of people have naturally gone to the position that, as long as I’ve got MFA, so Multi Factor Authentication or 2FA, Dual Factor Authentication, that phishing is a closed problem, it's a done deal, you know, it’s not a problem, I can let my credentials run wild.

Michael Bird: So what have we discovered then? Where are we? What do we know now?

Adam Louca: Ok, so we started off, round one of discovery, that we knew it was our CRM involved and that was really the first iteration. So we went through that discovery analysis – CRM’s the problem, we verified that it is an issue. Second iteration, we will have verified the text file that had been dropped by the external threat actor as their verification that they had remote code execution on that box. That would have identified the user that created that, the timestamp when it was created and potentially the system that remotely accessed that system to actually drop that file on. So that’s our second iteration, so we’ve now identified this second client in the network that has been potentially compromised. So we’ve now gone to that client, we’ve gone, ‘ok, what information can we grab from this, potentially,’ memory images from it, looking the proxy logs, looking at the sim, looking at wherever that client is connected, we’ve identified a piece of malware on that machine and we've identified that malware has been delivered by an internal email. So it's been delivered by an email sent from a member of staff to another member of staff, and saying, essentially, click and download this link. Potentially there’s an element of social engineering as part of that – they’ve chosen somebody who normally communicates with each other and shares information. So we've now identified that. That's now given us a few bits of information, number one, about potentially some of the malware characteristics that they’re using, potentially some of the back channel or C2 command and control communication methods that exist that allow the attacker to be active on your network, and obviously we've identified this other user that sent this piece of spearfishing to you with the malicious link. So now we’re going back out into that concentric circle again, so we’re out one more layer, we’re going to go and have a look at that initial user that sent that malware over and at that point we now identify that that user had received a fairly benign, or almost run-of-the-mill phishing attack that had caused them to disclose their credentials. And I think now it's at this point where we would probably be scratching our heads a little bit because as IT users or IT administrators, we know that we actually, in our scenario, have Multi Factor Authentication turned on, so this is where we’d probably be scratching our head and scratching our head trying to look at it and we’d probably go and have a look at potentially the Office 365 or Exchange online or potentially our Exchange server logs and actually see the authentication has come via ActiveSync and actually then the light bulb will click off that even though we have Multi Factor Authentication, as many people may not have realised, but probably will now when they think about it, when was the last time you ever asked for a Multi Factor Authentication when you set up the iOS Mail client or the Android mail client? Actually that control area isn't applied on the ActiveSync channel, so we’re actually seeing this very much in the real world, that the typical MFA solutions that people are deploying to defend against standard phishing attacks are becoming less effective as people migrate towards using ActiveSync as the mechanism for delivering phishing emails.

Michael Bird: So we now know this information, what is the next step, Adrian?

Adrian Richings: So assuming all that information that you’ve collated already and you’ve gone looking in the rest of your environment for, to see if it's present, because obviously you don't want to make the assumption that it's this one single person that received the email, this one single ingress point into your network, because there may be multiple points, and it was just this one particular user or link that was successful for our threat actor. So once you’ve packaged all that up and you’re successful that you know all those points in your infrastructure that have been, or could potentially be compromised, then you can look to go, ‘ok the analysis bit’s done, I've got a root cause,’ bearing in mind some of the findings from an incident like this might not be straightforward and immediate fix if, for instance, your anti virus product’s not up to the proverbial mustard. You might want to say, ‘actually let's move it to the latest and greatest,’ that's a whole chunk of work, especially in a large organisation, so yeah, you do the root cause analysis, once you're happy you’ve got all the findings, and you’ve remediated and successfully restored all those systems from a known good position, those that were compromised, then you think, ‘actually now let's move onto next stage,’ which is kind of doing some notification reporting bits.

Adam Louca: Ok so that’s really interesting, that reporting stage sounds like it's going to be pretty detailed and it's going to take quite a lot of time. I guess, probably a question for you Mark, as our resident GDPR expert unfortunately… as he throws something at me now…! How does that fit with the breach notification timelines? Have you got a feel at all for the level of detail and the ability for you to actually get something to a supervising authority within that time frame?

Mark Overton: 72 hours for breach reporting from when you become aware of a breach and that's quite a hotly debated position as well - where do you define ‘aware’? For us in this scenario, I would say that moment we validated that we have that text file on our server, we’ve absolutely become aware of it. Now we don't need to provide a comprehensive answer within 72 hours of everything that has happened and all the data that's been affected, the key thing is that we start communicating with the ICO and tell them what we're doing and as we find out more, we keep them updated. That’s the important thing for organisations to do. No one expects anyone to have a complete IR scenario played out and done in 72 hours.

Adrian Richings: In reporting, it's not all or nothing, your reporting will be tailored to the audience, so for instance, I'm assuming the new GDPR notification will come from your legal team, you may choose another part of the organisation, as long as you've defined it, that's grand. They’ll be doing that, but your reporting to your senior manager will be very light and high level whereas your reporting to your technical teams will be very granular, obviously, especially your incident response team will be getting full information, so reporting just isn't ‘package it all up and send it to everyone and hope for the best’, you will need to tailor your reporting to the stakeholders, the recipients, because obviously not everyone needs to know everything, especially with something like this, you certainly don't want it posted on your internal intranet site, but you might want to notify your users that, actually, that really innocuous phishing email that kicked it all off, there might be more out there, or we might see further iterations of it if the threat actor comes back for another go. You may want to do some comms to do awareness bits, so tailoring your report is a key thing here, not just packaging it all up as a single magic bullet - doesn't tend to work.

Mark Overton: We’re talking about reporting internally but in this scenario where we’re Reynholm Industries we’re a well known UK corporation, we’re likely to have to report this externally too and not just directly to the users who have been impacted, but potentially to the media. So who in your organisation is going to have that wonderful job of standing in front of a television camera and explaining not only what's happened, but what you're doing about it and why your company should not be dragged through the coals, when it's very likely that there's very limited information the team has been able to provide you at this stage.

Adam Louca: I think interestingly, the ability to perform that job effectively has a massive impact on the long-term reputational damage an organisation has. I mean, I've seen a number of security incidents where you end up actually coming out of it positively, so if you look at something like LastPass, which was a very, pretty minor breach but actually was still a breach - their ability to grab enough information to allay fears for what is a potentially very sensitive service for end users, you know, password vaulting, actually saw them come out of that incident positively. They were seen that they were articulate, that they had a good grasp on how to deal with incidents, they knew enough information to come back to users immediately and they took steps that were industry recognised as best practice, in terms of resetting passwords and alerting people, even though they were fairly confident that there actually wasn't a major impact to private data.

Michael Bird: Ok, so the incident has happened, we have communicated it out, we’ve figured out what's happened. Is that it? Do we go home, crack on?

Adrian Richings: There's a post incident review, the last step, PIR.

Michael Bird: So step 6 then, post incident review.

Adrian Richings: So effectively this is the bit that goes, ‘right ok we've had this incident, we're still here, we're still being employed, the company’s still viable, it's all good.’ You go back and say, ‘ok what went well? What didn't go so well? What could we have done better?’ If you had any challenges or there was any blockers or constraints within your organisation say, for instance, let's take the example of where we had a third party forensic support and actually the engagement information we've got for the third party support’s out of date so we actually had to ring about 10 minutes and to Google to find the website, actually PIR should say, ‘contact information needs to be up to date so we can respond immediately,’ that type of stuff. So going back and looking at what went well and what could have gone better is quite key because again it goes back to that ooda loop, where you go back and you do the whole improvement process bit. So PIR is really important, and it's the bit that generally gets left off because in a scenario where you're doing multiple incidents it’s like, ‘ok the incident’s done because step five, reporting, we’ve put it all to bed, oh, we’ll do the PIR, post incident stuff next week’ and another incident comes along the next week and, you abandon it, it's gone, so the PIR’s really important, in my opinion.

Michael Bird: Ok so let's just summarise and have a look at what Reynholm Industries did. So, step one we had the security incident which was…

Adam Louca: The exfiltration of the CRM database by an external party, so that was our initial discovery. We were informed by an internal user via email.

Adrian Richings: So that's your discovery phase.

Michael Bird: Step two was verification?

Adam Louca: Yes so we took steps to verify this was actually our CRM database, that these guys had actually exfiltrated it, that the data was both valid and timely. They also provided us a alternative mechanism for verifying that they had obtained access to the server and that was via leaving a text message on the desktop.

Adrian Richings: That’s verification.

Michael Bird: And step three, which was analysis?

Adam Louca: So then at that point we started to analyse the extent of the breach, so not only the additional information that we’d taken from the actual CRM system itself, so the text message that was left on the database, but also looking at the memory, looking at the logs and other events that were correlated to that central system and from there, we then started to perform additional discovery as we tried to understand the extent of the incident.

Michael Bird: And then step four was the response.

Adam Louca: Yeah so there was an initial response and there will be gradual responses over the iterations as we go through that discovery, verification and analysis loop. The initial response was to take the CRM system offline, so this was a decision taken by strategic leaders in the business, that actually we were unable to keep that system online as there was concerns that the attacker was still active within the network. So that system was switched off while it could be restored to a known good state.

Michael Bird: And then we went back from analysis to response, analysis to response until we'd really refined what happened.

Adrian Richings: It's quite iterative, that particular phase, generally isn't it?

Adam Louca: Yeah and it's until we’d reach the end. At the end of the day, I guess it's a good analogy would be, you’re a police crash investigator, you've arrived at the scene of a crash, and you’ve gone, ‘there's been a crash, that's my analysis. I’ve verified there’s been a crash, I’ve seen there’s been a crash and I have taken the car off the motorway.’ If you’d stopped your investigation there, you’d never know actually how the car crashed. Was there other cars involved? Did it skid? Was there a mechanical failure? Was the driver drunk? So it's constantly that iteration loop that gives you a more granular and more refined understanding until eventually you've reached that point where you’ve reached a root cause analysis.

Michael Bird: So we’ve fixed everything, we have understood as much as possible about how it happened. So the step five is the reporting. What do we do for that?

Adrian Richings: So you'd communicate out. You’d normally, as part of your incident you’d have kept an incident log. You’d have had ‘these are the steps that we would take and this is the information that was found, this is by whom’ and all that kind of good stuff, so you’d capture all that and provide it in a format. Depending on who you’d reporting going would depend on the granularity of that report, but assuming it's to your incident response technical teams, they will get a full workup that says ‘this is what we believe happened, here’s the evidence to support that hypothesis’ and obviously you’d communicate that out. And then to your senior management who sees to it, you would probably take a subset of that, effectively give them a summary of what happened. They’ll obviously want to know the implications to costs.

Michael Bird: Then the final step is a post incident review. So just explain that one again.

Adrian Richings: Yeah so that's effectively your process improvement. You go back and you look at how well did this thing go? What could we have done to make it better? You can reward those guys that went out and above and beyond, they stayed all night, they endured, the pizza and all that kind of good stuff and you look back and go, ‘actually, if this happened again I probably want these systems to maybe have better granular logging in a more… I might want that team to engage quicker or I might want to do something,’ so, look at what went well, look at what went not so well and try and make sure that next time you put steps in place so that it goes better.

Adam Louca: And I bet, from our experience, how many of those people who go through post incident reviews say, ‘I really wish I had an incident response company on retainer.’

Mark Overton: ‘I really wish I’d planned what I was going to do before the incident happened, because looking back on it, it looks so much simpler.’ Perfect with hindsight. What we’re talking about here is not the realm of fiction, the reason we're all here in this room talking about it is because we all know first-hand of stories of mid market businesses who this type of stuff has happened to. The attack chain we've covered is absolutely one that we all worry is something that could happen to every company and we have seen happen to companies we work with. There is nothing in here that is overly fancy, most of this stuff can be taken care of by a good security program that follows guidelines and frameworks like CIS and the 10 steps to cyber.

Adrian Richings: Absolutely.

Michael Bird: Excellent, well Adrian, Adam and Mark thank you so much for your time it’s really interesting talking to you all. Listeners if there's anything in the show that has piqued your interest or if you’d like to talk someone at Softcat about anything we’ve talked about on this episode please do check out the show notes, we’ll put some of the stuff we've talked about today as well as some contact details if you want to speak to someone at Softcat. Please also make sure you click subscribe wherever you get your podcast and you've been listening to Explain IT from Softcat. Thanks for listening and goodbye.