One hour before the opening ceremony of the 2018 Commonwealth Games was due to start, the network operations team at the event’s network provider, Optus, started seeing massive traffic spikes on their telco network. Here comes the distributed denial of service (DDoS) attack, they thought, and they had plenty of reasons to be worried.
Only two months earlier, the Olympic Destroyer worm had disrupted the opening ceremony of the Winter Olympics in South Korea, deleting files and corrupting systems.
One month before the April 4 ceremony on Australia’s Gold Coast, the internet had seen a DDoS attack hitting 1.3 to 1.7 terabytes of data per second.
Optus wasn’t just the Games’ network provider either. It was a Tier 1 sponsor.
“Our brand was going to be all over everything you saw to do with the Games. That’s like putting a target on your back,” says Narelle Wakely principal security advisor with Trustwave, an Optus company.
“We had similar infrastructure and applications to the Winter Olympics. And so it really put us on heightened alert,” Wakely told APNIC 48, the twice-yearly conference of the Asia Pacific Network Information Centre, in Chiang Mai, Thailand, on Tuesday.
“We had rising escalations between the UK government and the Russian government, with the former spy Sergei Skripal and his daughter getting poisoned on UK soil,” she said.
“We also had the traditional foes of the US and North Korea talking about coming together for face-to-face meetings for the first time that could possibly be happening in Singapore. The timing wasn’t known. Our parent company is a Singaporean company, so that was adding heightened risks.”
The Games network team wasn’t seeing the traffic spikes seen out on the consumer network, though. Further investigation would show the potential threat was just a false positive.
“It was Fortnite doing a very large update, and of course that had to happen an hour before our opening ceremony, didn’t it,” Wakely said.
“Everybody had come home, done their homework, had their evening meal, and gone to turn on PlayStation.”
How to avoid emergency-patching 133 switches
The 2018 Commonwealth Games was the first to have a single company provide a unified network that handled everything from video streaming for TV broadcasters to recording the results.
“When a swimmer reached out and touched the wall winning the race, when an athlete crossed the line, those results had to get from the Gold Coast across to Perth [in Western Australia, where the data centre was located] and back again within milliseconds,” Wakely said.
One of the key tools for successfully delivering that network was a detailed map of the network, she said, from both operational and cybersecurity perspectives.
“Make sure that you visualise, put on a page, get those diagrams going,” Wakely said.
“It really aids in communication to everybody in your team and into your management layers… It also enabled us to very quickly highlight where changes were happening from a cybersecurity perspective, and what the impacts of those changes were.”
Just as the Games network had gone live, Cisco issued two critical vulnerabilities that were rated 9.8 out of 10.
We’re all taught to patch such critical vulnerabilities as soon as possible with an emergency change, but Optus faced a dilemma.
“This network, we’ve just got it going, the events are running. Do we patch and risk disrupting the network availability? Do we not patch and risk being exploited by this vulnerability? What can we do to help make this business decision?” Wakely said.
“[The] security blueprint on a page enabled us as a multi-vendor team to come together as one team”, allowing them to “very quickly highlight where changes were happening from a cybersecurity perspective”, and their potential impact.
“Our management team by now were very familiar with this diagram and what the changes meant. And we were very quickly able to articulate that we would only apply the critical patch to three routers, and we would not apply it to 133 switches,” Wakely said.
“Now that’s a big call to make for a 9.8 out of 10 critical vulnerability. But we were able to evaluate that from a risk perspective at a business level due to having clear communication diagrams like the one in front of you.”
Wakely also stressed the importance of having the right people. Not only did they have technical security analysts on-site, but they also had executive-level cyber leadership physically in the room.
“That person was also able to engage with government teams, engage with senior people in the Commonwealth Games authority, engage with our executive,” she said.
“So don’t think cybersecurity is a technical analyst any longer. It is an executive type of resource.”
Disclosure: Stilgherrian travelled to Chiang Mai, Thailand, as a guest of APNIC.