As Spring cracked the Moscow frosts and March rains doused the streets, a computer in an innocuous server farm somewhere in the heart of the city winked to life. It was 2007, a year when many people became truly invested in online life. Twitter was a year old and the most popular smartphone was the Blackberry Curve – a pure email machine. It was a year ripe with promise for cyber-everything. And a group of hackers, unnamed to this day, wanted to grab their piece.
The server first sent a blast of emails containing a link to a piece of software that many around the world wanted to download. Once they grabbed the Trojan Horse, the infected program took over computer after computer, creating something security experts call a botnet – a collection of infected machines controlled by a central command and control unit (CnC).
This Moscow server, hidden behind IP address 72.232.49.214, began receiving incoming messages from a number of computers around the world. A whole swath of California lit up as communicating programs came online. Then New York, then London, Berlin, and Minsk. Computer after computer began chirping out requests to the mothership. The infection spread thanks to a mixture of gullibility and trust seen time and time again in the annals of computer security. The Grum botnet was born.
It took a few days, but ultimately 120,000 machines spoke to the Command and Control server in Moscow and the server messaged back. Some machines dropped out of the network thanks to vigilant users but others quickly took their place. It was like a mold grown over the globe, spores spreading through various networks.
Grum sent over a quarter of the world's spam and was one of the most ingenious botnets ever created. But, with savvy, a lot of luck, and cooperative ISPs, the Grum botnet dried up and died last month.
Here's the whole story.
That March, Internet users began receiving emails from admin@microsoft.com with the subject line "Internet Explorer 7 Downloads." A click later and they were at a bright splash page purporting to offer a fresh download of the latest Microsoft web browser, Internet Explorer 7.
The download was a dud. Clicking on the link brought nothing but a small file called ie7.0.exe. Running it revealed nothing – just a little gibbering in the hard drive and then silence. Users could click all they want – IE 7.0 wouldn't appear.
To many, this was just another bum link on the Internet. But inside their computers, something was happening. The skittering meant something had been installed on the hard drive, within a temporary Windows directory. The file was winlogin.exe, an innocuous enough name that might have been familiar to slightly savvy PC users. In less than a second, however, the program burrowed its way into the computer's registry – a database of information about the machine – and added itself to the list of programs run when the computer begins to boot.
Eventually, the program was identified as the Grum-A aka Tedroo and Reddyb. It was probably written somewhere in Russia and carried a payload called a "rootkit" – a program that gave an outside user administrator access to the hard drive. Grum listened for a set of commands sent by the CnC servers. The simple commands came through a standard HTTP port and could "update" itself automatically. Initial reports saw the worm as fairly harmless. One security firm described it this way:
The primary locus of infection was a program that ran every time Windows booted. By adding code to a kernel library called ntdll.dll, the virus was able to hide and run itself automatically every time the user started his or her computer. Deleting ntdll would be catastrophic and because it was a high-level, privileged file it was nearly impossible to pull it off.
More importantly, however, is the way Grum worked internally. Each copy of the virus spoke with a set of CnC nodes and the CnC system could segregate infected computers into different secondary groups. However, the program had a fatal flaw.
The virus contained a set of hard-coded master IP addresses. Instead of sending commands to, say, grummaster.com, the program sent messages to a set of two or more CnC IP addresses. Like a biological virus primed to thrive in a certain type of medium, the Grum virus was susceptible to defeat if someone knocked out each of those CnC IP addresses. The commands weren't human readable – there was no "SEND SPAM" command – but it was fairly easy to see what was going on with a bit of effort.
Grum's creator's foresaw this problem and placed their CnC servers in countries that had, in many cases, lax or nonexistent, cybercrime laws. The initial IP addresses were in Russia but others popped up in Panama, the Netherlands, and the Ukraine. To be clear, there was nothing inherently bad about these ISPs. They weren't about to practice Internet censorship and given the distributed nature of the CnC system, the Grum botnet kept a low profile even as it sent its commands out to various parts of the network.
As the botnet spread, its creators sent out periodic updates that fixed bugs and identified new CnC servers. If a CnC server went down, the coders would update a new binary with the new IPs. These binaries would spread slowly because not every infected machine would check back in with the mothership every day. Like Microsoft or Apple pushing out OS patches, the Grum makers were upgrading their virus regularly, adding new features and fixing problems.
The Grum botnet was one of the most robust and powerful in the world. Aside from its single, glaring flaw, the system worked without peer and slowly began spamming the world, mostly with poorly worded pharmaceutical emails. Every time someone pulled the plug on a CnC server, a new one popped up somewhere else.
"Look at it from a criminal's perspective, you have that much of a resource," said Carel van Straten, a security researcher at Spamhaus. "You're going to try to keep it online and try to keep it going."
And that's what the Grum creators did – for half a decade.
Spamming isn't very lucrative. Brian Krebs, a security reporter, notes that while businesses spend $40 billion per year for anti-Spam technology, the estimated revenues of most major spammers hover at around $150 million in a good year. In the bell curve of spammers, however, most end up on the side of making very little.
In an excellent series, Krebs was about to track down the creator of Grum and its leader, a hacker name Ger@ or Gera/GeRa. By tracing money back to the source, Krebs was able to assess who, specifically, was making the most money from spamming. Gera's affiliate account, gleaned from a list of payments for the pharmaceutical sales program SpamIt and Glavmed, showed that his efforts brought in $6 million in 2010.
This data suggested that Gera was a very prolific spammer. Further leaked documents showed repeated conversations between SpamIt leader Dmitry Stupin and Gera. Stupin called Gera out for his practices, saying that he was beyond compare when it came to "trouble with hosting providers."
Krebs' big find, however, was a name:
This is as close as anyone has gotten to Grum leadership. "No one has been convicted as of yet. Nor officially assigned to be Grum's botnet-herder," said Bogdan Vovchenko with Group-IB, computer security response team in Russia. However, it was clear that whoever was behind it was very wary.
My own attempts to contact Grum leaders – including an account associated with the 26-year-old Kostogryz, failed. The Grum team really didn't want to be found.
What this reticence meant, in short, was that Grum was probably run by a small team led by Gera and that, even given its reach and relative lucrativeness, the entire operation was streamlined. While it could move fast, this could also mean the organization wouldn't be able to react to a massive shutdown. Other botnets had ways to dynamically reassign CnC servers very quickly. Grum did not. Gera was also not particularly beloved by ISPs or even the affiliates that used Grum's botnet to send pharmaceutical Spam. It was, in other words, a nearly perfect target for some dedicated anti-spam researchers.
In 1998, a former Pink Floyd production manager and songwriter, Steve Linford, realized his computer-consulting clients had a huge problem: spam. Over the course of about a year, Linford began collecting the source of most of the spam circulating on the Internet and created a list called ROKSO – Register of Known Spam Operations. This living list, updated regularly with new and reformed spammers, has long been the first line of defense for most spam fighters. For years, Linford lived on a houseboat in the Thames but now the organization has grown, with headquarters in Switzerland and the UK.
Linford's organization, Spamhaus, went on to become an anti-spam powerhouse, garnering respect and fear from ISPs around the world. ROKSO itself blocked the 100 known spam operators responsible for 80% of spam and systems that used its data were able to reduce spam considerably. However, some still got through, and the worst of the spam came from the relatively anonymous botnets.
One researcher, Carel van Straten, worked from Amsterdam and watched botnets rise and fall. A cheerful senior spam researcher, he was very well-versed on the ins and outs of rogue server hosting.
Spamhaus had a big stick with which it could police the Internet. All it needed was a target.
The soft spoken Senior Staff Scientist for FireEye in San Francisco, Atif Mushtaq, had that target. Mushtaq studied computer science at the University of Management And Technology Lahore and worked as a network architect for Palmchip in 2008. He moved from Pakistan to the Bay Area where he began writing a series of concise, sometimes breathless, posts about his efforts to find and shut down popular botnets.
None of these security experts enjoyed the limelight. Anti-spam researchers have been harassed, threatened, and their websites have been shut down by angry spammers. Spamhaus, for example, rarely publishes photographs of its researchers in order to protect their privacy online.
Meanwhile, in Moscow, a computer security rapid response team was also following the Grum virus. In 2011 the botnet remained stable and strong but in the spring and early summer of 2012, researchers noticed that the number of CnC servers was falling slightly and that multiple servers were in only three countries – the Netherlands, Russia, and Panama. Perhaps all it would take was a few good taps to shut it down?
"Grum was the world's number one spam botnet back in January 2012," said Mushtaq." Then in the last six months, there were less command and control servers and it was sending less spam. I didn't know why it was happening but I told myself 'Okay, this is the right time to do it.'"
Mushtaq began by assessing the list of CnC servers for holes. Immediately, a few things stuck out.
190.123.46.91 Panamaserver
195.190.13.150 SteepHost DC-UA
195.190.13.182 SteepHost DC-UA
195.190.13.206 SteepHost DC-UA
195.190.13.222 SteepHost DC-UA
195.190.13.78 SteepHost DC-UA
91.207.4.215 SteepHost DC-UA
91.207.6.134 SteepHost DC-UA
91.207.6.234 SteepHost DC-UA
91.207.6.35 SteepHost DC-UA
91.207.7.6 SteepHost DC-UA
91.207.7.98 SteepHost DC-UA
91.207.8.102 SteepHost DC-UA
91.207.9.252 SteepHost DC-UA
91.239.24.251 GazInvestProekt ltd.
94.102.51.226 ECATEL LTD
94.102.51.227 ECATEL LTD
91.236.120.6 PROEKTPROFDEVELOPMENT-NET
Although it looked like a large list, most of them were in the same location and some even in the same building. SteepHost DC-UA, for example, was based in Kharkiv, Ukraine in a building by the main train line. ECATEL was a Dutch ISP and Panamaserver was, as expected, in Panama. The rest were in Russia, including GazInvestProekt, a small ISP in Pskov.
None of these ISPs were "rogue," per se. It was generally bad business for an ISP to shut down a server or IP address based on complaints by security firms – he-said-she-said back and forths were rarely constructive. However, they did respond quickly whenever someone reported true abuse.
"ECATEL does have a very long history of hosting shady things," said van Straten. Seeing Mushtaq's detailed posts, van Straten reached out to FireEye to see if they could help take down some of the servers. As luck would have it, Mushtaq was ready to move on his first decisive attack.
On July 9, 2012, Mushtaq began musing on a Grum takedown.
"For a successful takedown attempt, we need to clearly identify Grum's command and control coordinates. We also need to find out what would happen if the master CnC servers become unavailable during a takedown attempt. If Grum has a fallback mechanism, then we need to disrupt the secondary CnC structure as well and so on. The most important of all is the geo location of active command and control servers. Historically, it has been relatively easy to shutdown CnC servers located inside of the U.S. as compared to countries like Ukraine, Russia, and China," he wrote on his blog. "Keeping all of this information in mind, I am getting mixed feelings. I can see a few factors that can go in favor of the Grum botnet. At the same time, Grum has some obvious architecture-level weaknesses."
However, as he examined the servers, he noticed Spam levels were dropping precipitously – down 30% over the last year at least – and the thought the time might be ripe to pull the plug.
"And then I thought about all these reasons — those servers are taking less and less spam traffic – I thought that if I tried to take it down now I'd have to do less work," he said.
Mushtaq reached out to his network of researchers and began aiming at the servers in the Netherlands. These seemed the most ripe to his entreaties as he had no contacts in Panama or in Russia who could help. In Amsterdam, he had Van Straten. On July 16, 2012, Atif wrote on his blog "Dutch authorities have pulled the plug on two of the CnC servers pointing to IP addresses 94.102.51.226 and 94.102.51.227.1 Thanks to the Dutch authorities for swift action." Part of the botnet was down.
However, that was just the beginning. With the Dutch servers down, the botnet creators had a few days in which to bring up new servers and send out updates to all of the infected computers. At that point, time was against him. He began to reach out the other providers. One developer, Isidro Gonzalez, told Atif that he could try to help shut down the botnet in Panama.
"I'm a software developer from Panamá City, Panamá and I've been following your recent saga with Grum. I thought about spammers within our country but I had no idea our country was part of a huge botnet like this. So, I wonder, how can I help?" he wrote.
Around the world, sysadmins were watching the Grum takedown with interest. In Moscow, a response team from Group ID was at the ready to begin taking down the Russian and Ukrainian servers. Van Straten volunteered to assist in contacting various authorities.
"Atif could not get those providers to respond to him and well, we have been around for 12 years or something now. We have a lot of good relationships and a big hammer. So we contacted the ISPs that were still hosting the last servers and we managed to get all of them online in a reasonable timeframe because the problem basically with botnet like this is that if you keep one server online, it allows the operators to push out a new binary that has an updated list of command and control servers," said van Straten.
"Here at FireEye labs, we are monitoring Grum's activities on a 24/7 basis. Any attempt to recover this botnet will be noticed. I don't know if the security community will eventually be able to take down the rest of the Grum botnet, but we are trying and trying very hard. We did not give up after the first failed attempt and will continue to contact the Russian and Panamanian authorities through different channels. So this is an operation still in progress. I will keep you informed with the latest updates," wrote Atif.
Van Straten began working more intensely with Atif and the pair was able to convince SteepHost in the Ukraine to shut down their servers. The worked closely with a response team in Russian, Group ID, to hit the servers quickly and quietly.
"At that point, I think there were four remaining, one in Panama, one in Russia, and two in the Ukraine," said Van Straten. One of them, Ecatel, was very interesting.
They took down most of the servers – the Netherlands servers were gone and Panama was about to wink out.
"Ecatel does have a very long history of hosting shady things," said Van Straten.
However, Spamhaus' "big hammer" worked. The Ukraine servers were toast. And then one more came back up again.
"It got a little bit fishy. One of the IPs that used to be a CnC of the Grum botnet was taken offline, but it came back, and the ISP said, 'Yeah, we have a security issue. Some servers have broken into,'" said Van Straten.
"Well, I mean, what are the chances that that same IP would become a command and control node again? You can never tell. The ISP says, 'Yeah, we formatted the machine.' Okay. Well, they're in the Ukraine, which is not like we can go over and check."
As the Grum "bot-herders" saw their servers die one after the other, they continued to try to bring up new servers.
Mushtaq wrote:
5 years, 3 months, and 17 days after the first emails began spewing out of the Grum botnet, the last server was dead.
The Internet got just a bit quieter.
Mushtaq was stunned. The bot was dead. 120,000 Grum IP addresses dried up to about 21,505. These zombies, unable to communicate with their CnC nodes, would eventually disappear, unable to send out any more spam. The only way to restart Grum would be to reassign the dead IP addresses, and Spamhaus would make sure all of those were on a watch-list.
"In a certain sense, we were kind of lucky with this, that all the ISPs that involved here, that we have an existing relationship so that when we contact them, we don't have to explain who we are and why this is bad and what it is and et cetera, et cetera," said Van Straten. Spamhaus allowed them to attack with a purpose and not needle ISPs with random requests. But Mushtaq wasn't stopping there.
On the 18th, Mushtaq wrote: "There are no longer any safe havens. Most of the spam botnets that used to keep their CnCs in the USA and Europe have moved to countries like Panama, Russia, and Ukraine thinking that no one can touch them in these comfort zones. We have proven them wrong this time. Keep on dreaming of a junk-free inbox."
"We are definitely very happy," he told me. He reminded me, however, that Grum is only the first of many. "When we are monitoring spam botnet in real time it's a good feeling to see the level of spam going down."
"Did you guys go out to dinner or anything?" I asked.
"Unfortunately, all of the Spamhaus guys are from different parts of the world so it was not like that. There was no get-together, I would say. But yeah, I went out with my family and we had a good time."
"I was really happy," he said.
No hay comentarios:
Publicar un comentario