by Nirmal John
‘Operating in 30 countries, too poor to invest in security?’
‘This is what happens when you pay your engineers 10-15 lakh while American companies pay them crores.’
Some saw the lighter side:
‘Half of Gurgaon will starve to death today,’ said one user on Reddit.
Zomato updated the blog as soon as fresh information emerged to allay fears and quell speculation that the breach might be worse than Zomato was describing it to be.
[Update] 60% of our users use third party auth services (i.e. Google and Facebook) for logging in to Zomato. We don’t have any passwords for these accounts. Therefore, these users are at zero risk, both within Zomato, as well as on Google and Facebook (and any other services where the same Google/Facebook ID is being used to log in). For all our other users, as a safety measure, we strongly advise changing your passwords on other services where you might have used the same password as for Zomato. We are also sending emails to such users prompting them to do the same as we speak.
This was not the first time that flaws in the security framework at Zomato had been exposed. In 2015, a Bengaluru-based ethical hacker—Anand Prakash—had highlighted a vulnerability at Zomato.9 ‘While creating an account, a user can store his phone number, address, date of birth, link Instagram account etc. In one of the API (Application program interface) calls, they were reflecting the user data based on the ‘browser_id’ parameter in the API request. Interestingly, changing the ‘browser_id’ sequentially resulted in leakage of data of other Zomato users. The data leaked also had the Instagram access token, which could be used to see private photos on Instagram of the respective Zomato users,’ he wrote on his blog after his discovery of the issue. This meant that by sequentially changing the number associated with each user’s access request, data of other users could be accessed.
Prakash, a young engineer who has now started his own cybersecurity firm, was a stickler for responsible disclosure and had alerted Zomato and given them the opportunity to fix this issue back then. Patidar too, had responded quickly to fix the issue. Things were not going to be as easy this time with nclay.
Eighteen arduous, nail-biting hours went by. Patidar, Sinha, Goyal and the rest of the team got little sleep on the night of 18 May and were in the office early on the morning of 19 May 2017. They were fast losing hope that the hacker would reply to their message on Hansa Market. And then, at around 2 p.m., the hacker replied. It was just three short words.
‘What is it?’
Judging by the curt, almost dismissive tone, it didn’t appear to anyone in the war room at Zomato that it was going to be easy to engage with nclay. But the message, whatever its tone, was a positive development and they had to try and capitalize on whatever light appeared at the end of the tunnel.
‘I’m from Zomato and I want to talk to you about the data you have put up on Hansa Market,’ Patidar replied. He was careful not to reveal his designation, but instead looked to engage nclay to ease him or her into a conversation.
Hansa Market’s messaging function is more akin to the personal messaging option in Internet forums and was not a real-time chat. This meant that the time taken between messages felt excruciatingly long to Patidar. The messages slowly went to and fro.
The hacker knew that Zomato was based far away from where he was and there was no chance that the tentacles of India’s law enforcement could reach him. ‘You are in India. What will you be able to do? It is not like you are in the US,’ he would say, at one point in the conversation.
He also did not shy away from showing his sense of superiority, bordering on arrogance, that black hat hackers tend to display when talking about their conquests. He knew that he held all the cards in this game, and Zomato had to go looking for an opening.
At times in the conversation, Patidar could feel his anger rising, but had to keep it under check. He couldn’t afford to antagonize nclay. The important thing to do was to calm nclay down and understand the hacker’s motivation behind breaching Zomato. It was crucial that this was done at a pace and in a tone that didn’t spook the hacker.
Zomato tried to appeal to the hacker’s softer side, telling him that a leak of this sort would be bad for the company and for the many people whose jobs and lives were intrinsically tied to it. They requested the hacker not to let copies of the cache of data spread until they were able to sort the problem out.
Shrey Sinha’s counter had by then ticked once, making many in One Horizon Centre nervous. While nclay didn’t remove the listing despite requests, the hacker claimed that the transaction that had showed up on Sinha’s counter was for the Edomodo data.
There was no way to confirm that this was indeed the case, and Zomato had to take the hacker’s word for it. Despite Zomato’s requests, he did not send any screenshots to prove his statements. He may, Patidar believes, have been fearful of inadvertently disclosing his location.
Messages went to and fro between Zomato and nclay till around 8 p.m. The conversation lasted nearly six hours, including the couple of hours in between when nclay left it. Eventually, Zomato asked nclay the one question they wanted an answer to.
‘How do we find a solution to this, nclay?’
During their conversation, Patidar says, he could sense that the hacker was angry about something. Nclay’s reply validated this hunch.
‘You have a very shabby security program on HackerOne.’
This was a moment of truth for Zomato—the turning point they had been waiting for. Nclay was referring to a bug bounty programme that Zomato ran on the global ethical hacking platform HackerOne.
‘We are listening. What is it that you have found is wrong about the programme?’
‘You don’t treat people who show your vulnerabilities very well.’
For the uninitiated, bug bounty programmes are a form of crowdsourced security testing where companies open themselves up and invite hackers to find vulnerabilities. Those who find bugs in the products and services get paid—bounty—by the grateful companies. Such programmes are popular globally among white hat hackers and are a rather smart way to channelize the energy of the world’s hacking community. It allows for a legal way for hackers to find bugs and report them, and in return earn rewards, while for companies it is an effective way to do penetration testing.
BugCrowd and HackerOne are prominent names among the platforms engaging with white hat hackers for bug bounty programmes. These sites act as intermediaries between companies who want to shore up and secure their digital assets and hackers who help them by finding vulnerabilities which could then be fixed. They help launch and manage bounty programmes and validate vulnerabilities that hackers report; they then take care of the communication and transaction with the corporation on behalf of the hacker.
Zomato had run a bug bounty programme a while back on HackerOne. Hackers would send in reports of vulnerabilities and the tech team at Zomato would fix them immediately or add them to the development cycle. The catch was that there was no monetary reward for the hacker at the end of all this. All that the programmer would get was Zomato swag—think T-shirts, coffee mugs and such.
This has been standard practice for many companies, but according to many hackers, it constitutes an unequal system of rewards. Nclay clearly belonged to the camp that felt the job white hat hackers were doing was not sufficiently rewarded. For nclay, the rewards were not commensurate with the value of the vulnerabilities that hackers dredged up.
With nclay expressing his displeasure about the rewards, Zomato had something to work on, something to negotiate with. The interesting part was that nclay was not doing all this for money. Zomato now had an inkling of the motive behind the breach, and that meant they could attempt a deal with nclay.
Zomato apologized to nclay.
‘We told him that we are going to fix this and that we would take a relook at our bounty programme,’ says Patidar. He told nclay that Zomato would mend their ways and treat hackers with more respect, rewarding them w
ith something more substantial than mere swag.
Then, without nclay asking, Patidar told him one more thing. ‘Because you have definitely found a critical vulnerability, we would like to offer you a reward as well.’
The hacker told Zomato the amount of money he wanted in bitcoins. Patidar doesn’t want to disclose the exact amount of money they paid to nclay, but says that it was important for Zomato to take control of the situation. The priority was to convince nclay to take down the listing of the data cache.
A deal was finally in place. Zomato would pay the money to nclay and would also design a bug bounty programme on HackerOne that was significantly better than their earlier one. In return, nclay would take down the listing and destroy the file, but not before sharing a copy of the file with Zomato.
Do they know for sure that the hacker destroyed the file? The reality is, Zomato doesn’t, and there is no way to really know if nclay kept his end of the bargain. They do know for sure that the listing was taken down from Hansa Market.
‘It is trust,’ Patidar says. He believes nclay will hold up his end of the bargain. What was important at the time of the breach and the subsequent listing of the data was to stop the file from propagating easily, and Zomato achieved that by making the deal with the hacker. That made it possible for them to gain time to get users to change their passwords and to enhance their security systems. Once people changed their passwords, the data that was compromised would not be of much use. These steps, they say, should eventually render the information that was in the data cache far less useful to anyone who may acquire it.
The conversation with the hacker also revealed the exact manner in which the information was stolen. Zomato would later update its blog with this information, in case it turned out to be of help to other companies who found themselves in a similar situation. A breach often flows from another, as this one incidentally had, and it was important to share the learning. Zomato posted this:10
It all started in November 2015, when 000webhost’s user database was leaked online (with plain text passwords). One of our developers had his personal hosting account with the service. As a result of 000webhost’s user account data breach, his email address and password also became available publicly.
Unfortunately, the developer was using the same email and password combination on Github. Back then, when 000webhost passwords leaked, we were not using two-factor authentication on Github (we have been using two-factor authentication on Github for the last few months). With the login credentials of the developer, the hacker was able to use the developer’s password to get into his Github account and review one of our code repositories to which the developer had access (this happened some time last year, but for some reason the hacker only exploited the code very recently).
Getting access to a part of the code didn’t give the hacker direct access to the database. Our systems are only accessible for a specific set of IP addresses (a number given to identify systems accessing a network). But the hacker was able to scan through the code (Zomato’s programming code), and he ended up exploiting a vulnerability in the code to access the database via remote code execution. (Remote code execution allows attackers to execute codes from a remote server.)
What that means in layman’s language is that when the webhosting platform 000webhost was hacked, among the passwords that were compromised included one belonging to a Zomato engineer. This engineer had used the same password in Github, a platform used by developers to collaborate with others while writing code. Nclay accessed the Zomato engineer’s Github account using the compromised password and used that to find repositories of the Zomato code. Nclay then found vulnerabilities in this code to access parts of the Zomato user database, which lead to the hack. Think of it as the Zomato engineer losing the key to his house, a thief finding that the key also opens the engineer’s cupboard, where he finds the key to his office.
Zomato still doesn’t know the real name of nclay, although it speculates that he may not have been a native speaker of English, judging by the quality of the language he was communicating in. The absence of fear of recrimination and the time of day he had come online suggested that the hacker may have been from eastern Europe. They also believe the hacker was young, probably still in high school or college. Of course, they did not have any means of confirming all this, which was only speculation.
True to its word given to nclay, Zomato is readying a programme, to be run on HackerOne, that will seek to handsomely reward those who can find vulnerabilities. Depending on the severity of the issue, the hackers will be given money, just as nclay wanted. It will never be swag any more.
If someone robs your home, your first reaction will almost always be to call the cops. For companies like Zomato in India, that never seems to be the go-to option when their data is stolen. Patidar believes that they did the right thing by tackling the incident on their own. He points out that in this case time was of the essence: ‘We know a lot more than some external person about our own systems. The time to action would have been much longer had we called the cops.’ While Zomato did file an incident report with other government authorities, that didn’t amount to much.
On the surface of it, the United States Department of Defense, Pornhub, Facebook and Ola have very little in common. Bug bounty programmes make for the strand that connects them all. They are among the increasing number of entities running these programmes, which should be a hugely important part of the cybersecurity arsenal for any company.
Many bounty programmes have, over the years, attracted the sort of criticism levelled at Zomato by nclay as being exploitative. Far too often, companies pay peanuts for what they get, or worse still, dole out random corporate goodies (T-shirts and mugs) to the mostly young hackers who find bugs in their systems. These hackers often get very little appreciation for their talents and often have no idea of the value of what they have found.
As Sahir Hidayatullah, chief executive of Smokescreen Technologies, says, ‘It’s the dirty secret—bounty programmes in India and many other places are often geared to just make the bounty hunter feel like a rock star. Some of these guys literally report bugs and beg for swag. If they figured out they could charge in increasing multiples of thousands of dollars for their work, they would understand what an asymmetrical, exploitative market it is.’
Some security companies also believe that bug bounty programmes are a potentially dangerous way out for companies who want to skimp on investing money in more comprehensive penetration testing.
There are also the legal issues that plague hackers who have often gone about finding vulnerabilities in companies that don’t have an official bug bounty programme. They often end up inviting threats of lawsuits.
This is especially true in markets like India, where there is a massive number of hackers looking for a way to make some easy dough but very few companies that run official bounty programmes. Companies which get mails from ‘proactive’ hackers pointing out vulnerabilities often get spooked and threaten them with legal action for ‘hacking’.
Despite all these issues, bug bounty programmes are an effective way to tap into the energy of the hacker community, but Indian companies have largely been shy of going down this road. That is even more puzzling because Indian engineers form the spine of the hacker community that participates in these programmes globally.
According to Facebook, India topped the charts in terms of the number of researchers who contributed to the company’s efforts in making the social network secure and the money that was paid to them. Facebook received more than 9,000 reports in the first half of 2016 and paid a total of $6,11,741 to 149 researchers. The total money paid out over the five years since they floated the programme stood at more than $5 million, paid to more than 900 researchers.11 Indians were at the top of this chain.
The amount these hackers earn varies, as Adam Ruddermann, technical programme manager on the Facebook bug bounty team says in a blog. ‘First, we look at the potential impact of a bug, w
hat could possibly go wrong, and who would be affected. We also consider the difficulty of exploiting the vulnerability and what kind of resources or technical skills a successful attack would require. After evaluating the considerations above, our team determines a base payout for each eligible report.’12
BugCrowd, a platform like HackerOne that helps companies run their bounty programmes, said in 201613 that they receive as much as 35 per cent of their submissions for vulnerabilities from India. It helps that India has one of the largest pools of coders globally. It also helps that, thanks to the exchange rates being what they are, a $500 reward is much more enticing in India than in most of the West.
India clearly has one of the most active bounty-hunting hacker communities in the world, but in a rather dichotomous twist, only a handful of Indian companies runs bug bounty programmes. Barring exceptions like Paytm and Ola, there are very few who leverage the energy of this community. Instead, the country has become a fertile ground for many companies from the West, particularly in Silicon Valley, who used their expertise to find flaws and vulnerabilities in their products.
The raison d’être for NCR-based BugsBounty is to convince more Indian companies to design and execute such programmes. ‘There isn’t a formal workflow internally for such issues in most organizations. Gaining an understanding of both sides, companies as well as hackers, meant that we could bring some semblance of harmony between them. Our mission is to make the world a secure place by uniting ethical hackers globally,’ says Ankush Johar, a London-based venture capitalist and a director at BugsBounty.
Johar, who a few years back had a breach of cybersecurity in his office and had used ethical hackers to solve the issue, has confidence in this model because ‘they [the hackers] find things that the best security staff can’t find, and I’ve seen this first-hand’. Johar, who is also director of the VC firm Lloyds Ventures, says that he has committed to put in the money required to grow the company, which earns its dough either when hackers find a vulnerability and are paid a bounty—a fifth of which the company keeps as commission—or through the fees it earns for running bounty programmes for others.