Keep your store and customers safe

One morning, a client sent us a frantic Slack message asking a developer and their project manager to attend a call because, “the Secret Service is here and they want to talk about how our site is siphoning off credit card information.”

This is definitely not the way you want your day to start. We certainly did not. It was a first for us and for our client. We would soon find out the issue at hand was much bigger than just one client and just one agency.

The Symptoms

As an ecommerce development agency, one of our testing responsibilities is to ensure the web store processes credit card transactions properly before pushing code to production. For this purpose, we have a test credit card that we use solely for test transactions on client websites. It should go without saying, but we immediately submit every transaction for a refund so it never carries a balance.

Weeks earlier, the card statement showed a couple of unrefunded transactions. This is odd considering we only use it on client websites, so we knew something was wrong immediately. A client also reported a card had been stolen from their site. We performed security scans of websites where we had used the card in the prior weeks, but the scans all came up blank.

We reset passwords, had Wells Fargo refund the purchases, and requested a new card. There were no more purchases made despite using the new card on the same sites.

Strange.

The Hack

Little did we know, hackers had gained access to the admin panels of multiple clients. The hacks both involved the creation of new admin users that had the identical email addresses across multiple sites. In all cases, the same malware was deployed via Magento’s content management system.

How did they gain access? We started narrowing down the possibilities:

• A brute force attack on the admin panels. The links to the admin panels are not common knowledge and contain random strings, so this seemed unlikely. Possible, but highly unlikely.
• Operational security of the merchants. Most organizations are guilty of poor security practices: sharing passwords over email or messaging apps (Teams, Slack, etc.), not checking for suspicious admin accounts, reusing passwords on multiple sites, weak passwords, and so forth. However, because these clients had so many things in common besides potential security practices, this seemed unlikely.
• Infection of Razoyo’s dev ops pipeline. Could the hackers have gained access to our repositories, which led to deploying malicious code? We investigated and eliminated this possibility right away. There were no anomalies in our deployment pipeline and the malware they implemented did not require it, ruling this possibility out.
• The hackers gained server control. This would have been the heaviest lift for hackers due to a variety of reasons: the hack didn’t require it, our production servers do not have the necessary permission to update code files and we have tight firewall rules and more.
• A core Magento vulnerability. Both clients had delays in their patches/regular updates from Magento/Adobe Commerce. However, neither of the sites seemed far enough behind in updates for this to be the likely culprit. None of the missing patches seemed to provide an adequate attack vector. Nonetheless, this seemed to be the most likely scenario so far.
• A vulnerable extension. Even if your Magento core code is up to date on security patches, your extensions might not be since there is often a delay in extension developer updates. Both merchants had extensions from developers who are known to have somewhat slow update rollouts. This option started to seem more likely.
• Operational security of a 3rd party vendor. The clients shared the same marketing automation service which routinely created admin access during the installation and configuration of their systems. If that vendor had suffered a breach it would have been possible to open the door to hackers unknowingly.

We also noted that the fake admin users were both registered with the same email address which bore a third-party vendors’ domain name. We concluded this was probably the attack vector. The access was probably gained by compromising the 3rd party account manager’s email. Since this account manager routinely requested access to the client’s Magento/Adobe Commerce admin panel, it is likely they were sent via email.

The Malware

Of all those possibilities, the Secret Service identified malware inserted into a content block in Magento’s content management system. The level of sophistication of the code was quite astonishing and would have required top-notch developers. The FBI tends to get involved in far-reaching attacks. We suspect this to be part of a much larger hack, with an even bigger agenda, across many sites that have no association with Razoyo.

First of all, the code was highly obfuscated: it avoided using a script tag (which makes code easy to scan for) by leveraging a vulnerability that allows a developer to insert JavaScript from a remote source when an image fails to load. From a UI perspective, it makes sense to have this capability so that if an image is missing or offline it can be replaced with some other type of code to keep the page from jumping around or breaking the HTML dom structure, among other things.

Secondly, the executable part of the code was entirely encrypted and required us to reverse-engineer it to determine what it was doing. A visual inspection would have not indicated this was malicious code at all.

Finally, when executed, it opens a web socket to a remote server that could load whatever it wanted to execute into the browser. In most attacks, it is easy to understand the purpose of the hack and what the inserted malicious code is doing.

However, because this one loads the code via a web socket unique to each visitor, we really don’t know what was executing for any given visit and it could change. The remote hacker’s system had complete control over the code insertion on page load.

Honestly, it is a little scary to think about the potential capabilities of this hack, especially given the involvement of two government agencies. It was likely a wide-spread hack involving many websites – the code could be inserted into WordPress, WooCommerce, and even Shopify for that matter. Given the government’s interest it may have involved state actors, organized crime, or both. We aren’t even entirely sure the criminals are US-based.

The Remedy

In the end, we were unable to determine conclusively how they gained access to the admin panel and how they were able to set up the same suspicious admin user on either site. It is possible the FBI or Secret Service know.

However, we were able to implement several measures to make that even harder moving forward:

1. We had the client review all admin users and remove any unknown entities. Merchants should do this periodically, anyway.
2. We had them implement the two-factor-authentication (2FA) option provided by Magento for the admin panel or other 2FA solution. Again, while 2FA is not fully secure, especially when a code is sent via SMS, it does create a substantial hurdle for hackers. In my opinion, this was the most important step and 2FA should be standard practice for ecommerce admin access.

1. We made sure the investigators from the government had the information they requested in their investigation (under guidance from the client, of course).

One merchant was already transitioning from Magento to Shopify, which eliminated some of the security risks they were facing. This seemed to appease the Secret Service despite the fact that this hack would have likely worked on that platform as well.

The Fallout

The consequences of being hacked can be very serious for merchants, including losing the ability to process credit cards, loss of trust, and even legal action. We work quickly with merchants to address the concerns of authorities, banks, and customers. None of our clients have ever lost the ability to transact due to a breach or hack.

You’re going to have a bad day

Besides the feeling of violation that comes with a hack, you will probably have to spend time and money with a lawyer to help with your communication plan. You will have to send an embarrassing email to your customers if their data may have been compromised.

You will also probably have to deal with banks or credit card brands and put together a plan that convinces them your business not only takes security seriously but is now secure. In this case, the clients had some very uncomfortable conversations with federal agents.

Finally, if it is not your company, you may end up having a rough conversation with your employer. I’ve never seen anyone who takes a hack seriously fired, even if they were at fault, but I have seen people quit over one. This was not an issue in this case.

Road to recovery can be long after being hacked

Depending on the type of issue, how long the malware has been operating, and other factors, you may need to jump through multiple hoops. You’ll need to prove you can pass security scans. You may need to supply your remediation steps and report progress on them to authorities or banks.

It’s important to note that all of the merchants involved in this hack recovered well and faced no severe consequences. It was a storm, but it passed.

You need to learn from this

Even if your security practices were not at fault, you should still take the opportunity to review and update them. Creating a plan to avoid future hacks can provide proof that you are doing everything you can to keep your customers’ and partners’ information safe. We’re in an escalating arms race with hackers and bad actors on the internet every day.

Keep your Magento / AdobeCommerce site safe

While there is far more you can do besides the items on this list, here are some of the take-aways from this experience.

• Keep Magento / AdobeCommerce or any software you have up to date with security patches. Most businesses are attacked on known vulnerabilities rather than zero-day exploits. This helps make you a hard target for most hackers.
• Keep your extensions up to date. If your extension providers fail to keep their code up to date with the most recent patches in the underlying programming language, you should consider using something else. The cost of a hack is high… a lot more than you save with cheap extensions.
• Use secure hosting. Limit permissions and users on production hosts to the bare minimum. Conduct access audits frequently or use a tool that continuously reports on new permissions granted and used.
• Practice good admin user account hygiene. Rotate passwords periodically and after any incident. Remove admin accounts you don’t recognize. Report unexpected changes to admin accounts to your developers and have them investigate as well.
• Train on social engineering. In the attack described in this article, it is entirely possible that this came through social engineering. In fact, given the advances in security technology, many hackers turn to social engineering to get credentials. You should train your staff on this from time-to-time to keep up to date with current attack strategies.
• Be safe online. Don’t allow yourself or anyone who works on your website to click unknown links, conduct personal business, or receive personal email on the same devices or accounts they use for managing your online store.
• Educate your organization. There are excellent resources available online for training your staff to stay safe. Make sure you have a plan to keep them up to date on the latest threats.
• Don’t give admin panel access to anyone you don’t know well. Limit their user access only to what they need. Remove or disable their accounts the moment they no longer need them. You should have someone audit the list of admin users at least weekly
• Designate a person in your organization to oversee security. While security is everyone’s responsibility, you need someone to make sure everyone knows their part and is doing it. This person should be responsible to keep other employees abreast of the latest threats and best practices.

While there is much more you can do, if you are not doing the above, you are likely vulnerable to attack. In a previous article, we discuss the top 10 security mistakes Magento 2 merchants make. This could be beneficial information to have when trying to keep your site safe.

If you think your website may be vulnerable or if you have been attacked, please contact us right away for a free consultation!