Red Teaming
Cybersecurity threats to banks and corporate networks are serious—but imagine the stakes when the targets are critical infrastructure like power plants, trains, or water systems.
Industrial control system (ICS) pentesting probes the very protocols that keep essential machinery running. Power generators, railway controllers, and even oil pipelines to name a few. A single vulnerability here could lead to devastating, real-world consequences.
With the recent announcement of Hack The Box (HTB)’s Alchemy ICS Pro Lab, Tyler Webb from Dragos sat down with HTB’s Dark to talk about ICS pentesting, operational technology (OT), and “Heavy Metal Hacking”. The 2-hour AMA session was packed with information on this emerging field of cybersecurity.
ICS pentesting uses many techniques and tools from “standard” pentesting. However, Webb described it as “trying to figure out how to pentest something that also has a physics component.”
ICS devices provide information, access, and operation functionality for heavy machinery used in power, water, and other industrial fields. The physical aspect of this type of testing makes safety a primary concern for ICS pentesters.
“Since we’re working with process controls and devices that govern physical things, safety is paramount,” Webb said. “These are plants with moving machinery, with turbines, and with all sorts of things that can cause human harm.”
As a result, these engagements tend to be collaborative. Webb noted that his team often recommends a pentest after other engagements, like incident response planning, network architecture reviews, and tabletop exercises.
After those, many ICS pentests start as “assumed breach” engagements, where testers begin with credentials or a network foothold. Webb described it as a blend between red teaming, purple teaming, and pentesting.
Hands-on skills for ICS security
Dive into the operations of a newly established brewing factory where your team has been tasked with evaluating the security of Sogard Brewing Co. The mission is to disrupt the production process and steal confidential information.
With the Alchemy Pro Lab, you’ll think like an adversary and deploy red team tactics as you learn:
ICS security fundamentals (interact and interpret protocols).
ICS network segmentation.
Active Directory enumeration in IT and OT networks.
Lateral movement, tunneling, pivoting, and privilege escalation.
Common attacking techniques against Programmable Logic Controllers (PLCs) and Human Machine Interfaces (HMIs).
ICS pentesting uses the Lockheed Martin Cyber Kill Chain as part of its foundation. Operational Technology (OT) pentesters like the Dragos team use it to identify points where an attacker could escalate their access to OT protocols and equipment.
Another big focus during ICS engagements is the SANS 5 Critical ICS Controls.
When Dragos sends Webb or other testers to investigate an ICS environment, they are typically asking about the following:
ICS-specific incident response plan: Does the organization have a plan for responding to incidents? Have they run tabletop exercises (TTX) and similar drills with key stakeholders and staff?
Defensible architecture: Was the network built with zones of demarcation between critical components? Is the Principle of Least Privilege in effect? How bad a day is a pentester going to have trying to move through this network?
ICS network visibility monitoring: What vendors and protocols are in use on the networks? Is there an asset inventory, and is it modern or an Excel spreadsheet?
Secure remote monitoring: Are teams using jump boxes and securely configured accounts? Is multi-factor authentication (MFA) enabled for remote access?
Risk-based vulnerability management: When vulnerabilities are discovered, are they addressed based on the potential impacts for the ICS assets or organization?
To the last control, he added that vulnerabilities are managed based on the organization’s context, rather than on best practices.
“Sometimes these protocols are 30 years old, and the person who created it lives in the Bahamas now, or has passed,” he explained, adding that often the solution was more about mitigating or working around something than fixing an issue with it.
Because of overlaps with network pentesting, getting general pentesting experience is a good foundation for ICS testing. Webb recommended that people learn pentesting, and “find skills there that will be applicable to industrial controls.”
He spoke of the familiar “rinse and repeat” of privilege escalation, lateral movement, and dumping more credentials to try and obtain Domain Admin access.
“It’s really important that everyone understands that Domain Admin is a tool, a level of access. It’s one more tool that, when you get there, you can look at getting the groups that get the access to get into the process control network (PCN) or the OT network.”
For hackers who want to delve even deeper into heavy machinery, Webb had further recommendations: read the materials, gain work experience, and practice communication skills.
In addition to the Lockheed-Martin Cyber Kill Chain whitepaper and the SANS 5 Critical ICS Controls, Webb also recommended reading articles on ControlThings.io and checking out classes taught by Justin Searle.
The recently launched HTB Alchemy Pro Lab, which focuses on ICS pentesting, is another resource he suggested. “You get to follow that kill chain all the way through, and you get to manipulate programmable logic controllers (PLC) at the end and cause impact.”
Both Dragos and Hack The Box worked on developing a realistic ICS/OT environment that allows participants to learn the many nuances of industrial environments. Participants will pivot from the enterprise environment, down into the ICS/OT where industrial components are created, manufactured, fabricated, and in this case, brewed.
Industrial environments keep our lights on, provide us with safe drinking water, and manufacture our everyday medicine. Playing the Alchemy Professional Lab will provide context behind how our world operates.
Jackson Evans-Davies — Principal Industrial Penetration Tester @ Dragos
One of the best ways to learn more about ICS is to meet and work with the people who actually operate it. To gain more hands-on experience, Webb mentioned three potential routes for developing your resume:
Working for a vendor, such as Schneider, Rockwell, or General Electric (GE). Companies that provide the technology that make up ICS networks.
Working for an ICS Security-focused consulting firm, such as Dragos. These companies often run pentests on OT networks.
Working for an asset owner or operator. Like an electric or water utility company, a railway operator, or an oil company.
Webb got his start with an asset operator, and he said it may be the “most valuable place to start.” After he worked there, “the lightbulb went off” as he realized, “Oh, this matters. This matters a whole lot.”
As many of the malware libraries for ICS systems are available for study, such as FrostyGoop and PipeDream, Webb added that people could also learn Python, track them down, and look at them. However, bear in mind that there are limitations to this strategy:
“You could get them off eBay, and set them up and program them, but doing so in a manner that replicates a real environment is a whole other thing.” Said Webb, “Then you need to be asking these engineers and operators—who have had 25 to 30 to 40 years of doing this in the field—how it’s supposed to be done.”
💡Tip: Use the HTB cybersecurity job board to find a role aligned to your career goals.
Both verbal and written communication are critical skills for an ICS pentester. This is because of the social dynamics in an ICS facility that a pentester may need to navigate, as priorities for Operations and Information Security teams can be at odds.
Webb compared this to “learning how to be a little bit of a relationship therapist,” and added that a pentester needs to be self-aware about how they interact with people, and able to mediate between the OT teams and the IT security teams.
He said that the ability to “be a buffer and a ‘calm down’ point for both parties is really important.”
Where the IT teams are likely to focus on policy and procedure, with an eye to overall asset security, the OT teams will push back on anything that breaks their processes, and could ultimately cost the asset owner time or money.
“Sometimes it [the pentest] proves IT’s point, sometimes it proves OT’s point,” he said.
As an example of this point (and another way that ICS pentesting differs from network pentesting), Webb mentioned that some critical ICS protocols travel the network unencrypted. Typically, IT teams may recommend finding a way to encrypt and protect them from man-in-the-middle (MITM) attacks or network sniffing.
However, the data those protocols provide is crucial for the operators who need to understand and troubleshoot ICS equipment. “Visibility is super key”, Webb explained, noting that encryption gets between operators and the information they need quickly.
This gives the pentester an opportunity to push back on a common IT assumption, and suggest workarounds to mitigate these unencrypted protocols. Webb highlighted the importance of “being able to pull yourself back from being super technically focused, and bring it back to a digestible level for folks.”
Written communication is also a key area to practice, as the report is what gets left behind after the pentest is over. When Webb is writing a report, he said that he wants to include all of the following elements:
Timestamp.
Target.
Source.
The action he did.
The code.
A screenshot.
“Maybe I’m a stickler, maybe it’s just aesthetics, but I think how you present something is really, really, really important.” He said, “Even if it’s just the same screenshot, but if you've got a couple edges of another window [in one screenshot] compared to just a cropped command line. I think one is far more effective.”
Webb does this for any significant action that causes a state change, or contributes to pentest objectives. Providing the details of any state changes also allows the customer to return later to roll-back his changes.
Explore a whole new, evolving security domain and step into the virtual boots of an ICS environment crafted with the support of Dragos, a leading ICS/OT cybersecurity technology and solution provider!
The Alchemy Professional Lab scenario takes cybersecurity teams through a series of security challenges that cross 9 Machines, 7 PLCs, and 21 flags to complete.
As a fresh brew out of the Hack The Box business offering, this complex hardware simulation inspired by the Purdue model’s specifications, allows red teamers and penetration testers to gain familiarity with an adversarial experience of attacking within OT environments and understanding the critical role of ICS security in business operations.
Our commitment to skills development within ICS does not end here. Along with Alchemy, there is a strong variety of theoretical and hands-on content to help strengthen key skills and develop a holistic workforce development plan.
Prior to engaging with this complex attack scenario members can engage with relevant learning within the HTB Enterprise Platform, such as learning with Modules on Supply Chain Attacks or diving into hands-on practice through the ICS & supervisory control and data acquisition (SCADA) curated collection of labs.
Having practiced with particular techniques and engagements in these scenarios, the Cyber Industrial Quest CTF Challenge Pack provides another opportunity for benchmarking while boosting team collaboration and engagement through a Capture The Flag event.
Note: Alchemy is available for all Hack The Box community members as part of the Pro Labs subscription on HTB Labs.
For more information about HTB Pro Labs, see the Pro Labs page.