In the first post on this topic we discussed the terminology of DevOps and how segregation of duties can get complicated with DevOps. In this post we will continue to investigate DevOps and discuss the issues you can encounter with change control, documentation and PCI scope.
Change Control
These days it is not unusual to hear DevOps people be proud of hundreds or even thousands of implementations or deployments per day. That is until someone like a PCI assessor starts inquiring about what, if anything, is done to formally approve all those deployments? The conversation with developers typically begins to deteriorate as you discuss requirement 6.4.5.2 which states:
“Documented change approval by authorized parties.”
The normal response is that the approval is provided in Jira, ServiceNow or whatever change management tool is being used. That leads to a discussion of the guidance for requirement 6.4.5.2 which states:
“Approval by authorized parties indicates that the change is a legitimate and approved change sanctioned by the organization.”
With the rapidity and the volume of changes, then next question asked is how can an authorized party assess a change is legitimate and sanctioned if they never actually physically see and review the change that is deployed?
This leads to a discussion of how Jira, Jenkins, Puppet, whatever CI/CD toolsets work along with the automation involved in the change process as well as the “controls” embedded in the workflow of those tools. The bottom line of which is usually that the only potential human intervention in the process might occur if the code needs a manual code review.
What requirement 6.4.5.2 is about is ensuring that the change process involves human intervention by getting management to approve what is being put into production and that segregation of duties has been ensured and no fraud or other illegal activity has been introduced into the process. The reason is that we are talking about code that is processing, storing or transmitting sensitive authentication data (SAD) or cardholder data (CHD). The potential for implementing code that skims that information or does other nefarious actions is too great to just trust a fully automated process with no human intervention. This risk can all be driven home with a discussion of the 2013 Target breach where the CI/CD process was compromised to repeatedly push malware to thousands of point of sale devices.
While I am only talking PCI in-scope code in this case, fair warning, HIPAA, SOX, GDPR and other regulations are going to require a similar control. Sensitive information is worth too much today and there is just too much risk that people will take any opportunity to siphon off sensitive data any way they can if appropriate controls are not in place. If your process is totally automated and cannot detect such fraudulent activities, the ability to put who knows what into the code is too easy. The last thing any organization wants is to be breached and then try to defend itself when they had poor or no internal controls to prevent the breach.
The bottom line here is that in our haste to push out software we have compromised the controls in that process. Those controls need to be put back into place to minimize the risk presented by pushing malicious software into applications without a thorough vetting by management.
Documentation
Another area where compliance falters with DevOps is documentation. Confluence, SharePoint Wiki or a similar tool will be used for documentation and that is where most assessors/auditors will be pointed for their requests for formal documentation.
The first problem with this is when you are an outside assessor/auditor, because you do not have access to the internally used tool. That can be remedied several ways, but it is always a hurdle because insiders are so used to the fact that everyone they typically work with has access.
Once the assessor/auditor has access, the next problem for all assessors/auditors is finding what they need for their assessment/audit. Regardless of whether the assessor/auditor gets PDFs or has online access, the most common reason for this issue is terminology. A lot of times what an assessor/auditor is trying to find will be referred to by the organization in terms that are not consistent with industry or technology accepted terminology. While all of these documentation tools have search capabilities, searching the document trove for what an assessor/auditor needs for evidence can be highly problematic. Never mind the fact that clients get frustrated as well because the evidence exists, but the assessor/auditor cannot find it.
Related to these documentation systems is the fact that it can be difficult, if not impossible, for the assessor/auditor to get hardcopy or even usable PDFs of the documentation. Let us face it, screen shots while readable can miss sentences at the end of a screen and therefore be missed altogether. As a result, obtaining usable and legible evidence for the assessor’s/auditor’s work papers is not readily possible let alone having it searchable. The fix to this is to use a browser extension or addon that will create a PDF or image of an entire page. But that too can run into issues if the organization has locked down their browsers and do not allow such installations.
Regardless of Agile/Scrum or Waterfall, the next problem with documentation is the fact that the documentation is limited or simply does not exist. I have encountered more and more organizations that again point to The Agile Manifesto, Scrum and the like and state that none of these approaches specify that documentation is required. It seems that the age-old adage of “if it was hard to develop, it should be hard to understand” is back in vogue. Never mind the fact that with hundreds or thousands of deployments a day, keeping up with documentation outside of can be impossible.
Consistent use of a change management ticketing system such as Jira or ServiceNow also can be an issue. It seems that some organizations have exceptions that do not require every change to their environment be entered into their change management solution. Worse, the criteria used to determine what is and what is not entered is not consistently applied because the criteria were never officially documented nor formally approved. As a result, there is no way to rely upon the information contained in the change management system to determine that change management is performed as required.
As a result, I am never surprised to have organizations scrambling to develop even business and IT policies, standards and procedures in addition to network diagrams, data flow diagrams, application documentation, database schemas, operations documentation and a whole host of other missing or incomplete documentation.
PCI Scope Implications
Lastly, there is the scoping issue related to the DevOps infrastructure. Not all of it is usually in scope, but that all depends on how it has been implemented.
At the very least, the Jenkins, Puppet or Ansible portion of the infrastructure are going to be in-scope for PCI. The reason is that those components feed the application updates into the cardholder data environment (CDE). So those are considered “Connected To” systems and must be properly configured and secured to be PCI compliant.
Because these CI/CD solutions are “Connected To”, this can become problematic because of who has access to Jenkins, Puppet, et. al. As I spoke of earlier, because of poor segregation of roles in the Active Directory system, it can turn out that developers have access to these systems and therefore come into scope for PCI compliance as well. As a result, the whole concept of development separate from production for requirement 6.4.1 does not exist.
Obviously, this segregation of development and production problem only gets worse if you drag even more of the development infrastructure into scope. Therefore, you want to ensure that only the Jenkins, Puppet, Ansible portion of CI/CD is in scope.
This will mean moving Jenkins, Puppet, Ansible, etc. into your “Connected To” or “Shared Services” network segment. This can create some issues with the rest of the development environment because of firewall rules and access to it through a Jump Server. So simply moving that solution into the new network segment may not be as simple as it appears.
Development Metadata
Before we go, there is one more topic that needs to be discussed and that is the metadata in all these development solutions.
We have touched on the controls surrounding the development toolsets, but we have not discussed securing these toolsets and the risks they present. This may seem a bit odd because since when have we worried about the security of Visual Studio or other integrated development environments (IDE). However, with the implementation of CI/CD solutions, all these tools become interlinked and integrated. Essentially, all these tools make up an automated assembly line for building applications.
But even more importantly, for these tools to work together seamlessly, they need to share metadata about what they are doing. This metadata might seem like it is benign information, but it is particularly important and controls how the applications are built, tested and deployed. Essentially the metadata is the “secret sauce” that makes your application work as an application within your organization.
We have already discussed the security controls that will be required around the deployment toolset. But the rest of the development toolset is also going to require security and controls to be implemented to ensure that your software factory’s assembly line does not become a huge risk for attacks. Attacks that could maliciously modify your applications to stopping the assembly line altogether.
Before you think that this is unrealistic, I would again point you to the infamous 2013 Target breach. I wrote about the breach at the time and walked people through how what was known about the breach at that time would have made the breach possible. The success of that attack was to compromise the CI/CD process to implement their malware that skimmed cards in the point of sale system. So, the idea that development environments are not a viable attack point is not out of the realm of possibility. And it gets even worse when you add in the use of contract workers to the development process.
So, what should an organization do to address these risks? I would recommend securing the entire application development environment to PCI configuration standards so that security monitoring of the entire environment can be performed. That does not mean that all the environment needs to reside in your ‘Connected To” or “Shared Services” DMZ with the CI/CD solution. But I would create another DMZ to contain the rest of the toolset that feeds the CI/CD solution. Servers should be properly security hardened and monitored as though they are in-scope for PCI compliance even though they are not in-scope.
There you have it. The basics of how Agile and PCI can coexist.