DevOps are all the rage in organizations that develop applications. The move to become “Agile” through the implementation of methodologies such as Scrum to replace the traditional waterfall SDLC is ongoing in most organizations. But these changes can create compliance issues with the PCI standards regarding software development.
Understanding The Terminology
First and foremost, we need to address the terminology surrounding DevOps.
But before we talk about those specific terms, we need to address the elephant in the room which is “Agile”. The Agile approach to development traces its history back to early 2001 when a group of developers met at a Utah ski resort. The result of that meeting was ‘The Agile Manifesto’. However, the roots for Agile were sown even earlier as application development became unable to keep pace with business changes starting in the late 1980s.
The important thing to remember about Agile is that it is not a methodology. It is merely a set of values (4) and principles (12) related to the development of software. The Agile Manifesto never describes a roadmap or steps to follow as to how those values and principles should be used. So, to refer to Agile as a methodology is a misnomer but you will constantly encounter it being referred to as though it were a methodology.
Interestingly enough, the methodologies used with the Agile approach were actually developed before Agile. Of the number of them that sprang up in the 1990s, Scrum seems to have won out when it comes to a methodology. Scrum was one of many methodologies such as Kanban, Crystal Clear, Extreme Programming, Feature Driven Development and Dynamic Systems Development Method that came out at that time to address the delivery of software solutions in a timelier manner. But while Scrum is the most followed methodology, it can also include some of these other methodologies such as Extreme Programming (XP) for example when used.
Scrum involves three types of roles.
- Product Owner: The Product Owner needs to be a person with vision, authority, and availability because they are responsible for continuously communicating the vision and priorities to the development team.
- Scrum Master: The Scrum Master is not a project manager. The Scrum Master’s primary responsibility is to remove any impediments that are obstructing the team from achieving its sprint goals. The Scrum Master also is the primary contact with the Product Owner.
- Team: The Scrum team is responsible for completing the work. For application development, a Scrum team can contain anywhere from three to nine members. For software projects, a typical team includes a mix of software engineers, architects, information security personnel, programmers, analysts, QA experts, testers, and UI designers. The team is responsible for determining how it will accomplish the work to be completed.
The final term from Scrum that needs to be defined is Sprint. A Sprint is a one month or less in duration project that will result in a releasable increment of a product, in this case, an application or application enhancements. When a Sprint’s horizon is too long the definition of what is being built may change, complexity may change, and risk may change. The concepts of Sprints are to enable predictability by ensuring inspection and adaptation of progress toward a Sprint Goal at least every calendar month. Another benefit is that Sprints limit risk to one calendar month of cost.
Once defined, some of the key characteristics of Sprints are:
- No changes are made that would endanger the Sprint Goal;
- Quality goals do not decrease; and,
- Scope may be clarified and re-negotiated between the “Product Owner” and “Team” as more is learned.
With these behind us, let us now turn to the terms DevOps.
DevOps is a merging of development and operations staff to work together to develop and implement solutions that will essentially run 24x7x365 with (hopefully) minimal operational interaction. DevSecOps merely formally adds in the collaboration of information security into that mix even though information security should be included in DevOps as well.
The final topic in our discussion of terminology regards the tools used by DevOps. While there are a number of vendors in this space, the “Big Dog” at the moment in DevOps is Atlassian with their tools Confluence and Jira. Another “Big Dog” is Microsoft’s GitHub and the other “Big Dog” in the DevOps tool world is Jenkins which is open source from CloudBees.
- Confluence is used as a documentation repository for such items as policies, standards and procedures as well as business, application, network and other important documentation.
- Jira is used as a project and change management ticketing system.
- GitHub is used to manage the versions of applications.
- Jenkins is used for automating the build, testing and deployment of applications into production.
All of these tools have competitors from vendors such as ServiceNow, Puppet, Ansible, Chef, Google, and other commercial and open source development and operations tool vendors. Regardless of vendor, all solutions seem to have these three basic components of documentation repository, project/change management and deployment automation. It is also not unusual to find multiple tools in place particularly with Jenkins, Ansible, Puppet and Chef.
Segregation Of Duties
The first and most contentious issue that comes up with DevOps is the segregation of duties. This is typically one of the biggest discussions/arguments an assessor/auditor will get into regarding DevOps is when Agile fans argue that segregation of duties is inconsistent with Scrum, Agile and DevOps. Their primary reason will be to point to the fact that nowhere in any of the documentation regarding these topics is the term ‘segregation of duties’ or the requirement to ensure segregation of duties. They would be correct in that regard.
Unfortunately, corporate life is not driven by Scrum, Agile or DevOps in a vacuum. Corporations are still required to comply with laws, contracts and regulations promulgated on them by government entities, business partners, financial institutions and other parties regardless of what is in their methodologies and approaches. So, while the argument can be made that the methods and approaches do not state anything on the subject, there are other documents, contracts and requirements that do state it is required.
Whether we are discussing PCI, NIST, SOC, COBIT or any other recognized audit or compliance programs, segregation of duties between roles is and always has been required. It is one of the key principles to ensure that people do not have the ability to corrupt a process because they have too much control over that process. The reason it is, that time and again one of, if not the primary root cause of such illicit activities, is the failure to segregate duties and roles thus allowing one person too much control over a process. The concept of segregation of duties being that the more individuals involved in a process the less likely a process can or will be abused.
In DevOps, the issue of segregation of duties gets complicated because it gets extended into the tools used in the process. The concept of continuous implementation (CI)/continuous deployment (CD) relies heavily on the use of tools such as Jira and Jenkins to enable such an approach. This means that the assessor/auditor needs to look into who has access to these tools and what rights they have to influence the workflows that exist in these tools.
This gets even more complicated by the fact that this requires analysis of user and access control information from tools such as Active Directory, RADIUS and even the tools themselves. In my experience, it is not unusual to peel the onion on these access controls to reveal the fact that segregation of duties really does not exist as thought because all roles are granted to everyone in DevOps and the organization is relying on individuals’ honesty to ensure compliance.
DevOps also can suffer from segregation between production, quality assurance (QA), test and development environments. This is because a lot of organizations that move to DevOps have the mistaken belief that the “Operations” and “Security” components become part of the development group. The argument will be made that Agile is all about “breaking down silos”. While that is true, the mistake they make is that Agile and Scrum were not a call to abrogate the knowledge and controls that all of the players involved bring to the table as separate disciplines. The goal is to make the disciplines better work together to achieve a common goal in a Sprint.
Where this manifests itself most often is that developers have unfettered access to the production environment. In a DevOps environment, it is not unusual to find developers scattered all throughout the environment. They are developing code, they are operating production, they are diagnosing bugs, they are everywhere with no delineation of roles and responsibilities. It is essentially a free for all. Everyone pitches in where they need to be involved.
This organized chaos is supposedly “controlled” by Jira through its ticketing. Agile advocates will claim that since everything has a ticket (not always a true statement) that they maintain segregation through Jira. They will show the tickets to the assessor/auditor and display that there are different names on the ticket for the developer, the QA person, the people who approved promotion, etc. While this is true, as I described earlier, the access controls will show that virtually everyone they gave as evidence of segregation can fulfill any of those roles whenever they so choose. By definition, that is not segregation of duties because there are no actual controls in place to stop someone from running the whole process.
The bottom line in this discussion is that the segregation of duty controls in an Agile environment is usually illusory. As such, it is management’s responsibility to periodically ensure that segregation of duty controls are truly implemented and testable.
In the next post we will discuss documentation, change control and PCI scope in an Agile environment.