The National Institute of Standards and Technology (NIST) has issued the NIST SP 800-204C “Implementation of DevSecOps for a Microservices-based Application with Service Mesh.” The guidelines include many best practices on how organizations and secure their CI/CD pipeline and enhance the software delivery processes.

NIST released on March 8, 2022 a new Special Publication (SP) 800-204C, which provides a “comprehensive guidance for the implementation of DevSecOps primitives for a reference platform hosting a cloud-native application.”

The new guideline includes an overview of the DevSecOps platform to include CI/CD pipelines, its building blocks, the design and execution of CI/CD pipelines and automation for efficient workflows.

Cloud-native apps

Cloud-native applications consist of multiple “loosely coupled” components called microservices (e.g., containers) supported by infrastructure used to provide application services, such as service mesh. Each of these microservice components are hosted within a container orchestration and resource management platform.

Examples of such container platform management tools include Kubernetes, Docker and Openshift, just to name a few.

As part of the cloud-native architecture, NIST describes the application environment as made up of five primary code types:

Application code (which embodies the application logic)
Application services code (for services such as session establishment, network connection, etc.)
Infrastructure as code (for provisioning and configuring computing, networking, and storage resources)
Policy as code (for defining runtime policies such as zero trust expressed as a declarative code)
Observability as code (for the continuous monitoring of an application runtime state).

To enhance security and agility of these components, DevSecOps (aka, “Development, Security, and Operations”) has surfaced as a trending approach to automate the integration of security early and at every phase of software development lifecycle.

Similarly, organizations look to “shift left” and ensure shared responsibility of security through the entire IT lifecycle.

DevSecOps platform

As NIST described, “DevSecOps incorporates security into the software engineering process early on. It integrates and automates security processes and tooling into all of the development workflow”, which is also known as the CI/CD pipeline.

The DevSecOps platform consists at minimum of these four components:

Pipeline software (i.e., CI and CD software):
- Continuous Integration (CI) – pulls code from a code repository, invokes the build software, invokes test tools and stores back tested artifacts to Image registry.
- Continuous Delivery or Deployment (CD) – pulls out artifacts, packages and based on computing, network and storage resource descriptions in IaC deploys the package
SDLC software:
- Build tools (e.g., IDEs)
- Software scanning and testing tools (e.g., SAST, DAST, SCA)
Code repositories:
- Source code repositories (e.g., GitHub)
- Container image repositories or registries
Observability or Monitoring Tools:
- Logging and log aggregation tools
- Tools that generate metrics
- Tracing Tools (sequence of application calls)
- Visualization tools (combine data from above).

CI/CD pipeline tools

In order to manage and automate the CI/CD pipeline process, DevSecOps and DevOps teams need good CI/CD pipeline tools. Each phase of the process can be triggered via CI/CD tools to help automate workflows for seamless and secure code delivery.

Examples of popular CI/CD pipelines tools to consider include, but not limited to: Jenkins, GitLab, AWS CodePipeline and Bamboo, just to name a few.

AST tools

Static application security testing (SAST) tools are used to scan and analyze an application’s source code for vulnerabilities during the programming of testing phases of software development lifecycle. It can flag issued or vulnerabilities and mark the software commit as insecure and block the deployment. SAST can also find hard-coded passwords or secrets in code that should be removed.

Dynamic application security testing (DAST) tools are used to analyze applications in their dynamic, running state during testing or operational phases. Examples would be to scan a “live” web application URL or simulate attacks against an application, containers and/or executables.

Software composition analysis (SCA) tools are used to identify open-source and third-party
components in use in an application, as well as imbedded security vulnerabilities.

Some examples of solutions that offer some or all of the above software testing tools or capabilities include, but not limited to Veracode, Fortify, Checkmarx, Gitlab, or BlackDuck, just to name a few. Additionally, cloud hosting providers such as AWS also offers CI/CD pipeline integrations with open source SAST, DAST and SCA tools.

Infrastructure as code (IaC) involves programmable code used for the automating the provisioning/deprovisioning of IT infrastructure (e.g., deployment of servers, networks and services) in the cloud. A resource manager or deployment manager is the software used to contain IaC.

Thus, IaC also needs to be scanned for potential vulnerabilities before it enters the pipeline and is merged.

In addition, application security testing (AST) tools, such as Twistlock (being re-branded as part of Prisma Cloud), are also used to scan container images for vulnerabilities, issues or bad practices (such as hard-coded secrets), as well as used for hardening scans to check images against regulatory/compliance standards (e.g., CIS, PCI).

Policy as code

According to NIST, “policy as code involves codifying all policies and running them as part of the CI/CD pipeline so that they become an integral part of the application runtime.”

Examples of policies include networking, zero trust, container, storage, access control, supply chain, and audit & accountability:

Networking policies and zero trust policies:
- Blocking designated ports
- Designating ingress host names
- Network access control policies
Implementation artifact policies (e.g., container policies):
- Hardening of servers, vulnerability scans for base images
- Ensuring that containers do not run as root
- Blocking privilege escalation for containers
Storage policies:
- Set persistent volume sizes
- Set persistent volume reclaim policies
Access control policies:
- Ensure that policies cover all data objects
- Ensure that policies cover all roles for administrative and application access
- Ensure that data protection policies cover data at rest, data in transit, and data in use
- Ensure that policies of all types do not have conflicts
Supply chain policies:
- Allow only approved container registries
- Allow only certified libraries
Audit & accountability policies:
- Ensure that there are policies associated with audit and accountability functions.

Observability and Monitoring

According to NIST, observability as code deploys a monitoring agent in each of the application’s service components to collect the three types of data (e.g., logging, tracing and monitoring/metrics) and sends the data to “specialized tools that correlate them, perform analysis, and display the analyzed consolidated data on dashboards to present an overall application-level picture.

Examples of observability tools include, but not limited to Prometheus, Grafana Labs, OpenTelemetry, AppDynamics, and Splunk, just to name a few.

Securing the CI/CD pipeline

Given the importance of CI/CD pipelines are for deploying secure code, it goes without saying the CI/CD pipeline itself must be secured.

NIST provides these good best practices on how to secure the CI/CD pipeline:

Harden servers hosting code and artifact repositories.
Secure the credentials used for accessing repositories such as authorization tokens and for generating pull requests.
Implement controls on who can check-in and check-out in container image registries (used by CI/CD pipelines).
Log all code and build update activities.
If build or test fails in CI pipeline, send build reports to developers and stop further pipeline tasks. The code repositories should be configured to automatically block all pull requests from CD pipeline.
If audit fails, send build reports to the security team and stop further pipeline tasks.
Ensure that developers can only access the application code and not any of the five pipeline code types.
During the build and release process, sign the release artifact during each required CI/CD stage (preferably multi-party signing).
During production release, verify that all required signatures (generated with multiple phase keys) are present to ensure that no one bypasses the pipeline.

Benefits of DevSecOps

In addition, NIST describes these major security and business benefits of DevSecOps approach:

Better communication, collaboration and productivity among various IT teams (such as developers, operations and security teams).
Streamlined software development, delivery and deployment process that results in less downtime, faster time to market, and lower infrastructure/operational costs with automation.
Reduction of attack surfaces by implementing zero trust, that helps restrict lateral movement and privilege escalation by attackers.
Better security through validation and continuous monitoring, alerting and feedback mechanisms (e.g, observability as code or modern behavior prevention capabilities).
Provide feedback to the right repository to update code and retrigger the pipelines when defective code is detected.
Ability to kill malicious containers in runtime.
Monitor new and terminated services and adjust associated services (e.g., service proxy).
Enable security assertions (such as non-bypassable by proxies executing in same space, secure sessions, robust authentication/authorization and secure state transitions.
Enable Continuous Authorization to operate (C-ATO).

Finally, readers can review the full SP 800-204C guidelines for more details, to include references to additional security guidelines on the DevSecOps and CI/CD pipeline processes.