Pipeline Hardening
Runner Isolation CI/CD runners should execute in isolated environments that prevent cross-contamination between builds and limit blast radius from compromised builds. Ephemeral runners that are created for each build and destroyed afterward eliminate persistent state that could be poisoned by malicious builds. Shared runners create risks where malicious builds can poison runner state, steal secrets from subsequent builds, or establish persistence through backdoors. Dedicated runners per project or team provide isolation while maintaining performance through runner pools. Container-based runners provide process isolation through containerization, while VM-based runners provide stronger isolation at the cost of slower startup times. The choice depends on threat model and performance requirements. Network Restrictions Build runners should have outbound network access disabled by default, with explicit allowlists for required package repositories, container registries, and cloud APIs. This approach prevents data exfiltration through DNS tunneling, HTTP requests, or other network channels. Egress filtering through proxies or firewalls enables logging and inspection of outbound connections, detecting anomalous network activity that may indicate compromise. Package repository mirrors within controlled networks reduce external dependencies while improving build performance. Least Privilege Access Pipeline automation should use least-privilege credentials scoped to specific operations rather than broad administrative access. Source control management (SCM) access should be read-only for code checkout, with write access limited to specific branches or tags through separate credentials. Container registry and artifact repository access should be scoped to specific repositories with separate read and write credentials. Cloud provider access should use workload identity federation (OIDC) that provides temporary credentials rather than long-lived access keys. OIDC workload identity enables pipelines to authenticate to cloud providers using short-lived tokens issued by CI/CD platforms, eliminating long-lived credentials that create persistent security risks. Token claims can include repository, branch, and commit information, enabling fine-grained access policies. Secret Management Secrets should be retrieved from secret brokers or vaults rather than stored in pipeline configuration or environment variables. Secret masking in logs prevents accidental secret exposure, while secret rotation limits the window of opportunity for stolen secrets. Secrets should never be echoed to console output or written to files that might be archived as build artifacts. Secret detection tools scan code, logs, and artifacts for accidentally committed secrets, preventing credential leaks. Exfiltration detection monitors for secrets being transmitted to unauthorized destinations, using network monitoring, data loss prevention, or secret usage analytics to identify suspicious secret access patterns.Provenance and Integrity
Signed Commits and Tags Commit signing with GPG or Sigstore provides cryptographic proof that commits were created by authorized developers. Signed commits prevent attackers from impersonating developers through compromised credentials or malicious pull requests. Branch protection rules should require signed commits for protected branches, preventing unsigned commits from reaching production. Tag signing provides additional assurance for release tags, ensuring that releases correspond to authorized code. Required code reviews and status checks provide human and automated validation before merging changes. Review requirements should specify minimum reviewer count and prevent authors from approving their own changes. Reproducible Builds Reproducible builds produce identical artifacts from identical source code, enabling verification that published artifacts match source code. Reproducibility requires eliminating non-deterministic build inputs including timestamps, random values, and environment-specific paths. Build reproducibility enables independent verification of artifacts, detecting supply chain attacks that inject malicious code during build processes. Reproducible build tooling provides deterministic compilation, packaging, and artifact generation. SLSA Provenance Supply chain Levels for Software Artifacts (SLSA) framework defines maturity levels for supply chain security, with provenance attestations documenting build process details. SLSA provenance includes source repository, commit hash, build platform, build parameters, and dependencies. Provenance attestations are cryptographically signed by build systems, enabling verification that artifacts were built through approved processes. Higher SLSA levels require stronger build isolation, more comprehensive provenance, and cryptographic binding between source and artifacts. Artifact Signing and Verification Build artifacts including container images, binaries, and packages should be cryptographically signed by build systems. Signature verification before deployment ensures that only artifacts built through approved pipelines reach production. Binary Authorization (GCP), Kritis (Kubernetes), and Connaisseur provide policy-based deployment controls that require valid signatures before allowing artifact deployment. Signature policies can specify required signers, key rotation schedules, and attestation requirements. Container image signing with Docker Content Trust, Notary, or Sigstore provides cryptographic verification of image provenance. Image signature verification prevents deployment of unsigned or tampered images.Environment Segregation
Separate Cloud Accounts Production and non-production environments should use separate cloud accounts, subscriptions, or projects that provide strong isolation boundaries. Separate accounts prevent lateral movement from compromised development environments to production. Distinct credentials per environment ensure that development credentials cannot access production resources. Production access from CI/CD should be limited to controlled deployment orchestrators rather than direct pipeline access. Deployment Orchestrators Deployment orchestrators mediate between CI/CD pipelines and production environments, providing additional security controls including approval workflows, change windows, and deployment validation. Orchestrators can enforce deployment policies like required approvals, automated testing, and gradual rollouts. Allowlists specify which artifacts can be deployed to production, preventing deployment of unapproved or unverified artifacts. Change windows restrict deployments to approved time periods, reducing risk of deployments during high-traffic periods or outside business hours. Deployment orchestrators provide centralized audit logging of all production deployments, including who deployed what artifact when and with what approvals.Supply Chain Controls
Dependency Management Dependency pinning specifies exact versions of dependencies rather than version ranges, ensuring consistent builds and preventing dependency confusion attacks. Lockfiles capture complete dependency trees including transitive dependencies. Vendor allowlists restrict dependencies to approved sources, preventing malicious packages from untrusted repositories. Private package mirrors provide additional control by proxying public repositories through internal infrastructure. Software Bill of Materials (SBOM) generation documents all dependencies included in artifacts, enabling vulnerability tracking and license compliance. SBOM verification ensures that deployed artifacts contain only approved dependencies. Vulnerability Scanning Automated vulnerability scanning should run during build processes, detecting known vulnerabilities in dependencies and base images. Vulnerability gates prevent deployment of artifacts with critical or high-severity vulnerabilities. Continuous vulnerability monitoring scans deployed artifacts for newly discovered vulnerabilities, triggering remediation workflows when new vulnerabilities affect production systems. Vulnerability databases should be updated frequently to detect emerging threats.Observability and Monitoring
Pipeline Audit Logging Comprehensive audit logging captures all pipeline actions including builds, deployments, secret access, and configuration changes. Logs should be stored immutably in centralized logging systems with long retention periods. Audit logs enable forensic investigation of security incidents, compliance reporting, and detection of suspicious activities. Log analysis identifies unusual patterns like builds at unusual times, unexpected secret access, or configuration changes by unauthorized users. Alerting and Anomaly Detection Automated alerting detects suspicious pipeline activities including unusual access patterns, pipeline configuration changes, failed authentication attempts, and secret exfiltration indicators. Alerts should integrate with incident response workflows for rapid investigation. Anomaly detection using machine learning or statistical analysis identifies deviations from normal pipeline behavior, detecting novel attacks that don’t match known patterns. Baseline models of normal pipeline activity enable detection of unusual build times, resource usage, or network activity.Conclusion
DevSecOps pipeline security requires comprehensive controls across runner isolation, secret management, artifact integrity, and environment segregation. Security engineers design pipeline security that treats CI/CD infrastructure as critical attack surface requiring hardening comparable to production systems. Success requires defense-in-depth with multiple layers of controls that prevent, detect, and respond to pipeline compromises. Organizations that invest in pipeline security fundamentals build software supply chains that resist attacks while maintaining development velocity.References
- SLSA Supply Chain Security Framework
- NIST SP 800-218 Secure Software Development Framework (SSDF)
- GitHub Advanced Security Best Practices
- GitLab CI/CD Security Best Practices
- CNCF Software Supply Chain Security Paper