Securing Salesforce DevOps: Multi-Job Workflows in GitHub Actions

Getting it working is not the same thing as getting it working securely! While seeing automation work for the first time can be exhilarating, it's important to remember to take a step back and consider the security of your implementation.

One nuanced yet critical challenge I've encountered over the past two years is securely splitting GitHub Actions workflows into multiple jobs that interact with the same Salesforce org. This post delves into this challenge, why it matters, and how we can address it securely.

This article was inspired by multiple posts I've seen from people trying to split up CI/CD workflows across multiple jobs, often misusing build system features like artifacts and caching against security guidance from the vendor and best practices in general.

Not a GitHub user? No problem! Most of this post applies to your CI/CD system of choice. Almost all CI/CD systems have a concept of Workflows, Jobs, and Steps and some mechanism for passing information between them. Just map those words to your platform and read away.

Overview: Understanding Workflows, Jobs, and Steps in GitHub Actions

In GitHub Actions, a workflow is an automated process triggered by events. Each workflow comprises one or more jobs, which are collections of steps executed on the same runner. Here's a quick breakdown:

• Workflow: The entire automation process, defined in a YAML file.
• Job: A set of steps executed in sequence on the same runner.
  • Run in their own isolated containers, including custom Docker images
  • Can specify dependencies on other Jobs (needs) and run in parallel
  • Can use matrix inputs for parallel testing of multiple org setups
  • Can be individually rerun on failure
• Step: An individual task, like running a command or an action.

Jobs are clearly a critical component of designing Workflows. Being stuck in a single Job constrains our ability to fully use all of GitHub's features around Actions.

The Challenge

Both Salesforce CLI (sf) or CumulusCI (cci) store org credentials in local files. Most build scripts start with connecting the target org(s) to the sf or cci keychain.

When splitting workflows, persisting data between jobs becomes necessary. Specifically, for Salesforce workflows, sharing the ~/.sfdx and ~/.cumulusci directories—which contain org authentication details—is essential, especially for workflows that create a scratch org and need to pass that org to another Job. However, doing this securely without violating GitHub's security guidelines is challenging.

Security Implications

As part of our focus on Securing Salesforce DevOps, let's take a deeper dive into the security implications hiding below the surface of that working automation...

The Sensitivity of CI/CD Tokens

CI/CD tokens for Salesforce orgs are highly sensitive. They often grant extensive permissions, and mishandling them can lead to significant security breaches. Referencing our previous post on Least Privilege Access Control, it's clear that over-privileged credentials are a risk.

"But It's Just a Scratch Org!"

It's tempting to downplay the risk because scratch orgs are temporary. However:

• Potential Data Exposure: Scratch orgs may contain sensitive IP like metadata or configurations.
• Credential Leakage: If the ~/.sfdx directory contains other org credentials (like the DevHub), inadvertently sharing it exposes more than just the scratch org.
• Compliance Violations: Using methods that contravene GitHub's security guidelines can lead to non-compliance with industry regulations.

GitHub's Security Guidance

Specifically regarding passing sensitive data between jobs, GitHub provides a skeleton code example of using an external secrets store, along with this note:

"If you want to pass a masked secret between jobs or workflows, you should store the secret in a store and then retrieve it in the subsequent job or workflow." — GitHub Documentation

Ignoring this guidance not only puts your data at risk but also goes against best practices.

Secure Options for Data Persistence Between Jobs

Unfortunately, GitHub doesn't provide us many options here. In fairness, it wasn't really built to handle this type of challenge and their documentation clearly says to use an external secrets manager.

Option 1: Using a Single Job

The most straightforward solution is to keep all steps within a single job. Since jobs run on the same runner instance, the ~/.sfdx and ~/.cumulusci directories persist throughout the job's execution.

I know... this isn't solving the original problem. This is pretty much where we've been stuck in designing D2X's reusable workflows.

Pros:

• Security: No need to transfer sensitive files between jobs.
• Simplicity: Easier to manage and less prone to security mishaps.

Cons:

• Limited Parallelization: Cannot run steps in parallel within the same job.
• Longer Build Times: Sequential execution might increase total workflow duration.

Option 2: Using an External Secrets Store

Utilizing an external secrets manager like HashiCorp Vault or AWS Secrets Manager allows you to securely store and retrieve credentials across jobs.

Implementation Steps:

1. Store Credentials Securely: Keep your org credentials in the secrets manager.
2. Retrieve in Each Job: At the start of each job, securely fetch the necessary credentials.
3. Access Control Policies: Define strict policies to control which jobs and runners can access specific secrets.
4. Auditing and Logging: Monitor access to secrets for compliance and security auditing.

Pros:

• Security: Credentials are centrally managed and securely accessed.
• Flexibility: Jobs remain independent and can run in parallel.

Cons:

• Complexity: Additional setup and management overhead.
• Cost: Potential additional costs associated with external services.

Insecure or Unworkable Options

Here's a brief rundown of the options we've explored and decided against.

Using GitHub Secrets

Secrets are GitHub's recommended way of handling passing sensitive values to workers. If you can use Secrets, use them! Unfortunately, in this case we can't:

• Limitation: Secrets are read-only and cannot be modified by jobs.
• Risk: Jobs cannot write back updated credentials (like refreshed tokens) or create new secrets, and attempting to misuse secrets can lead to security vulnerabilities.

Artifacts

Artifacts are meant to publish files from the workflow, annotating the workflow run with the files. Artifacts are typically used for packages, image names, JUnit test reports, etc.

Warning: Do not store any sensitive data in artifacts. Artifacts are accessible by anyone with read access to the repository. — GitHub Docs

• Persistence: Artifacts persist beyond the job and can be accessed by anyone with read access to the repository.

Caches

Warning: Do not store any sensitive data in the cache. The cache is stored on GitHub servers and is accessible to anyone with read access to the repository. — GitHub Docs

• Not Secure for Secrets: Caches are designed for dependencies, not sensitive data.
• Risk of Exposure: Caches can be restored in unintended contexts, leading to credential leakage.

Environment Variables and Outputs

• Size Limitations: Cannot handle large data like entire directories.
• Security Concerns: Outputs are logged and may inadvertently expose sensitive information.
• Secrets Masking: Passing structured data like JSON can cause GitHub's secrets masking functionality to fail to mask a secret.

Conclusion

Balancing efficiency and security in Salesforce DevOps is a complex task. While splitting workflows into multiple jobs offers significant benefits, it's crucial to handle data persistence securely. Using a single job or integrating an external secrets manager are the secure options.

Remember, the convenience of cutting corners isn't worth the potential security risks. Always prioritize safeguarding your credentials and comply with platform guidelines to protect your organization's assets.