Learning Poisoned Pipeline Execution (PPE) with CI/CD goat

Poisoned Pipeline Execution (PPE) is a pentesting methodology and attack vector abuses permissions against an SCM repository, in a way that causes a CI pipeline to execute malicious commands. Users that have permissions to manipulate the CI configuration files, or other files which the CI pipeline job relies on, can modify them to contain malicious commands, ultimately “poisoning” the CI pipeline executing these commands. If you need more information, follow awesome resource HackingTricks.

Mechanisms of Poisoned Pipeline Execution

Two common mechanisms of PPE are:

• Direct PPE (D-PPE): in a D-PPE scenario, the attacker modifies the CI config file in a repository they have access to, either by pushing the change directly to an unprotected remote branch on the repo, or by submitting a PR with the change from a branch or a fork. Since the CI pipeline execution is triggered off of the “push” or ”PR” events, the attacker’s malicious commands ultimately run in the build node with high level execution privilege.
• Indirect PPE (I-PPE): so rather than poisoning the pipeline by inserting malicious commands directly into the pipeline definition file, In I-PPE, an attacker injects malicious code into files referenced by the configuration file. The malicious code is ultimately executed on the pipeline node once the pipeline is triggered and runs the commands declared in the files in question. In such a scenario, the attacker can still poison the pipeline by injecting malicious code into files referenced by the pipeline configuration file, for example using:
  • make or build file;
  • scripts referenced from within the pipeline configuration file;
  • code tests. Testing frameworks running on application code within the build process rely on dedicated files, stored in the same repository as the source code itself;
  • automatic tools: linters and security scanners used in the CI.
  • So rather than poisoning the pipeline by inserting malicious commands directly into the pipeline definition file, In I-PPE, an attacker injects malicious code into files referenced by the configuration file. The malicious code is ultimately executed on the pipeline node once the pipeline is triggered and runs the commands declared in the files in question.

Exploitation Benefits

Knowing the three flavors to poison a pipeline:

• Secrets: Any pipelines require privileges for their jobs (retrieve the code, build it, deploy it) and this privileges are usually granted in secrets. These secrets are usually accessible via env variables or files inside the system. Therefore an attacker will always try to exfiltrate as much secrets as possible.
  • Depending on the pipeline platform the attacker might need to specify the secrets in the config. This means that is the attacker cannot modify the CI configuration pipeline (I-PPE for example), he could only exfiltrate the secrets that pipeline has.
• Computation: The code is executed somewhere, depending on where is executed an attacker might be able to pivot further.
  • On-Premises: If the pipelines are executed on premises, an attacker might end in an internal network with access to more resources.
  • Cloud: The attacker could access other machines in the cloud but also could exfiltrate IAM roles/service accounts tokens from it to obtain further access inside the cloud. Even able to use a cloud workload for crypto mining or doing a high performance operations like hash bruteforcing.
  • Platforms machine: Sometimes the jobs will be execute inside the pipelines platform machines, which usually are inside a cloud with no more access.
  • Select it: Sometimes the pipelines platform will have configured several machines and if you can modify the CI configuration file you can indicate where you want to run the malicious code. In this situation, an attacker will probably run a reverse shell on each possible machine to try to exploit it further.
• Compromise production: If you ware inside the pipeline and the final version is built and deployed from it, you could compromise the code that is going to end running in production.

Direct-PPE with CI/CD Goat

Let’s take an example of direct pipeline execution poisoning, the simplest and the most common PPE. To do so, I will show you a very nice done CTF labs called CICD Goat from Cidersecurity.io. Ready to rumble!?

All we need is Kali Linux machine with Docker installed.

The lab environment contains a few docker containers:

which are simply deployed with docker compose:

$ curl -o cicd-goat/docker-compose.yaml --create-dirs https://raw.githubusercontent.com/cider-security-research/cicd-goat/main/docker-compose.yaml
cd cicd-goat && docker-compose up -d

It may take a while, downloading all images and getting up and running. If you see all of containers up as below, you are ready to go:

Logining into CTF portal under localhost:8000. We will go with ‘White Rabbit’ challenge.

First, lets look into localhost:3000/Wonderland/white-rabbit repository. Noticed, there is a jenkinsfile which contains CI/CD logic. This is a good place to start:

After some googling, I have found an a way to get into Jenkins environment variables. The next step is modifying a stage to display new environment variable with flag1:

Next, create a new branch and a Pull Request. The PR will trigger a build pipeline. Go to Jenkins console, open wonderland-white-rabbit project, the new PR should be created. Now, we only need to find logs:

Copy Base64 encoded flag1 and decode it in console:

Here we go, first challenged done! It was a simple example how to use direct poisoned pipeline execution to get sensitive information using CI/CD flow.

Indirect-PPE example

It was pretty easy, but what if jenkinsfile is protected? Here is time to try indirect poisoning (I-PPE). Go to CTF portal and try the next challenge called ‘Man Hatter’.

Let’s have a looks at repo. We could easy detect a two repositorium there:

First simply contains jenkinsfile with flag3 credentials inside. Let’s try to echo as we did before. To do so, we need to fork repositorium, make changes and push. Hm. It looks like we have no permission to contribute in to the repo. It does not work even creating a new branch.

We already know the infrastructure-as-code repositorium which contains CI/CD logic is protected.

So, it need another way. As we saw in jenkinsfile, there is a make stage which build a application from source code which is located in another repo. What if we will poison a make file? Doing so, pull down a localhost:3000/Wonderland/mad-hatter repo and a try to make changes in Makefile:

Done! Now, go to gitea and create a new pull request:

Just getting back to Jenkins console and new build appears right there under wonderland-mad-hatter project:

Go and check make stage log for a token:

Well, all we done is getting a flag indirectly by modifying Makefile from separate repository we have access to.

Recommendations

We saw how easy it might be compromising CI/CD infrastructure. So, now let’s say a few words how to prevent and mitigate the PPE attack vector involves multiple measures spanning across both SCM and CI systems:

• First of all, ensure that pipelines running unreviewed code are executed on isolated nodes, not exposed to secrets and sensitive environments.
• Where possible, restrict from running pipelines originating from forks, and consider adding controls such as requiring manual approval for pipeline execution.
• For sensitive pipelines, for example those that are exposed to secrets, ensure that each branch that is configured to trigger a pipeline in the CI system has a correlating branch protection rule in the SCM.
• Use secured key vault like Azure KeyVault or HashiCorp Vault and appropriated role-base access control to list and get sensitive information.
• To prevent the manipulation of the CI configuration file to run malicious code in the pipeline, each CI configuration file must be reviewed before the pipeline runs. Alternatively, the CI configuration file can be managed in a remote branch, separate from the branch containing the code being built in the pipeline. The remote branch should be configured as protected.
• Apply Principal of Least Privileges. Remove permissions granted on the SCM repository from users that do not need them.
• Each pipeline should only have access to the credentials it needs to fulfill its purpose.

I suppose it would interesting talk, we went throw a few amazing CTF labs in CI/CD goat powered by Cidersecurity.io.

Be an ethical, keep your infrastructure save!