Implementing Nuclei into your GitLab CI/CD Pipeline for Scanning Live Web Applications

Security is a fundamental requirement for businesses of all sizes. In recent years, DevOps has evolved into DevSecOps, which emphasizes the integration of security at every stage of the development process. As previously discussed, it is critical to incorporate security into the DevOps cycle to detect and mitigate potential security risks early on. This approach enables organizations to identify and address security vulnerabilities and misconfigurations before they escalate into significant issues. Integrating security into the development process ensures that security is not an afterthought but rather a critical component from the outset. By adopting this proactive approach, businesses can protect their infrastructure, data, and reputation while maintaining the trust of their customers and stakeholders. Ultimately, it is essential to prioritize security at every level to build and maintain a secure and trustworthy business environment.

In this blog, we will explore how to use Nuclei, a powerful open-source tool for scanning web applications, in your GitLab CI/CD pipeline.

First: What is GitLab CI/CD

GitLab CI/CD is a continuous integration and continuous delivery (CI/CD) system built into GitLab, a web-based Git repository manager. It allows developers to automatically build, test, and deploy their code changes with minimal manual intervention.

This helps to ensure that code changes are thoroughly tested and deployed quickly and reliably, reducing the risk of errors and downtime. GitLab CI/CD can be used with a variety of programming languages and frameworks, and can be integrated with a wide range of tools and services.

GitLab CI/CD uses a file called .gitlab-ci.yml to define the CI/CD pipeline, consisting of a series of jobs and stages executed in a specific order. Each job defines a specific task, such as building an application, running tests, or deploying code to a production environment.

The stages are used to group jobs and define the order in which they will be executed. For example, you might have a "build" stage followed by a "test" stage and a "deploy" stage.

Nuclei - A fast & customizable vulnerability scanner

Nuclei is an open-source tool for scanning vulnerabilities with the help of pre-defined templates written in YAML. It can scan any vulnerability class, including cross-site scripting (XSS), SQL injection, and remote command execution. Nuclei are fast and efficient, making them a powerful tool for identifying potential application vulnerabilities.

It can be set up to run in an internal network environment for local and publicly deployed applications.

Power of Nuclei

One of the key benefits of Nuclei is its ability to scan a wide range of vulnerabilities quickly and accurately. It uses pre-defined templates, called "rules," to scan for known vulnerabilities, and it can also be customized to scan for specific vulnerabilities or issues. This makes it an ideal tool for identifying potential vulnerabilities in your web applications.

Now, let’s jump back to the original discussion and understand how to integrate nuclei in your CI/CD pipeline.

Integrating Nuclei in GitLab CI/CD pipeline

Integrating the nuclei scanning capabilities into your CI/CD pipeline is quite straightforward. We will create a private repository on GitLab and assume it hosts code for an application, say http://testphp.vulnweb.com/.

Note: Make sure you have verified your identity on GitLab. Otherwise, your shared runner pipeline request will fail, and you may end up being unsuccessful running the pipeline.

Follow the below steps to set up a CI/CD pipeline in GitLab and run a workflow to launch Nuclei Scan on testphp.vulnweb.com with every code change.

1. Navigate to GitLab at https://gitlab.com/ and log in with your account.
2. Click on the + Icon in the top right corner and select the “New project/repository” option.

3. Create a new repository with any name of your choice and make it private.

4. Now, click on the “Set up CI/CD” button.

5. Click on “Configure Pipeline” which will create a “.gitlab-ci.yml” file that is a workflow file to run the CI/CD pipeline.

6. Now put the below workflow code in the ".gitlab-ci.yml” file and save it.

Explanation of the code: The above code will use the latest version of golang image and further will install nuclei using the go install command. It then defines a single job called "scan" that runs the nuclei command. The -u flag is used to specify the target to scan, which is https://testphp.vulnweb.com. The -json flag is used to specify the output format, and the “only” keyword is used to specify that this job should only run when code is pushed to the repository.

7. Change the -u parameter with your application URL in the above code. It also supports multiple URLs. You can use specific nuclei commands as it will work similarly to how you run nuclei in your command line. A detailed guide on nuclei can be found here: https://github.com/projectdiscovery/nuclei

8. Now, save your code and observe that the pipeline will start executing. Simply click on CI/CD Pipeline and select your runner.

9. You can further inspect your runner by clicking on Jobs and opening the specific job.

This is how you run nuclei on your web application on every code push using a GitLab CI/CD pipeline.

Now, let’s take this one step further and discuss how you can use the nuclei for regression testing using the GitLab CI/CD pipeline as we did for the GitHub CI/CD pipeline. In case you missed our previous blog on how to implement nuclei in the GitHub CI/CD pipeline, read it here: https://blog.projectdiscovery.io/implementing-nuclei-into-your-github-ci-cd-for-scanning-live-web-applications/

Performing regression testing with Nuclei custom templates

Keeping track of known vulnerabilities and ensuring that they are properly remediated is an essential task for any organization. With the constant evolution of technology and the frequent release of new code, it is crucial to perform regular regression testing to ensure that vulnerabilities are not reintroduced. However, this can be a tedious and resource-intensive task that can impact release schedules.

Fortunately, there is a solution that can streamline this process and make it more efficient. Nuclei is a powerful tool that allows organizations to perform regression testing using custom templates. By writing templates that specifically target known vulnerabilities, organizations can easily scan their code for any potential issues.

The process of writing templates for Nuclei is straightforward and user-friendly. Nuclei provides a comprehensive guide on its website that walks users through the process of creating templates. This guide can be found at https://nuclei.projectdiscovery.io/templating-guide/.

Let's dive into how custom templates can be utilized in the CI/CD pipeline to perform regression testing. By incorporating custom template scanning into your workflow, you can easily scan for known vulnerabilities and ensure that they are properly remediated with every code change.

To implement this feature, simply replace the code in your workflow file with the following instructions:

You can also use the “-tu” flag instead of “-t” that fetches templates from a URL which is more easier in case of CI/CD pipeline.

Automatic issue reporting using Nuclei scan results

Once Nuclei has completed its scan, it is important to report any identified issues and notify the relevant parties. This can be done using a tool like GitLab Issues, which allows you to create and track issues within your repository.
First of all, we will need to create a report configuration file to feed nuclei for automatic issue reporting.

In the above report.yaml file, change the username, token, project-name with your values. Next, we will convert this file (report.yaml) to a base64 string so that we can set it as a variable that can easily be called in our CI/CD pipeline.

Step: Converting report.yaml to base64

Command: cat report.yaml | base64

Next, we will set up a GitLab CI/CD variable using the above base64 string. You can find documentation to setup a CI/CD variable here: https://docs.gitlab.com/ee/ci/variables/

Step: Setting up GitLab CI/CD variable

1. Navigate to your Project > Settings > CI/CD and in the Variables section, click on “Expand”.

2. Click on “Add Variable”. In the Key field add “GITLAB_REPORT” (without quotes) and in the Value field add the base64 string of the report.yaml file.

Now, we have done all prerequisite steps, next, let’s re-create our CI/CD configuration file to implement automatic issue reporting.

In the above workflow file, replace the code with the following:

Explanation of the code: In the above code, the script will read the CI/CD variable and re-create the report.yaml file which will further be fed to nuclei using “-rc (report config)” parameter. The rest script is similar to what we previously used but now it is more powerful as it will perform automatic issue creation.

Now, commit the new changes, and you should see it will start a new workflow process, and at the end, it will report the issues. The reported issues can be found in your project’s Issues tab.

Conclusion

Nuclei is an invaluable resource for anyone looking to ensure the security and stability of their web applications. Not only is it a powerful tool for identifying vulnerabilities, it is also open-source and easy to integrate into your GitLab CI/CD pipeline.

It's important to remember that once you have identified any issues with Nuclei, you should notify the relevant parties so that they can be properly addressed. This will help to protect your web applications and the sensitive information they contain. Happy hacking!