The Dark Reality of the Shift Left Movement: How Security and Developers Are Struggling to Keep Up

Authored by Ivan Milenkovic, Vice President Risk Technology EMEA, Qualys

Over the past decade, there has been a prevailing belief that if developers could take on more responsibility for security early in the development process, the digital realm would become safer, faster, and more cost-effective. However, the conflict between speed and security has only escalated, leading to a fundamental issue.

The primary reason for this failure lies in the immense pressure placed on developers. The traditional project management triangle of Fast, Good, Cheap has been shattered, with businesses now demanding all four aspects – fast, good, cheap, and secure. In this scenario, speed usually takes precedence, resulting in an overload of tasks for developers.

When developers turn to public container images to expedite development, they inadvertently expose themselves to potential risks. Understanding the root cause of this problem and finding solutions is crucial.

Business Priorities vs. Security Measures

A common misconception in the security industry is labeling developers as lazy or negligent. In reality, developers are dedicated professionals facing overwhelming workloads and responding to the incentives presented to them. If a security scan obstructs a build process that is time-sensitive, developers are likely to bypass the scan to meet deadlines.

Businesses’ increasing demand for rapid results has led to a perception that security protocols impede productivity rather than enhance it. Noisy, slow, and disjointed security tools are often viewed as obstacles in the workflow.

Consequently, organizations have relinquished control over their operational environments. Automated pipelines deploy code autonomously, infrastructure scales without human intervention, and AI agents generate and execute scripts independently.

In this fast-paced, automated environment, public registries are often treated as reliable libraries, assuming that images from Docker Hub, for instance, are secure. However, pulling a container from a public registry requires a trust decision.

Major tech players like Docker, Amazon, Google, and Microsoft operate public container registries, leading to a mistaken trust assumption. By the time a container image reaches the deployment pipeline, it is already considered a trusted component integrated into the application.

Unveiling the Reality of 34,000 Images

The Qualys Threat Research Unit (TRU) recently conducted an extensive analysis of over 34,000 container images sourced from public repositories to unveil the underlying truths.

Among the total images analyzed, approximately 7.3 percent (2,500 images) were identified as malicious, with 70 percent of these containing cryptomining software.

Furthermore, 42 percent of the images harbored more than five secrets capable of granting unauthorized access to other resources or accounts. These secrets encompassed valuable items like AWS access keys, GitHub API tokens, and database credentials embedded within the image layers.

The primary issues surrounding malicious containers often boil down to simple factors. Typosquatting emerges as a prevalent method for attackers to disseminate malicious containers. While advising developers to double-check spellings is crucial, it represents a minimal effort response to a critical problem.

Relying on developers to exercise caution is not a foolproof security strategy. Although public registries offer convenience, developers should ideally avoid pulling from them altogether.

In a mature setting, every external image should pass through an internal artifact repository acting as a quarantine zone. While the need for speed persists, efforts should focus on enabling developers to progress swiftly while upholding security standards.

Shifting Focus

The prevailing notion is that rectifying a bug during the design or coding phase is more cost-effective than addressing it in production. Consequently, incorporating security measures early in the Software Development Life Cycle (SDLC) should mitigate risks in later stages. However, this approach burdens developers with scanning their code, verifying dependencies, and managing infrastructure.

In reality, this shift merely transfers the burden to developers, who are predominantly evaluated based on feature delivery speed. The intended collaborative nature of “shift left” security measures often translates into added responsibilities for developers without enhancing security practices.

To rectify this, security should be ingrained within the infrastructure by default rather than as an afterthought. This necessitates genuine collaboration between developers and security personnel, ensuring that development aligns with security requirements for a secure delivery.

Implementing a “golden path” for developers can streamline the process. By adhering to standard templates, pre-approved base images, and official CI pipelines, developers can seamlessly integrate security measures. Deviating from these norms to create custom solutions would entail additional security reviews and manual configurations.

This information should be communicated to the business from the outset, fostering a united front between security and development regarding associated costs.

Emphasizing secure deployment as the default route incentivizes developers to prioritize security effortlessly. This approach shifts responsibility to the infrastructure layer managed by a specialized Platform Engineering team. Custom requirements can be collaboratively addressed to ensure error-free solutions from the outset.

For instance, instead of tasking a developer with enabling versioning on a specific S3 bucket, the platform team could establish a policy using Terraform modules, Crossplane compositions, or Open Policy Agent that mandates versioning for all buckets. This proactive approach minimizes room for error.

Automating security measures is also crucial. Upon detecting a vulnerability in a base image, the platform should automatically generate a Pull Request to update it. Similarly, if a runtime security tool identifies a container exhibiting suspicious behavior, such as spawning a shell for persistence, the tool should isolate the container promptly.

Instead of adhering to conventional practices across security and development, organizations must adapt to evolving circumstances. This may entail a fundamental shift in inter-team operations.

Persisting with the “shift left” approach of overloading developers with cognitive tasks is unsustainable. It risks burnout and circumvention of security controls to meet business demands.

Hence, a proactive stance on implementing and supporting suitable platforms for secure operations is imperative, ensuring automatic security measures.

Presented by Qualys.