How do you balance security and agility in DevOps or cloud IT operations?

As DevOps and IT culture keep evolving, It is advisable to incorporate security practices early and ensure developers and operations are in sync, While doing so it will be of great advantage for both parties to automate security testing, foster a culture of security awareness, and responsibility across development and operations.

The "security first" mindset is difficult to maintain when most people's jobs are driven by OKRs that are business value driven. As such, the best implementation is to ensure your security engineers are staffed enough to work along side project teams from start to finish. Though they may only need to commit about 5% of their time to a given project, being available is critical. When security is faceless or a checkbox, the doers forget to check in until it's too late or expensive to change.

Rather than thinking about security toward the end of the application, the best strategy is to push security to the beginning left. Starting from container image creation following NIST 800-190 compliance standard.Incorporate SAST, into your CI/CD. Communicate with your security team to ensure compliances like CIS, DSS, NIST, SOC2, ISO-27001 are strictly implemented.Ensure to use monitoring and analytics tool, to determine performance metric as well as event monitoring for infrastructure and cloud services, tool that can monitor services such as server’s database and tools. runtime security tools that will continuously monitor your workloads, search for abnormal behavior.Again setting up incident response and forensics, is very important.

In security we talk about tiers of maturity, and this automated feedback loop is broadly associated with the final tier. Part of building to this level of maturity includes understanding how your systems work, what is "normal" and what is "abnormal". I highly recommend that organizations study the Google Site Reliability Engineer handbook which talks about building reliable systems and metrics. A second piece of advice is to review how the organization understands things work. Anytime the word "should" or "think" comes up, double click and change those words to "does" and "confirmed". Security often breaks in the assumptions and unknown.

In organizational culture, it's important to remember security's role is to enable the business. This requires that security be included on business discussions to understand the why behind requests that break security norms. It also means the security team needs to work with the business to work towards "yes" over a flat no. It's not the business' job to find a secure path to a required business end. I have seen organizations that promote a culture where security and the business fight and the last ideal standing wins. The core principal is that the idea with most merit will triumph and the business will select the best path. However, it rarely delivers as expected. I recommend deploying rogerian rhetoric as you build your culture.

I highly recommend instilling a "Growth Mindset" culture within your organization as described by Dr. Carol Dweck. At its core, the growth mindset is about viewing knowledge deficiencies as an opportunity instead of a threat. At scale, teams build their knowledge collectively and avoid harboring information for "job security" which creates a rich foundation to innovate and strive through challenges. Beyond culture, investing and promoting time for your teams to develop is important. When organizations over burden their teams with deliverables without considering professional development, growth will stagnate. Growth can only occur when intentional time is carved into our busy work schedules.

Using automation allows for security to be tested for throughout the entire development process. Test-Driven-Development fosters tiny-iterative commits to kick-off automated security pipelines at the speed of change. By allowing security testing to be done earlier in the development process, vulnerabilities are caught faster and less costly to resolve.

For a lot of organizations, to rightfully balance security and rapid delivery, restructuring needs to occur. The best implementations of cross-functional delivery has been in matrixed organizations that build delivery teams comprised of security, business, and delivery members. If your organization operates in silos with process gates, then any attempts to create a unified organization will be limited.

Threat Modeling: Conducting threat modeling exercises to identify potential security risks and design mitigations from the start. Compliance as Code: Defining compliance requirements as code and automate compliance checks to ensure that security and regulatory standards are met. Both the above practices if adopted early in DevOps lifecycle of software release cycle can help to maintain a balance between security and agility.