Chris Wysopal is Chief Technology Officer at Veracode (ACSC Member). He oversees technology strategy and information security. Prior to co-founding Veracode in 2006, Chris was vice president of research and development at security consultancy @stake, which was acquired by Symantec.
For an insider’s take on the evolving areas of DevOps and Application Security, the Advanced Cyber Security Center (ACSC) spoke with Chris Wysopal, co-founder of Veracode, which was recently acquired by CA Technologies. The ACSC sat down with Chris to learn more about the intersection of devops and security, how the emergence of the cloud affects application security, and what’s next for Veracode. Read on for highlights and advice from one of Boston’s biggest cybersecurity startup success stories.
Bringing DevOps Into Security Operations
Q&A - ACSC: You have deep roots in the security industry. Can you tell us how software development has changed since you began at Veracode, and what it says about the challenge of securing software?
Q&A - Chris Wysopal: We started Veracode in 2006, so it’s been 11 years — software development is constantly changing. My first job was as a developer in 1987 for Lotus, it already had changed dramatically from those days to 2006 when we started working with software companies.
It’s interesting that over those almost 20 years, you can learn things and apply them today to the way security and development is changing.
When I first started as a developer, Quality Control (QC) was a different team. Programmers were separate, each with a separate management chain. Back then, it was build the app and throw it over the wall and then the Quality Assurance (QA) team tests it. It was not integrated at all, so I began building automated testing tools. In 2006, it was so manual. Now, QA is fully integrated into Development - they write their own unit tests. A lot of times, there isn’t even someone with a QA title, the developers are just required to write their own tests for their own code. It’s a team responsibility, there’s no one else who’s going to fix it. Now, we’re starting to see that same shift happen with security, as they become part of the development team. Security as an audit team is antiquated thinking. The mega-trend I’ve seen over my career is moving away from separate teams doing all the different little pieces of the software development process, and it’s all being compressed into one team. This makes it so much more efficient - the handoffs, systems, process, workflows, and management interaction all goes away when you have a single team with a single data repository. We can now build software faster and more cheaply.
“Security as an audit team is antiquated thinking.”
Do you get higher quality software and fewer bugs this way, when the people who understand the software are also the ones checking it?
The big efficiency is finding things earlier in the lifecycle - it’s about moving as far left as possible. The automation tools help find issues within minutes of the code being written. It eliminates inefficiencies like task switching and storing data and then having to go back to it. It’s like spell check for software. Security needs to move there just like everything else. If we don’t, then security concerns will essentially get left out. It’s not just about cheaper and faster. If security is going to happen at all, it has to be integrated.
“It’s not just about cheaper and faster. If security is going to happen at all, it has to be integrated.”
This is really exciting for us in the software security field, because we’ve always wanted the developers to engage in security, we always thought it would be better if developers were educated and responsible for securing their own code. Now we’re getting what we’ve wanted not just because it’s a good idea but because it’s required. Organizations that have compliance requirements (Financial Services, Healthcare, etc.) need to demonstrate that they are proactive and preventive about building software. So, they have to do security, and they have to do it in an integrated process. Otherwise they will be left behind by competition. They have to move fast with software development to stay competitive. The combination of compliance requirements and market forces is getting them to integrate security into DevOps. The appetite by customers for faster, faster, faster is insatiable. It used to be they wanted results in days, now they want it in an hour, and next they will want it in minutes.
Aligning DevOps to the Organization
You are currently CTO at Veracode, which provides a software platform that aligns security with DevOps. What are some of the challenges with aligning these groups?
The Sec team needs to not micromanage the process anymore. They have to become comfortable not running every test and result, take more of a governance role, and do things like setting up policies and then enabling the developers run the process themselves. Basically, they need to teach developers to fish. The security team is going to have to give up some control. This is a challenge because to some degree, security and developers are opposed. Security wants to find every last problem and deal with it, developers want to find fewer problems, just focus on what really needs to be fixed. Security teams will say there is a lot of risk here, and the software teams will say, “I don’t understand why it’s so risky, I want to ship my software.” There has to be some compromise. The job of the security person should be to articulate and quantify the risk, and the business owner (the funder of the development) needs to calibrate how much risk they’re willing to take. So the parties need to work together to get to this point.
If security teams need to be more strategic, how do we move there? Does it force a skillset change for the security teams?
First, shift all automated work to the developers. Maybe different skill sets are needed, so train security teams for those. There is some security we haven’t figured out how to automate yet, such as threat modeling and design reviews, and those still need to be manual and performed by the security team. Hopefully over time, they work with the development teams and train them along the way.
One great way to embed security into the development team, which we actually do here at Veracode, is to have a security champion. Their day job is developer but they’re interested in security. So you provide these champions with extra love and care such as training, monthly knowledge sharing, capture the flag exercises and bring them into the security side of the culture. Then they become aware enough that they can engage the security team when needed.
The classic example is when product management wants to make a password reset design, the security champion reviews it during the development and flags it as needing a security design review and they bring in the security team to help. This allows security to scale without having a huge organization with a centralized team, because you have pockets of expertise spread out across the whole development organization. This approach takes strong management buy-in, including the head of development and the VP of engineering, they need to all buy into this approach. The move to agile development has really set the precedent for this integration. So, it’s expanding on the concept already in place that they’re familiar with.
Is DevOps security more preventive or does it also come into play during an attack?
It does come into play during an attack. The feedback loop from Ops back into Development about how the software is performing is already in place. So the idea of instrumenting your production systems so you can know where the bottlenecks are, what’s using the most memory, what’s using the most CPU, network, where are the inefficiencies and latency. It’s a single team using the same data to make the software better. If you take that to security, the development team should be instrumenting the application so that it can be understood from a security perspective. It’s both emerging technologies that run at run time that look at the application, like Interactive Application Security Testing (IAST) and Runtime Application Self-Protection (RASP). The development team can also build in the functionality to see security events/logging while it’s in production and be able to put that out to Splunk or to a Security Information and Event Management (SIEM) system, and get the visibility of what’s happening in real time to make the software better. All that data can be used during a breach to detect and respond quickly.
“...the development team should be instrumenting the application so it can be understood from a security perspective.”
In your view, does DevOps become the internal frontline security defense team in an organization, or do DevOps and Security team up to fend off threats?
It’s evolving as to how they all work together - there doesn’t seem to be a good standard or framework for this currently. There has to be really good communication and relationships between the information technology (IT) security team and the development team. The security team needs to know what information the developers have at their disposal, so doing the up-front work on what’s there will make it faster to respond when something has gone wrong. It’s a tough one.
As we move towards custom applications, with 10 microservices running in the cloud, where is the IT security team in that equation? The whole notion that the infrastructure is disappearing and it’s all just virtualized makes it difficult to use all those traditional security technologies like firewalls and intrusion detection systems (IDS). There’s a new cloud native awareness. That means security is obscured, which makes it even harder. It was a challenge to know what was going on when you had a segmented corporate network and your own hosts. Now, there’s this virtualized environment. There’s definitely challenges between development in building software these days and the way IT normally functions. There’s still lots of dumb mistakes happening because people just don’t understand the cloud environment.
“There’s still lots of dumb mistakes happening because people just don’t understand the cloud environment.”
How the Cloud Has Changed Application Security
Has the growth of cloud computing had an impact on application security, and if so, what have you seen?
CA is looking at what security’s going to be over the next 10 years, questions like, ‘is there even going to be a host level, a host security or network security business?’ Those are huge businesses, but if you look 10 years down the road, isn’t that just going to all be done by your cloud provider? Isn’t that all going to go away, even though it is huge today? For example, at the furthest out extreme we see today, AWS is running their “Lambda code” serverless applications where you upload your code and you set a trigger for it to run, and it just runs. There’s no server, no container, no virtual machine, it’s just the application. They take care of everything else. It’s a new way of thinking about this stuff. People are building microservices with Lambda, we have some customers doing that.
We see all of our customers moving to the cloud. It’s extremely rare that customers says “we’re never moving to the cloud.” Even big, old school companies are doing it, and many have already done it. It’s really an inevitability, so security has to embrace application security then. I think it’s a great idea, there’s so many old, antiquated systems out there that are so hard to maintain, and where security problems come from. The older your infrastructure is, the worse your security is.
Do the threats and attacks just move to the cloud then?
The architecture of the cloud gives so many more control points for security. It just allows for so many fewer vulnerabilities. It’s being built with security in mind vs. legacy tools where it was an afterthought, so it would make sense that’s the case. When the cloud was being built, in order to convince companies to move to there, cloud service providers had to convince companies that the cloud was secure. One of the weakest links is the enterprise desktop, where there’s malware, vulnerabilities and patching. Controlling that and keeping that up to date is very difficult. So organizations are moving to virtualized desktops, all the files are stored in cloud storage.
Working Outside of Veracode
You mentioned CA. Veracode recently was acquired by CA Technologies. How do you see the technologies and strategies of these companies fitting together? Any changes you anticipate as part of the acquisition?
CA, as a large company, is made up of many different business units (BUs), such as the continuous deployment BU, and of course the security BU, which is all identity and access management (IAM). When we merged with CA, the thought was, “why don’t we make a security BU that is both IAM and application security?” But actually, we have more in common with the continuous deployment business unit than the security BU, which is really great, because we are more tied in with development than a separate team where they do security. So for now, we are our own separate Veracode business unit, which enables us to work closely with the continuous deployment BU. Within a week, they had our API’s integrated with that BU. There are lots of opportunities for us to be integrated with the other software development tools that CA has. There’s so many different places already for us to integrate: they have testing tools that create sample test data so not using production data to test, they have an API gateway product. There’s a lot of products that don’t have any security aspect to them that we can inject some security into. We’re not just a revenue stream to CA, we actually have technology to integrate with the other products to make them better. We’re excited about all the ways we can make 1+1=3. It’s very exciting.
As a member of the ACSC, Veracode participates in activities such as Cyber Tuesdays for front line defenders and the Executive Forums. What value do you see in security information sharing and participation in organizations like the ACSC?
As a small company, we don’t have much visibility into what’s going on out there. Because of our tiny footprint, it’s very helpful for our team to see the threats. That was the original reason we joined. The Executive Forums and networking interactions with other sophisticated organizations has been very valuable as well. We educate others in our area of expertise and we get to learn what else is out there. All the organizations in the ACSC are really sophisticated, so I find there’s a lot more value in that networking than a typical CISO event or forum.