note: these are speaking notes that guided me through the speech, not a transcription. They’re also full of typos and grammar errors.

Third party Risk - a vendor perspective

Good Morning colleagues from across North America

First, I wanted to thank you for the opportunity to meet with you and I wanted to thank C-Vision and their partners for supporting this event. As it is still not possible us to get together in person it makes virtual conferences like this even more important to us as a community.

Today I wanted to share some thoughts on third parties and the risks as well as the benefits they present for us. I believe that managing third party risk must be a core competency for security teams because our businesses depend more and more on third parties. In my opinion, there is no clearer alignment for security with business value than third party risk management. Given the importance of third party risk management, it is essential that we do it well, and to do third party risk management well, and at scale, we need to move beyond the questionnaire and find more holistic ways to engage with vendors and service providers that are a fundamental part of our digital supply chain.

…but first I’d like to tell you a story.

About seven years ago I got to fulfill a lifelong dream, I got to lead security, privacy and compliance at a start-up. Not only was it a Canadian based B2B start up but it was one that could afford to pay a decent salary and hire a properly resourced security team. Then came the customers and their questions, they of course wanted to know how we would protect their data on our cloud platform. The questions usually came in the form of vendor assessment questionnaires; they came in every flavour from industry standardized Shared Assessments to a whole rainbow of excel based creations and even some horrible to use web-based assessment forms. Every form was different in structure, format and content; but they all were driving at the same goal… some sort of comfort that they should trust us to process their data. At it’s height we were processing a vendor assessment questionnaire every single day. It was the opposite of fun as we tried to place our responses to hundreds of questions into another and then another form, always different, always requiring tweaking. We were spending resources on dealing with forms and not towards actual security; of course it was a business requirement, part of selling to the corporate market, a cost of doing business.

We came up with smart ideas to address the workload. We licensed Shared Assessment’s SIG and completed that and we offered it to our customers instead of their form. A few agreed, but most wanted their questionnaire filled in anyways. We built a library of pre-canned responses and tried to teach sales engineers how to answer the forms; that worked a little but had limited utility and took the sales engineers away from actual selling. We decided to invest in a SOC2 and a quarter million dollars later learned that our customers were very happy that we had a SOC2 but still wanted us to fill in the form anyways. With the unrelenting pressure to sell, combined with the unending flow of security questionnaires our answers by necessity became short and simple.

The results of many customers trying to do the right thing to protect their data had the unintended consequence of degrading the overall quality of responses as we got shorter and shorter with our written replies.

When I started in that role I wanted to be open with our customers. I thought that we could have meaningful conversations about risk and discuss the likelihood of certain outcomes. I envisioned a meeting of the minds that would enable security professionals to address the fears of businesspeople and navigate towards mutually beneficial outcomes. I thought that fellow security people would understand the challenges of standing up a security program and would appreciate our transparency and recognize where we were going; I thought they would acknowledge security is a journey and be happy to do business with an honest partner. I quickly learned I was wrong and naïve. My transparency delayed several deals and caused quite a few frustrations for a company trying hard to make sales targets. I don’t fault the customer’s security people… they had a job to do, to make sure that their data was safe on day one. They, whoever was assessing us, also weren’t decision makers, they were usually junior practitioners there to follow a specific process and report back. And so, after making a few mistakes and getting some gentle coaching from sales colleagues we adopted the approach of always tell truth but keep your answers tight. It was our job to answer the question but it was the customers job to ask the right questions.

Again, well intentioned vendor security assessment processes led to a lower quality security outcome overall.

During my years at that organization, we lost only lost three deals: two because of regulatory issues we couldn’t address and one because they wanted a type of security technology that didn’t make sense for us to deploy. As a former vendor security person that leads me to believe either our business value was so compelling that it overrode any concerns, or that our responses were just putting a check mark in a compliance box; I’ll leave you to draw your own conclusions.

Now I don’t think my experience is unique; having been buyer side for a few years now, I think most vendor security teams will have a similar experience when it comes to supporting selling. They exist to instill customer confidence, to navigate the compliance process and hopefully to also establish good security. If the focus of the company is to make sales, then everyone in the company aligns on that goal.

So, here’s the a question for all of us: are we making it easy for our vendors to do the right thing? Are we as an industry running processes that make it hard for vendors? Are we unintentionally encouraging vendors to engage in behaviours that degrade overall responses? Do we engage in partnerships or do we engage in adversarial relationships?

I think it’s important we understand why third parties are important to businesses. The short answer is that most businesses are focused on their core competencies; the insurance company I work for wants to help protect the health and economic future of 12 million Canadians. What we don’t want to be is the best Microsoft Exchange operator so we outsource that to Microsoft. Their core competency is running Exchange. We also don’t develop software, we buy software or we integrate and configure off the shelf products; we understand our customers very well and we understand their needs and how to translate that into business processes that provide them benefit while producing good economic outcomes for our stakeholders. But no matter how good we are at understanding our customer doesn’t translate into an ability to build great insurance software. We benefit from outsourcing software development either in part or in its entirety by buying software. In every aspect of our business, we benefit from outside expertise so that we can focus on our core competency of servicing our customers. Beyond a strong focus on core competencies, there is a need for innovation. Some organizations, like tech companies, are geared for innovation, that’s their competency, but for most organization over a certain size, innovation is something to be acquired. Many businesses rely on innovation coming from outside their organization, from smaller organizations who have the ability to risk it all on a bright idea and make it real. Because innovation occurs at the edge, as our businesses go digital, we rely on third parties to provide us that innovation. Even small companies rely on other companies for innovation in areas they’re not proficient at. If the ability to focus on competencies and rely on third parties for the rest is a competitive advantage, then is there some risk-reward equation that optimizes on some level of security in exchange for better economic outcomes? Business is about taking some risk to produce some amount of return; we might design a new market offering at some expense and not get much return… a risk that didn’t pay off… or it be wildly successful… a risk that produced a reward. What if some third party provided the key technology that made the difference between success and failure? Is there some equation that says we’re willing to take a risk with a particular technology vendor, ofcourse within certain limits of reasonableness, in exchange for outcomes. Or… is it absolute? As a profession are we enabling our businesses or are we unintentionally stifling them? Are we protecting them from well intentioned but wrong decisions or are we taking the easy path in saying no instead of searching for middle ground and compromise? As business focuses on their core competencies they become more and more dependent on a complex web of providers to sustain their business. Consider for a second car manufacturers and their complex supply chains; the same tightly woven interdependency is becoming true for many organizations that rely on technology to operate. As businesses focus on their core competencies their need for innovation at the edge, from outside their corporate boundaries increases. This means that the ability to assess and manage vendor risks is a core competency for security teams and if it isn’t yet it will be in the coming years as more and more technology companies unbundle business processes to extract value by offering specific solutions for specific element of a business process. In short, we are likely to see a great proliferation of business technology vendors that our business colleagues will want to use. We’ll almost certainly see the same for security technology. Security professionals will be a key part of enabling the vendor ecosystem.

Consider the dozens or hundreds or thousands of business processes in your organization; how many today are powered by a dedicated on-premises application? How many today are powered by an excel spreadsheet with complex macros shared by several users? How many are manual email workflows? Eventually a third-party solution will come along for each of those processes and the business will want to buy it.

My next question that I ask you to think about is how do we keep up with that demand? More importantly how do we keep up with that demand in a manner that preserves quality and gives us a proper understanding of risk? Firing out forms faster, scanning with machine learning, using scorecards based on observable data signals or outsourcing to third parties are likely all good measures but ultimately don’t yet get at the core problem of true risk understanding at scale.

It is important that we consider the core of vendor assessments… not the penetration test or the vulnerability scan but the questionnaire. Many jobs ago, when negotiating a supplier agreement, a lawyer I worked with told me that small companies will agree to anything and do nothing because they want the business and large companies will agree to nothing but probably do what is reasonably expected of them. I think that translates into questionnaires as well; small companies will give you enough of answer to get through the process and probably elide on as much as they can. It may not even be intentional… small companies may not truly understand the intention of the question being asked them.

I think there is a very real need to treat our smaller vendors not only as suppliers but also partners; I suggest to you that the future of third party risk management is one of partnership, one of benevolence where larger organizations as buyers educate smaller companies because it is to their mutual benefit. Education both at the time of assessment but also beyond that as the relationship continues.

That ongoing education is important because assessments are only point in time exercises. We can most likely expect that a large vendor will continue to do what they said they would; large organizations have an inertia to them, they have lawyers and compliance people and their commitments to security are embedded as a matter of good corporate citizenship and prudent business. However, small organizations are more likely to change and so it is important that we continue on in a journey with them so that as they change, they have the knowledge they need to meet our needs.

That, I think is an opportunity for the security profession, for this community… whether through consortiums or industry groups, to educate and nurture smaller organizations on their security journey. Outreach and partnership may be one of the ways we produce a scalable process for actively managing security risk.

Our next big challenge is broader than third party risk alone, it is the digital supply chain which is made up of third parties, fourth parties and off the shelf technology providers. It’s also made up of businesses that appear to be pure service organizations but have a lot of technology under the covers.

In the last few months, the challenges of securing the digital supply chain have been highlighted through two unique events:

The first is the solarwinds hack from December 2020. I suspect several of us in the audience had a challenging few days as we assessed the risk and took appropriate action. With perfect hindsight I ask how many of us assessed SolarWinds as something more than a commercial product? Perhaps we asked some questions about their software development practices? How many of us asked about the security of their software update process? Even if you did ask I don’t think you would have received anything other than a glowing response as evidenced by SolarWinds breathy blog posts in the prior year about their secure software development. I suspect many security professionals would have treated a piece of on premise software as nothing more than just that; a piece of software that was likely low risk for some given it didn’t process data… yet in hindsight it had access to an awful lot of important data.

I think to many vendor assessment processes, the idea that the build pipeline could be compromised wasn’t contemplated. That software could be compromised is not a new idea; Russia compromised a significant Ukrainian tax software vendor to spread NOTPETYA in 2017. Every few years some threat actor compromises a well deployed piece of software as a means to access their trusted user base; it’s a useful tactic and one that takes advantage not only of the unsuspecting user base who trusts the vendor but possibly even takes advantage of the good security practices to keep your software updated.

There are many possible responses on how to buffer against these sorts of attacks and I could speak for hours, as could many of you, on the different approaches to secure the software development processes. It’s highly unlikely that code scanning would have caught solarwinds trojan code; never mind that it was embedded as part of the compilation process (and not in the original code) but the code would have looked much like the rest of the solarwinds orion codebase… a module that scanned the network and did network things… just like the host software it parasitically latched on. Configuration management and stronger access control would probably have helped as would have reproducible build environments in which build environments are dynamically spun up from known secure configurations to build the software alongside the permanent build environment and the outputs are compared. I would propose that not only do you want software developers to build secure software but to build it securely. However, this keynote is about managing third party risk, so I suppose we could augment our security questionnaires to ask questions about how vendors who sell us off the shelf software not only write secure code but also provide assurance of their software pipeline. Chances are that many software development teams have heard of secure development such as OWASP but how many know of or have gone to the effort of establishing security assurance around their development pipelines?

Would we buy software from a vendor that doesn’t use reproducible build environments? What if they had never heard of the ideas? Could we educate them so that they build that assurance over the software pipeline?

What if the vendor didn’t see this as important? Perhaps they see themselves as just a software vendor; they sell you access to a blob of compiled bits; to them it’s quite transactional but to us they’re a key part of our digital supply chain. If they are compromised, we are compromised. Our education of vendors not only needs to be about what to do, but also why to do it. To actively manage our digital supply chain we need to convince our vendors that they are a part of it; that their obligations to us go well beyond the transactional delivery of compiled software or the maintenance agreements. We need them to believe that because anything less means we will get a partially answered questionnaires and surface level attestation reports.

The second big challenge in the digital supply chain takes us down a bit of a rabbit hole… software libraries. Whether our own internally developed software, software powered cloud platforms or off the shelf software they all have one thing in common. Software is built up from modules that provide various utilities; developers add the unique business logic and glue processes to connect the code together.

As a hobbyist developer, my python code usually has several software modules imported. The same is true for my Javascript code and when I refer to the modules I import I’m referring to the top most modules that I want to consume but those modules often have dozen or hundreds of underlying dependencies. From a security perspective, it’s probably turtles all the way down when it comes trying to secure the code from malicious modification.

In 1983, Ken Thompson delivered an important talk. Ken Thompson design and implemented the original Unix at Bell Labs. His seminal talk – “Reflections on Trusting Trust”, was delivered as a Turing Award Lecture. His idea, back then, was that software code, libraries and compilers could all be backdoored – he coined that idea – it’s known as the Thompson hack or the trusting trust attack. While the technical elegance of his talk is to be admired, what’s more important is the recognition that developers will trust code and compilers from sources that they can’t verify.

Back then, public repositories of code modules didn’t exist. Today there are millions of third-party libraries across dozens of popular languages. These repositories are not app stores, the modules are not curated. Anyone can upload a module; in some cases modules are abandoned by their maintainers and taken over by other maintainers. Sometimes those new maintainers have malicious intent; case in point a javascript module in 2018 was modified to steal bitcoin private keys when the module was included in a specific payment platform.

While it’s possible to run your own internal corporate repositories and only permit reviewed modules in, that’s massive overhead and likely beyond most development teams given the significant number of modules one would have to inspect. Such activity would likely impeded development velocity, although arguably with good reason. Prohibiting outside code entirely is likely to be overly expensive and possibly have adverse effects of increasing the likelihood of code flaws as developers implement their own functionality instead of using well tested opensource code.

For now, at least the best answer seems to be private module repositories with well controlled processes for how modules are onboarded but that is likely a challenge for many orgs who are in pursuit of fast software delivery to the businesses they support.

However even that’s not a bullet proof solution, if you can afford to do it, as one security researcher recently compromised every major tech company by identifying a flaw in the way development tools download modules. The security researcher identified a particular bug that caused the development tools to download his malicious code from a public repository instead of the safe modules inside the private repository. But I suspect this particular bug will get fixed in future development tools, so hopefully it’s a passing problem and not a permanent strike against internal private module repositories.

So this is the last big challenge I ask that you think about. The digital supply chain is not only outside our organizations, it is inside our own organizations. We have development processes that are dedicated to bringing in third party code into our environments; third party code that has been backdoored. That same problem exists for our vendors. I think this problem is going to be very hard to solve because we will depend on not only secure develop practices within our own organization but also secure development practices for every developer of every module we consume. We will not only depend on secure development practices but also assurance around the development practices themselves, again both within our organization and for every outside module maintainer we depend on.

With that I thank you for listening; as you go about the rest of the conference think about how we can partner with vendors through education and outreach to help them understand that they are a part of our supply chain. To help them understand what good practices look like. I ask that you think about how we can build a truly scalable third party risk ecosystem that goes beyond the questionnaire alone to proactively manage risk through mutually beneficial engagement, through partnership. I hope I provided you with something to think about and some ideas to augment your third party risk management. I’ll post a copy of this speech at thoughts.sapiro.net if you care to read it and would love to chat via LinkedIn if you’d like to drop me a line. I hope you get great value out of today – the presenters and panelists are discussing fundamentally important topics that I’m looking forward to hearing.