In March 2021 a few of Appvia’s senior engineers were invited to take part in Microsoft’s three-day One Commercial Partner (OCP) Hackathon, working directly alongside Microsoft engineers to ideate and work through a potential approach to implementing least privilege on Microsoft Azure.
The Principle of Least Privilege (sometimes referred to as just ‘Least Privilege’ or PoLP) is a security principle established with the idea that users and systems have only the necessary privileges required to perform what they need to do, for just the amount of time needed to do them. It’s crucial to address least privilege when working with creating Cloud resources programmatically (i.e. provisioning) as not limiting the scope for a program to provision unintended resources (or access) is a clear risk to a business.
We're embedding this principle throughout Wayfinder, making sure that we scope all our access to customer cloud accounts and cloud infrastructure around specific purposes with well-defined access requirements.
All of the main cloud providers have the capability to constrain Cloud access using “Managed Cloud Credentials”. These are often time-limited and are always tied down to specific programs running on computers in that Cloud. Each provider has a different name for this capability but we needed to uncover how this works within Azure and, more specifically, within Azure Kubernetes Service.
The other capability often used to constrain access is short lived role assumption which can be used to provide specific (i.e. fine-grained) and short lived access on demand. Azure doesn’t support short-lived role assumption for programmatic access, so we were left with a predicament: How do we align with industry-wide least privilege best practices, while also using Microsoft’s best practices for access to Azure?
The crux of the challenge was finding a way to use the Cloud Managed Credentials of our Kubernetes hosted program with the right level of access at run time. Without specific role based access, it’s too easy to have ill-defined or, even worse, over-privileged access (i.e. access that's ripe for exploitation—accidental or otherwise).
Below is an illustration of the problem with a program running in AKS —let’s call it ‘ms-hack’.
In this example, our Cloud Managed Access unfortunately has access to create more than it should (i.e.. Cloud Keys)!
We understood our problem from the get-go, but we needed a plan to work through it. The hackathon was structured around a three-day process, during which we had constant access to Microsoft solutions engineers who have the expertise in our chosen hack subject.
The first day was all about working through a proof of concept. Right out of the gates, Microsoft paired us with a handful of different solutions engineers from various teams, each bringing a different perspective and a host of fresh ideas.
By putting our heads together, we identified that it was possible to use multiple Azure Active Directory Pod-managed Identity bindings for a single Pod. Multiple Pod identities aren’t possible with all other cloud providers, and it was this sole insight that led us to a solid approach to achieve least privilege cloud provisioning in Wayfinder.
We didn’t know at this point if this approach was possible to achieve the access we needed, but we fleshed out what it could potentially look like (demonstrated below).
The remainder of Day 1 was dedicated to creating a proof-of-concept: Could a simple ‘Go’ application switch between multiple identities on-demand when running in an Azure Kubernetes cluster?
This leveraged work that had already been done in Wayfinder which enables Azure Managed Pod Identity. By the end of the day, we’d tested and proved that our concept could indeed work.
With the concept proven, Day 2 was spent bringing the feature to life within the platform, building out the Azure specifics within our cloud access libraries using the principals we’d previously tested out.
By the end of Day 3, we were able to demonstrate a working prototype in which Wayfinder was successfully switching between identities, each holding the minimum privilege needed to successfully orchestrate a new Azure Kubernetes cluster. When testing was finished, we presented our findings to the Hackathon.
Below is a screenshot of the working prototype of Wayfinder configured so that it would use Azure Managed Identities (aka Cloud Managed Identity):
Here's another screenshot, showcasing the prototype provisioning an AKS cluster and its network dependencies using two separate Pod identities:
Going from idea to proof-of-concept and, finally, a working prototype just in three days was a huge feat, made possible with the great support from Microsoft’s engineers. They provided rapid feedback on our ideas and suggested novel approaches that were not previously considered.
If you want to try out our hack code, check out the GitHub repository.
.png)
