Enforcing security policies across multiple Kubernetes clusters often leads to challenges with admission policies as API objects. This update addresses these issues, streamlining policy management for improved security and operational efficiency.
Untangling Kubernetes Security Policies
Managing security across multiple Kubernetes clusters can feel like navigating a minefield. A common challenge emerges from the way Kubernetes handles admission policies, which are stored as API objects. These policies can be inadvertently—or intentionally—deleted by users with sufficient privileges, leaving your security posture exposed during crucial moments, like when the cluster is starting up.
Fortunately, Kubernetes v1.36 steps in with an alpha feature aimed at mitigating this issue: **manifest-based admission control**. This allows you to predefine admission webhooks and implement CEL-based policies in files on disk. What's crucial here is that these configurations are loaded by the API server at startup, ensuring they are in place before any requests are processed.
A Glaring Gap
Most current implementations of Kubernetes policy enforcement can only operate through the API. This means that every ValidatingAdmissionPolicy or webhook configuration must first be created as an API object and recognized by the admission controller. While this process works acceptably well once everything is running smoothly, it reveals glaring weaknesses.
There exists a precarious gap during the bootstrap phase—specifically between the moment the API server begins handling requests and when your policies become active. This window can expand significantly if you're recovering from a backup or addressing an etcd failure, putting your security measures at risk when you can least afford it.
Moreover, there's an inherent problem with self-protection. Admission policies are sidelined when it comes to safeguarding their own configurations. Kubernetes is careful to avoid circular dependencies by skipping webhook invocation on types like ValidatingWebhookConfiguration. Consequently, a well-privileged user can remove critical admission policies without encountering any barriers within the admission chain.
That’s where the Kubernetes SIG API Machinery decided intervention was essential. Their goal? To ensure that certain policies remain active at all times—no exceptions.
The Mechanics of Manifest-Based Admission Control
To implement this new approach, you'll need to incorporate a `staticManifestsDir` option into your `AdmissionConfiguration` file, which you typically pass to the API server using the `--admission-control-config-file` flag. Simply point this field to a directory, place your policy YAML files there, and the API server will load them before it starts processing any requests.
It’s important to note that these manifest files must adhere to the standard Kubernetes resource definitions. The critical requirement is that every object must have names that end with `.static.k8s.io`. This naming convention avoids collisions with API-based configurations and simplifies tracking the source of admission decisions when you review metrics or audit logs.
For example, consider a policy crafted to deny any attempt to deploy privileged containers outside the `kube-system` namespace:
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: "deny-privileged.static.k8s.io"
annotations:
kubernetes.io/description: "Deny launching privileged pods, anywhere this policy is applied"
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["pods"]
variables:
- name: allContainers
expression: >-
object.spec.containers +
(has(object.spec.initContainers) ? object.spec.initContainers : []) +
(has(object.spec.ephemeralContainers) ? object.spec.ephemeralContainers : [])
validations:
- expression: >-
!variables.allContainers.exists(c,
has(c.securityContext) && has(c.securityContext.privileged) &&
c.securityContext.privileged == true)
message: "Privileged containers are not allowed"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: "deny-privileged-binding.static.k8s.io"
annotations:
kubernetes.io/description: "Bind deny-privileged policy to all namespaces except kube-system"
spec:
policyName: "deny-privileged.static.k8s.io"
validationActions:
- Deny
matchResources:
namespaceSelector:
matchExpressions:
- key: "kubernetes.io/metadata.name"
operator: NotIn
values: ["kube-system"]
New Layers of Protection
One of the standout features of this manifest-based approach is the ability to enforce rules on admission configuration resources themselves, which was previously impossible. With the API-centric method, webhooks and policies never act on types like ValidatingAdmissionPolicy or ValidatingWebhookConfiguration. This is a necessary safeguard: allowing such actions could lock you out, making it difficult to rectify mistakes via the API.
Manifest-based policies circumvent this pitfall entirely. If a problematic policy is hindering operations, simply edit the manifest file, and the change will be recognized by the API server immediately—no reliance on the API for recovery is needed.
This enhancement paves the way for creating policies that can, for example, block the deletion of critical API-based admission policies. For platform teams managing shared infrastructure, this is a transformative leap. They can now ensure their foundational security policies remain intact, regardless of actions taken by individual cluster administrators who might not have the best interests of security in mind.
In practice, you could enact a policy that prohibits any modifications or deletions of admission resources tagged with the label `platform.example.com/protected: "true"`. This is not just a technical improvement; it represents a cultural shift in how Kubernetes handles security, providing peace of mind to administrators managing sensitive workloads.### Final Thoughts on Kubernetes Manifest-Based Policies
As we wrap up this exploration of Kubernetes' new manifest-based audit policies, it's apparent that we are looking at a significant shift in how organizations can manage admission controls more securely and flexibly. The ability to safeguard API-based admission resources with immutable policies marks a notable advancement in operational integrity. Yes, the system is designed to prevent unauthorized changes, but what's even more crucial is that it ties everything to the file system, ensuring policies are hard-coded at a foundational level.
However, you'll want to be cautious about the inherent limitations of these configurations. The restrictions on referencing external resources and the lack of cross-server synchronization might not be apparent at first glance, but they could lead to complications in larger or more dynamic environments. For those working across multiple clusters or departments, this could create a risk of inconsistency that demands robust monitoring and management practices.
Here's the takeaway: while the new implementation is powerful, it’s also somewhat brittle. Organizations should prepare for the stricter rules at startup—if anything in your manifest is out of sorts, the entire API server won’t launch. This is a double-edged sword, emphasizing the importance of due diligence but also potentially stalling critical deployments.
Looking ahead, Kubernetes is empowering teams to operate more efficiently without the need for constant API interaction. The automatic validation checks during runtime and the ability to update policies without server restarts translate to a more agile infrastructure. Imagine rolling out policy changes across your fleet effortlessly, without the fear of downtime. That capability is here, but organizations must be methodical in their approach.
If you’re eager to experiment with these features, start with Kubernetes v1.36. The setup process appears straightforward, but take the time to familiarize yourself with the documentation and the community resources available. Engaging through channels like the Kubernetes Slack will not only enhance your understanding but might also pave the way for significant contributions to this pivotal area of Kubernetes technology.
In summary, as you explore the potential of manifest-based admission policies, remember: while they offer a promising leap forward in security and simplicity, they also require a sharpened focus on governance and maintenance. Happy experimenting, and I’m keen to see how these innovations evolve in upcoming releases.
