With the release of vSphere Kubernetes Service 3.6.0 supporting Kubernetes v1.35, VMware has introduced significant architectural changes to how cluster add-ons are configured and deployed. The new Addon Framework replaces the legacy package-based approach, bringing a more streamlined and declarative method for managing cluster extensions like Antrea CNI. In this post, I’ll walk through deploying the Antrea AddonConfig with NSX integration and EgressSeparateSubnet enabled—a configuration that has proven reliable in my VCF environment.
Understanding the Addon Framework Shift
Previous versions of VKS relied on the Tanzu Package Management system, where you would apply an AntreaConfig manifest to enable features. With VKS 3.6.0 and later, this has been replaced by the AddonConfig resource. This new approach provides better lifecycle management, improved version compatibility, and tighter integration with the cluster provisioning process.
Pre-requisites
Before proceeding, ensure the following components are in place:
- VMware Cloud Foundation 9.0.1 or later with a configured vSphere Supervisor
- NSX Management Proxy installed and operational (if enabling NSX integration)
- vSphere namespace created for VKS deployment
- VCF CLI context configured for the target namespace
The AddonConfig Manifest
Here’s the AddonConfig that I’ve settled on after extensive testing. This configuration enables both NSX integration through the Antrea-NSX adapter and the EgressSeparateSubnet feature gate, which allows dedicated egress subnets for workload traffic:
# vks-01-antrea-addonconfig.yaml
apiVersion: addons.kubernetes.vmware.com/v1alpha1
kind: AddonConfig
metadata:
name: vks-01-antrea
namespace: wkld-03-proj-01
spec:
values:
antrea:
config:
featureGates:
EgressSeparateSubnet: true
antreaNSX:
enable: trueKey Configuration Elements
Let’s break down what each section accomplishes:
EgressSeparateSubnet Feature Gate: This enables Antrea to use a dedicated subnet for egress traffic from your Kubernetes workloads. When enabled, you can configure separate networking for outbound connections, providing better traffic isolation and simplified network policy management. This is particularly useful when integrating with NSX-T and wanting granular control over how pod traffic exits the cluster.
antreaNSX Integration: Setting this to true enables the Antrea-NSX adapter, which allows your VKS cluster to appear as a Container Cluster in NSX Manager. This unlocks powerful capabilities like applying NSX Distributed Firewall rules based on Kubernetes labels, providing micro-segmentation at the pod level.
Deployment Workflow
The critical aspect of this approach is timing. The AddonConfig must be applied before deploying your VKS cluster. This ensures that when the cluster is provisioned, Antrea is configured with the correct settings from the start.
First, set your VCF context to target the correct vSphere namespace:
$ vcf context use admin:wkld-03-proj-01Apply the AddonConfig manifest:
$ kubectl apply -f vks-01-antrea-addonconfig.yaml
addonconfig.addons.kubernetes.vmware.com/vks-01-antrea createdVerify the AddonConfig was created successfully:
$ kubectl get addonconfig -n wkld-03-proj-01
NAME AGE
vks-01-antrea 15sDeploy the VKS Cluster
With the AddonConfig in place, you can now deploy your VKS cluster. Here’s the complete cluster manifest I’m using:
# vks-01-cluster.yaml
apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
name: vks-01
namespace: wkld-03-proj-01
spec:
clusterNetwork:
pods:
cidrBlocks:
- 192.168.156.0/20
services:
cidrBlocks:
- 10.96.0.0/12
serviceDomain: cluster.local
topology:
# Updated from 'class' to 'classRef' per v1beta2 schema
classRef:
name: builtin-generic-v3.6.0
namespace: vmware-system-vks-public
# v1.35.0---vmware.2-vkr.4
# v1.34.1---vmware.1-vkr.4
# v1.33.6---vmware.1-fips-vkr.4
version: v1.35.0---vmware.2-vkr.4
variables:
- name: vmClass
value: best-effort-small
- name: storageClass
value: super-01-storage-latebinding
# Indented under topology
controlPlane:
replicas: 1
metadata:
annotations:
run.tanzu.vmware.com/resolve-os-image: os-name=photon, content-library=cl-54dd7ecc3feccd58c
# Indented under topology
workers:
machineDeployments:
- class: node-pool
name: node-pool-01
replicas: 1Apply the cluster manifest and monitor the deployment:
$ kubectl apply -f vks-01-cluster.yaml
cluster.cluster.x-k8s.io/vks-01 created
$ kubectl get cluster vks-01 -n wkld-03-proj-01 -w
NAME CLUSTERCLASS AVAILABLE CP DESIRED CP AVAILABLE VERSION
vks-01 builtin-generic-v3.6.0 True 1 1 v1.34.1+vmware.1-vkr.4Verification Steps
Once the cluster reaches Available: True status, create a VCF context for the workload cluster to verify that Antrea is running with the correct configuration:
$ vcf context create --endpoint 192.168.27.2 --username jschulz@quadroolabs.com \
--insecure-skip-tls-verify --workload-cluster-name vks-01 \
--workload-cluster-namespace wkld-03-proj-01 --auth-type basic
? Provide a name for the context: vks-01
Provide Password:
Context created successfully.Switch to the newly created context and check the Antrea pods:
$ vcf context use vks-01:vks-01
$ kubectl get pods -n kube-system | grep antrea
antrea-agent-xxxxx 2/2 Running 0 5m
antrea-controller-yyyyy 1/1 Running 0 5mIf NSX integration is enabled, verify the cluster appears in NSX Manager under System → Fabric → Nodes → Container Clusters. You should see an entry for your VKS cluster with status indicating successful registration.
Benefits of This Approach
This configuration provides several advantages over the default setup:
Simplified Egress Management: With EgressSeparateSubnet enabled, you can configure dedicated IP ranges for outbound traffic, making it easier to implement network policies and firewall rules at the infrastructure level.
Label-Based Security: NSX integration allows you to apply Distributed Firewall rules based on Kubernetes labels, enabling zero-trust networking that follows your applications regardless of where pods are scheduled.
Declarative Configuration: The AddonConfig approach is more aligned with GitOps practices, allowing you to version control your cluster add-on configurations alongside your cluster definitions.
Lessons Learned
Through testing, I’ve found that this pre-deployment configuration method works significantly better than attempting to retrofit these settings after cluster creation. The Addon Framework expects configurations to be in place before provisioning, and trying to modify Antrea settings post-deployment can lead to inconsistent states.
Each VKS cluster requires its own AddonConfig—there’s currently no mechanism for global default configurations across all clusters in a namespace. While this may seem repetitive, it provides granular control over each cluster’s networking stack, which is valuable when managing multiple environments with different requirements.
Looking Forward
The Addon Framework represents a significant improvement in how VKS manages cluster extensions. By applying the AddonConfig before cluster deployment, you ensure consistent networking configurations that integrate seamlessly with your NSX infrastructure. This approach has proven reliable across multiple cluster deployments in my lab environment and provides a solid foundation for production workloads. In future posts, I’ll explore advanced EgressSeparateSubnet configurations and demonstrate practical DFW rule implementations for Kubernetes workloads.
Special thanks to Claude (Anthropic) for assistance with content development and technical writing.
Cloud Management | Jon Schulz 
Leave a comment