Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines
0 likes166 views
Talk presented at Kubernetes Community Day, New York, May 2024. Technical summary of Multi-Cluster Kubernetes Networking architectures with focus on 4 key topics. 1) Key patterns for Multi-cluster architectures 2) Architectural comparison of several OSS/ CNCF projects to address these patterns 3) Evolution trends for the APIs of these projects 4) Some design recommendations & guidelines for adopting/ deploying these solutions.
More Related Content
Similar to Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines (20)
More from Sanjeev Rampal (8)
Recently uploaded (15)
Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines
- 2. Kubernetes Multi-Cluster Networking - Are we there yet ? - Patterns, Projects & Guidelines Sanjeev Rampal - Red Hat
- 3. Agenda ● Problem Statement (Reminder) ● Multi-Cluster Solution Patterns ● Some relevant CNCF projects, APIs ● Recommendations ● Demo ● Q&A
- 4. About the speaker ● Senior Principal Engineer ● Emerging Technologies Group, Office of the CTO, Red Hat ● Many moons of experience building & architecting in Cloud, Networking, AI platform segments ● Worked up and down the stack (L1 to L7) @ Cisco, Red Hat ● (more than) once stayed at a Holiday Inn Express
- 5. Scope of This Talk ● What are we talking about ? ○ Application Networking (primarily containerized applications, also VMs) ○ Spread across multiple Kubernetes clusters (and VMs) ○ Networking aspects only ○ Primarily covering CNCF project based solutions ● What are we not talking about ? ○ Other aspects of Multi-cluster architectures beyond networking
- 6. 4 Types of questions we try to answer in this talk ● Patterns: ○ What are some key multi-cluster deployment design patterns ? ● Projects: ○ What are some of the relevant open source/ CNCF projects ○ How would we categorize and compare these architecturally ? ● Open source APIs evolution ○ How are the APIs of some of these key projects evolving and possibly aligning ? ● Guidelines and Recommendations ○ What are some deployment design recommendations ? ○ How to plan a multi-level strategy to adopt multi-cluster networking ?
- 7. (A few) Projects for other aspects of Multi-Cluster ● Cluster Fleet Management ○ Open Cluster Management ○ Rancher Fleet ○ Bespoke automation via Terraform, GitOps/ Argo, K8s ClusterAPI etc ○ Various Hosted & Public Cloud services ● Application/ Workload Life cycle mgmt across cluster fleet ○ Karmada ○ Open Cluster Management ○ Argo CD/ Workflows/ Rollouts ● Other Multi-cluster Operations (Observability etc) ○ Thanos + Prometheus ○ Clusterpedia ● Multi-Cluster Tenancy & CRD management, virtual clusters ○ KCP ○ vClusters ● And many more …
- 8. Use cases reminder: Why Multi-cluster ? ● Application HA & Disaster recovery, reduce unicluster blast radius ● Improved performance when serving clients across geographies ● Data gravity or compliance requirements ● Simplify infrastructure upgrades, rollouts ● Enabler for Multi-Cloud ● Re-use compute capacity from clusters across the enterprise However ● Several cost-benefit trade offs exist ● Multiple levels of multi-cluster adoption, which do you need ?
- 9. Some Key Architecture Pattern Decisions ● North-South (multi-cluster) networking ● East-West (cross-cluster) networking ● Single/ flat Network or Multi-network ● Single Mesh or Multi-mesh ● Single trust domain or multi-domain federation ● Operational needs: Admin vs Dev centric, namespace sameness ● Others: ○ Traffic types: http. tcp, udp, other protocols gRpc, mysql, ipv4, ipv6 address pools ○ Kubernetes endpoints + VMs/ non-K8s endpoints (+ serverless) ○ Multi-cluster Network policy, service discovery ○ Degree of Zero Trust support (Crypto workload identities, encryption options) ○ Commercial and community project health
- 10. Network 1 (private) Network 2 (private) External Network Cluster-1 Cluster-4 Cluster-2 Cluster-3 Ingress GW (Multi-Cluster) E-W GW Egress GW Egress GW E-W GW
- 11. Patterns: Multi-Cluster N-S/ GSLB ex. DNS based Cluster-1 Cluster-2 R B G ING R B G ING GC1 GC2 Ext. DNS e.g Rte 53 Client-1 DNS zone Delegation for foo.com foo.com -> 1.1.1.1 Egr External statefulset/ db GC: GSLB Ctrlr ING: Ingress
- 12. Options for “N-S only” GLB/ Multi-cluster Ingress Solution Architectures Example Analysis notes External Load Balancer Appliance OSS: Traefik, Envoy Commercial: F5, HAProxy, API GWs Appliance is data path performance bottleneck, SPOF needs its own HA, use if already in use by enterprise DNS based GSLB CNCF: k8gb Kuadrant,Gloo, Kong gw OSS, k8s native, fine-grained per-app failover but inconsistent failover performance, 2-clusters Public Cloud Anycast Service AWS Global Accel, GCP CLB/ Maglev, Cloudflare Coarse grained per IP failover, expensive, scale out data path high performance, fast failover, true global availability Custom BGP Anycast Custom via BGP, metallb Per IP failover, fast convergence, high perf, scale out, depends on BGP data center design Ref: Red Hat Blog: Global Load Balancer Approaches
- 13. N-S LB via Cloud Anycast LB service Cloud Provider Private Global Backbone Network App 1 K8s cluster A App 1 K8s cluster B Public Cloud Region US-East-1 Public Cloud Region EU-West-1 Cloud Global Load Balancer as a Service Edge proxy NYC Edge proxy Atlanta Edge proxy London 200.1.2.10 200.1.2.10 200.1.2.10 DNS: app.globaldomain.io -> 200.1.2.10
- 14. Multi-Cluster Patterns: Basic E-W Networking (with single network & namespace sameness) Cluster-1 Cluster-2 B1 G1 ING ING G2 B2 B4 G3 B3 Cluster-1 Details: Pod & service discovery across clusters ? Namespace sameness ? How does Policy work ? etc These depend on which specific project is used to implement this ns app-A
- 15. Patterns: E-W with Multi-Network Cluster-1 B1 G1 ING ING G2 B2 B4 G3 Network-1 GW-a GW-b Network-2 Additional topics: Load balancing across gateways. Service discovery, CIDRs for pods, services, hosts, gateways, Policy,
- 16. Pattern: Combine E-W + N-S to improve DNS failover Cluster-1 Cluster-2 B1 G1 ING ING G2 B2 B4 G3 B3 Cluster-1 Ext. DNS e.g Rte 53 GC1 GC2 Client-1 Client-2
- 17. CNCF OSS projects in Multicluster networking ● Lets categorize them into 4 buckets a. CNI+ projects b. Application networking/ service meshes c. Kubernetes native Multi-cluster networking API implementations d. User space application networking frameworks
- 18. CNI+ projects App Service Meshes K8s native MCS api implementation Virtual Application Networking Representative Projects Cilium Clustermesh Calico Istio, LinkerD, etc Submariner Skupper Single network Yes Yes Yes Yes Multi network No Yes Yes Yes K8s seamless Netpol in multi-cluster K8s like Cilium NetPol+Annotations No (Istio AuthZ) No (exp) No (separate authz) Traffic mgmt policy L3/ L4 + some L7 L4, L7 L3 L7 (fqdn) Zero Trust level Medium-High High Medium High Elevated privileges needed for some ops Yes Yes Yes No (Entirely in user space) Tied to a CNI Yes No No No Non-K8S support Yes (WIP) Yes (WIP) No Yes
- 19. Services vs Multi-Cluster Services in K8S & Istio ● Kubernetes: ○ A “Service” api instance is always scoped to a single cluster ○ For Multi-Cluster, need explicit new resources: Service, ServiceExport, ServiceImport ○ Also new concepts of ClusterSets and ClusterSetIps ○ Apps can’t move unchanged between single cluster and multi-cluster ● Istio ○ Implicitly treats a Service as a Multi-Cluster Service across all clusters in the same mesh (subject to namespace sameness) ● Cilium ○ No new resource but needs an annotation to tag a service as global/ multi-cluster apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute metadata: name: example-http-route namespace: default spec: parentRefs: - name: istio-gateway namespace: istio-system rules: - matches: - path: type: PathPrefix value: /example backendRefs: - name: service-A-B port: 80
- 20. So many APIs .. how do they relate/ align ? K8S Gateway API Istio APIs Cilium APIs “Ingress v2 api” GEP-1748 Some Multi-cluster support (k8s mcs api backends) GEP-1324, 1686 etc SMI/ GAMMA (Mesh mode) “Extensive Istio-native mesh & gateway APIs incl Multi-cluster” K8s GW api support GW api Gamma support Ambient mode K8S CNI, Service, Netpol APIs Cilium CRDs Multi-Cluster support K8s APIs + annotations K8s GW api support GW + mesh evolution + align w/ sig-mc, Sig-net (mcs, netpol) Evolution + GW api & k8s alignment Evolution + GW api & k8s alignment Also: K8s Service api, NetPol api, Multi-cluster service api ..
- 21. K8s Gateway API - Gateway and Mesh modes apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: example-route spec: parentRefs: - name: example-gateway hostnames: - "example.com" rules: - backendRefs: - name: example-svc port: 80 Gateway application apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: smiley-route namespace: faces spec: parentRefs: - name: smiley kind: Service group: core port: 80 rules: - backendRefs: - name: remote-svc Mesh application
- 22. Sample API - Cilium Multicluster apiVersion: v1 kind: Service metadata: name: blue annotations: service.cilium.io/global: "true" service.cilium.io/shared: "false" service.cilium.io/affinity: "local" spec: type: ClusterIP ports: - port: 80 selector: name: blue Multi-Cluster Service apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "allow-cross-cluster" spec: endpointSelector: matchLabels: name: red io.cilium.k8s.policy.cluster: cluster1 egress: - toEndpoints: - matchLabels: name: blue io.cilium.k8s.policy.cluster: cluster2 Multi-Cluster Network Policy
- 23. Guidelines and Recommendations ● Do start with a Multi-Cluster Kubernetes design plan ● Think holistically about Networking + other DevOps aspects (App deployment, Observability etc) ● Start simple and hold the line until you need complex ○ “No is temporary, Yes is Permanent” - Solomon Hykes, Docker creator. ○ But if you do need to go complex, don’t be shy, the tools exist ● Consider a multi-stage Multi-Cluster strategy (example on next slide)
- 24. Suggested Rules of Thumb 1. Prioritize N-S multi-cluster, you may not need E-W Multi-cluster ○ Many apps can work well via N-S Multi-cluster & namespace sameness only ○ Add E-W multi-cluster deployments only when necessary 2. If you need E-W Multi-Cluster, prioritize single network mode ○ Single/ flat network has fewer issues ○ Better for stateful workloads, policy, load balancing, performance ○ Requires large IP address pools, leverage VPCs and IPv6 if possible 3. For truly advanced and large scale or geo distributed deployments ○ Evaluate full Multi-network, multi-mesh, federated, global anycast etc 4. Evaluate the non-networking aspects at the same time ○ App orchestration, observability etc when moving to more advanced networking 5. Also match with your organizational model, your apps ○ Ex. Consider User-space options if needed
- 25. Guidelines - A possible multi-phase strategy ● Minimal inter-cluster dependency ● Basic N-S Load Balancing only ● Namespace sameness ● No E-W Multi-Cluster ● Basic N-S + Basic E-W networking & load balancing ● Single network (flat network) mode only ● Use all IPv6 addressing for pods, services if possible ● Simple Multi-mesh ● Multi-network mode if needed ● Remote clusters if needed ● Multi-trust-domain ● Multi-org federation ● Adv. design for high scale ● KVStore mesh ● Global Anycast Level 1 Level 2 Level 3 Note: These levels/ paths dont apply if using Skupper which runs in user space and is less coupled with infra and k8s
- 26. Demo 1: Make Level-1 (N-S only) mode more robust Cluster-1 Cluster-2 R B G ING R B G ING GC1 GC2 Ext. DNS e.g Rte 53 Client-1 Regular DNS GSLB doesnt handle all error scenarios. Lets fix that External state/ db
- 28. Conclusions ● Plan your phased Multi-Cluster roadmap ○ Start with simple (Level 1, maybe level 2), plan conservatively for more complex ● Many excellent CNCF projects, OSS, vendor, SaaS ● Understand low level data plane, control plane details across the available projects ○ There are subtle differences which impact function, performance ● So, are we there yet ? Yes (& it’s a journey, more to come)
- 29. Thank You! Feel free to contact me for any follow ups DM me at Twitter/ X: @sr2357 Connect/ DM on Linkedin Longer version of this talk to be uploaded on Youtube Cloud-Clarity @cloud-clarity4190