Kubernetes on EGO : Bringing enterprise resource management and scheduling to Kubernetes

Kubernetes on EGO : Bringing enterprise
resource management and scheduling to
Kubernetes
Da Ma (madaxa@cn.ibm.com)
Software Architect, IBM
Owner of kube-incubator/kube-mesos-framework
Yong Feng (yongfeng@ca.ibm.com)
Senior Software Architect, IBM

Why “Kubernetes on EGO”?
Computing, Storage and Network
Application Application
Container Runtime
The container runtime packages and launches application
instance in a sandbox with portable and flexible capability.
Docker and rkt are container runtimes.
Workload Management
The workload management component
manages the life cycle of an application as
well as access to the application, including
service compose, service discovery, load
balance
Kubernetes and Marathon are workload
managers.
Resource Management
The resource management component
provides an abstraction of resources (cpu,
mem, …) for application and then
allocates/provision resources among tenants
and applications.
Mesos is an open source resource
manager.
EGO is an IBM enterprise resource
manager.

Why Kubernetes on EGO?
1992
PBS/SGE/LSF
Res mgr & wld mgr
tightly coupled
Batch wld only
Monolithic
2003 2016 future
??
Shared state
between fws by
Optimistic Offer
Shared state
Two Level
Scheduling
Mesos/YARN/EGO
……

Architecture Overview
EGO Master
VEMKD
MLIM
BASE API
LIM PEM
EGO Agent
LIM PEM
EGO Agent
LIM PEM
EGO Agent
UDP SocketTCP Socket
PLUGIN
k8s-apiserver
k8s-controller-manager
k8se-scheduler
kubelet
kube-proxy
resreq alloc
1. Get Pods
2. Send resource request to EGO
3. Get allocations from EGO
4. Bind Pods with Host
5. Run Pods by kubelet
1
2 3
4
5

EGO: Enterprise Resource Manager
• Hierarchical consumer
• Enterprise sharing policies
• Smart preemption
• Rich resource attributes
and resource requirement
language
• Unified management console
• Security
• Monitor and alert
• HA and multiple site
• Resource usage analysis

EGO: Hierarchical Consumer
Dept-1
ORG
Dept-2
Dept-n
Team-1
Team-2
Proj-1
Proj-2
Hierarchical resource
budget plan
Hierarchical role
based access control

EGO: Enterprise Sharing Policies
Time-windows based resource
plan per resource group
Ownership and one-to-one
lending/borrow policy

Dynamic sharing from top
down to leaf consumer
Hybrid sharing polices
o At T0, A has a demand of 20
A = 20
o At T1, B1 has a demand of 20
and reclaims its parent’s 16
A:B1 = 4:16
o At T2, A cancels all workload
and becomes idle
B1 = 20
o At T3, B2 has a demand of 20
thus reclaims its 12
B1:B2=8:12
/
B1
B
S=1
S=3S=1
S=4
(A=4)
(B1=4) (B2=12)
20 slots in total
A
B2
Hybrid Ownership Share ratio
Sharing by default X x
Reserve slots from being
shared
X X
Plan configured by
absolute number
X X
Sibling first borrowing X x
balance checking X X
Proportional borrowing X x
Proportional reclaiming X x

Flexible framework of
scheduling plugin for
customized sharing policies

EGO: Smart Preemption
• Asynchronized resource negotiation protocol
Issue resource request via allocation which allows client to orchestrate
multiple services from different tenants; update resource request on the
fly; receive resource allocation by event;
• Grace period in resource plan
Contract between resource lender and borrower used to decide how
resources will be returned if required
• Candidate resource list
Allows the borrower to optimize when making decisions on which
resources to return within grace period

EGO: Rich Variety of Resource Attributes and
Resource Requirement Language
• Various types of resource attributes and ways to define and collect them
Static vs dynamic; integer vs Boolean vs string vs ip vs topology; user
defined vs collected by script
• Resource requirement language
select(), order(), affinity(), antiaffinity(), rusage() …

We’re Contributor
to the community !!!

Mesos: OpenSource Resource Management
• Hierarchy Role (MESOS-6375)
• Multi-Role Frameworks (MESOS-1763)
• Scheduling (Pending)
oResource Revocation
oOversubscription for reservation
oQuota Chunks
o…..

Kubernetes on Mesos
• Sponsor: Tim Hockin (Google)
• Champion: David Eads (Redhat)
• Owner: Klaus Ma (IBM)
• Github: kuberntes-incubator/kube-mesos-framework

Kubernetes on Mesos (kube-mesos-framework)
1. Get Pods
2. Match Pods and Offers
3. Bind Pods with Host
4. Update Pods status
5. Run Pods by kubelet

IBM Spectrum
Conductor for
Container
Spectrum
Conductor with
Spark
Watson /
Cognitive
Container Cloud
Session
Scheduler
Workflow
Installer
(Deploy,
Reconfigure,
HA, Scale,
Rolling
update)
Mesos Agent
K8s executor
pod pod pod container container
containercontainer
Mesos Master
Kubernetes
GUI
Service
Discovery
Authentication
Authorization
Distributed
Key-value
Store
Image
registry
Monitor
HPC
App Store
Persistent
Volume
Service Load
Balance
Trouble-
shooting
Network
Topology

Community Value IBM Value-add Customer Value
Docker Hub Registry holds a repository of 75000+ Docker
images
Lots of application integrated with Mesos
Kubernetes enable micro-service architecture
• Client unique registry available on premises
• Security readiness guidance via the Vulnerability Advisor
• Build-in applications of popular open source projects and IBM enterprise
products in App Store
Access to the images and application you require to
deploy containers that meet your business needs
and strategy
Open-source, standardized, lightweight, self sufficient container
technology
• Balance workload between on-prem and off-prem
• Deployment choice with openPOWER and x86_64
Flexibility to choose on-prem and off-prem or mix
for your business
Build, ship, and run standardized containers
• Integrated monitoring & logging
• Elasticity to grow storage & container needs
• Integrated CI/CD flow
• Life-cycle management of containers and data volumes
Docker ease of use combined with enterprise-level
integrity and confidence

Create a Container Cloud for developers
supporting DevOps practices and cloud-native
apps. Pre-built app catalog for fast deployment of
OSS tools. Reduce developer friction, creating
faster time to results
1
Improve Developer Productivity
Fine grain, dynamic allocation of resources
maximizes efficiency of Spark instances sharing
a common resource pool.
2
Increase Resource Utilization
Proven architecture at extreme scale, with
enterprise class workload management,
monitoring, reporting, and security capabilities.
3
Reduce Administration Costs

Mesos
Kubernetes
(role = *, bigdata-daemons)
Myriad
Slaves
(weight)
Spark
Slaves
(weight)
App Area (label: app) BigData Area (label: bigdata)
role = bigdata-daemon: Reserve resources for HDFS and Yarn/Spark master
role = bigdata-comute: Reserve resource for Yarn/Spark agents
Spark Session
Scheduler
Myriad Masters
Dep 1 Dep 2 Dep 3
ns1
+
quota1
ns2
+
quota2
ns3
+
quota3
Container service
role = *
BigData Service & Applications
(role = bigdata-comute)
Resource
Sharing
Hierarchy
Consumer
Smart preemption
&
Sharing policies
NS/Quota
Network/DNS
Scheduling
Dream ???
Resource
Requirement
Spark with
kube-mesos

What’s next?
• Support Sharing Policies & Smart Preemption:
Revocable resources support (#19529)
Scheduling enhancement (# 31068)
• Support Hierarchical Consumer:
Namespace/Quota support/integrate (#31069)
Multiple roles support
• Kube-DNS integrate with external DNS (# 28453)
• …

Roadmap of kube-mesos-framework (DRAFT)
Nov, 2016 End of 2016 2017
v0.7 release
new code base
v0.9 release
new features
v0.8 release
k8sm refactor v1.0 release
Production Ready

Kubernetes on EGO : Bringing enterprise resource management and scheduling to Kubernetes

Recommended

More Related Content

What's hot (20)

Similar to Kubernetes on EGO : Bringing enterprise resource management and scheduling to Kubernetes (20)

More from Yong Feng (6)

Recently uploaded (20)

Kubernetes on EGO : Bringing enterprise resource management and scheduling to Kubernetes

Editor's Notes