Building cloud stack at scale

Editor's Notes

• #2: End of day So we’ll start gentlyHopefully something for everyonePlease bear with me if seems obvious to you.Bear with my english accent and phrases/terminologyBasics and more advanced ideasPaul AngusEngineering and Science degreesCloudStack 2.13
• #3: Design Phase – some tips from experienceRedundancy & resilience – again some thoughts on building redundancy and resilience into the infrastructureAutomation – some examples of automation in the building of a cs architecture.Finanally a ‘word’ on documentation.
• #5: Storage, networking, overall architects technical and managerialEverything is interconnected and feels like everything relies on everything elseSomeone chipping in can be invaluable – particularly if they have past experience.
• #6: One that gets everyone:Switchsupports 4096 VLANs - but not at the same time.Gotchas…[add more]
• #7: Private clouds – you have your current usage to judgePublic clouds harder to predict – is guided by offerings.
• #8: Performance and / or capacityStorageNetworkNetwork (to storage) is often the limiting factor as jump to 10 Gbe is large(Although LACP in XenServer 6.1 and ESXi 5.1 will help to mitigate this)Not much point being able to run VMs per host if only 1Gb/s link Not much point to 256GB RAM with a single quad core processor unless a specific workload.
• #9: A few words on designing the infrastructure to maximise uptime.
• #10: I’ll look at the major CloudStack management elements – CSMan MySQLAnd then look at considerations if you’ve virtualised your management farm
• #11: CSMan internally, worth having a second management server as a ‘hot spare’Otherwise you’re going to need to load balance your connections to them.
• #12: Master / Slave is the ‘supported’ configuration, manual switchover not idealbut in a opensource environment anything goes.Some alternatives..
• #13: Example of active/active load balanced elements
• #14: Really cool setup – trader media group >autotraderThey can suffer the loss of an entire datacenterUserightscale to burst to amazon.
• #15: As well as what you have; ‘where’ you have it is importantRedundant pairs – DNS ServersWant anti-affinity (two MySQL servers)Want Affinity CSMan and MySQL Master
• #16: Similar to RAID 0In similar 2 hosts – if split the double the probability of loss of systemWLB / DRS rules or switch off WLB/DRS
• #18: Look at:Advantages of automationTools – from behemoth infrastructures to the simplest of toolsHost deployment, configuring management vms and configuration of cloudstack+ a couple of odds and ends
• #19: How do we achieve that…>
• #20: Aim to remove as much human error as possibleSame time speed deployment up.Running a script is also quicker than typing and far more repeatableKick a ‘load’ of scripts off at the same time
• #21: Automation can come in multiple formsChef & Puppet – enterprise grade automation – works for in-house use (required infrastructure makes it less useful for SIs)KickStart and Python – enables to learn one language and stick to it, requires the interpreter to be installed, but Python is Shell scripts don’t have to be fancy – have to learn awk & sed – have to learn the different languages.API calls
• #22: Simple example using BASH scripting.A management farm with a reasonable level of redundancy can easily have more than a dozen servers.The particular installation that this example is based on was a multi-tiered network with three interfaces on each vm plus static routes, but one of the networks I worked on had 7 tiers and used proxy servers to reach the internet-a lot of typing and to configure it all
• #23: We actually tend to combine these steps and create a VM template with these baked in.But essentially this is all we’d do
• #24: We have a single CSV with all the networking information.The actual one this is based on had 3 interfaces in each VM
• #25: Script itself:Ask for the hostname thenRead values from the csv usinggrep and awk based on hostname
• #26: Write theifcfg-ethx files, ntp and network files.
• #27: In this case there were routes which changed depending on which network the ‘secondary’ interface was connected to...takes out human error
• #30: Req: static IP and fdqnPure management server (no nfs or kvm)Separate scripts to add nfs and kvm(thanks to Wido who built the repo)
• #31: Again BASH scripting to build the server, however the configuring of hypervisors requires other scripting languagesInckickstart for ESXi
• #32: Use a management VM created by previous scriptRepeat with new script to configure the server
• #33: Write configuration file. Conscious decision to limit the number of files required.Self contained (requires hypervisor installation files)Look at some elements of the file >
• #34: After yum install of DHCP, suselinux and httpSee that script writes files rather than importing / downloading themAdds complexity in script because of escape characters
• #35: This section writes the default file for pxe bootingThis section is just shows the ESXi option XenServer or XCP as wellCould have a different script for each host, but then need to generate a new file for each host to tie it to its mac address.Simpler to add a line in the final CSV
• #36: Xenserver answer fileNote escape characters for quotes, but variables come from earlier in full script
• #37: CSV file for hosts
• #38: Note escape characters \\Weird stuff because of Xen XE command syntaxESXi has the ESXCli and vicfg commands
• #39: Xen updates are usually painful
• #45: In word or excel click on link in documentationImagine a spread sheet of the required storage with the final command built at the endWe don’t tend to fully automate this as ‘press-and-go’ because we want to keep an eye on what’s actually happeningThrough the GUI itself
• #46: Cheat for finding out what the cloudstackgui is actually up to
• #47: See the call in the to the API and the resulting responseCan we used to ‘see how the gui does it’
• #48: Example of global settings(still need to restart the management server)
• #49: Can retrieve information using the guiOtherwise only available through database
• #50: Example of deploying a virtual machine
• #51: Paste into browser and keep pressing refreshSpin up 150 hosts to stress test an environment
• #53: Take your pickOnly way to remember what you did,only way for others to replicateRun scripts from scratch impossible to update code and separately make changes