.conf 2015 - Splunking Distributed Logs for IT Policy Alignment
0 likes132 views
Indiana University implemented a centralized Splunk deployment to provide log management services across the university's decentralized IT environment. They developed a scalable multi-tenant architecture using deployment servers, forwarders, and a search head cluster. To onboard new customers, they created a script that automatically configures forwarders, indexes data by department, and applies the necessary configurations. Over time, they expanded their services by refining their apps, training customers, and increasing the number of departments and servers supported. Their goal is to further develop apps to help audit compliance and expand Splunk's use for additional IT systems across the university.
More Related Content
Viewers also liked (16)
Similar to .conf 2015 - Splunking Distributed Logs for IT Policy Alignment (20)
Recently uploaded (20)
.conf 2015 - Splunking Distributed Logs for IT Policy Alignment
- 1. Indiana University: Splunking Distributed Logs for IT Policy Alignment
- 2. Allen Tucker Manager, HELPnet Central Systems Team Kelly Zimmerman Systems Administrator, HELPnet Central Systems Team Daniel Daily Systems Administrator, HELPnet Central Systems Team
- 3. Disclaimer During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.
- 4. • Background and Culture • Infrastructure • Scalable Architecture • Multi-Tenant onboarding • Phases • Expansion of service • Q&A
- 6. Indiana University, est. 1820 • $3.3B enterprise • Partnered with $6B IU Health system • 115,000 Students • 1.3M Credit Hours per semester • >20,000 Degrees per year • $1.1B in Financial Aid • $450M in research grants • 8,000 Acres • 882 buildings, 36M square feet • >600,000 living Alumni • 10,500 Faculty and Staff
- 7. CENTRALIZED enterprise I.T. with DECENTRALIZED departmental I.T. 109 Departmental IT Groups 5213 Total Servers within IU
- 8. Safeguards IU I.T. Policy – IT-12 list of ‘best practices’ for system management IU Internal Audits – In depth departmental checks for IT operations – Alignment with IT policies Log management in IT-12 – Success/Failed User Logons, Success/Failed File Accesses
- 9. Implications Costs associated with log review – Its overwhelming Different log sources, many servers TONs of logs. – Costly if departments DIY – Staff time is at a premium Admins can make much better use of your time being impactful to their departments
- 10. Service Timeline
- 11. Internal HELPnet Deployment • 120 Servers Initial Departmental Offering Departmental Growth & UITS Interest •20 Departments •Many Regionals •~375 servers Issues with Scalability New Product Testing & PoC Proposal to Cabinet •Approval in August 2014 Deployment •Training •Certification •Build of Architecture •App Development Customer Onboarding (Since June) •42 Departments •2000 Active Servers
- 12. Infrastructure
- 13. 100% Virtual
- 14. Indiana University Storage and Virtualization Team 2014 VMware Innovation Award Winners
- 15. Physical Hardware Dell PowerEdge M620 - Intel Xeon E5-2690 Processors - 20 CPU cores @ 3GHz 512GB DDR3 RAM VMware ESX Hosts Hitachi VSP G1000 SAN Storage
- 16. Bloomington Indianapolis Multi-campus Deployer Deployment License Master Forwarders Department A Forwarders Department B Forwarders Department C Indexing Search Cluster
- 17. Performance Statistics IOPS > 1062 via Bonnie++ ~270,000 events per second dense search in smart mode ~5400 events per second sparse search in smart mode
- 18. Bloomington Indianapolis Multi-campus Deployer Deployment License Master Forwarders Department A Forwarders Department B Forwarders Department C Indexing Search Cluster
- 21. Search Head Cluster Use of a deployer Knowledge object replication Ease of additional search head expansion Deployer Search Cluster
- 22. Indexing Deployment Server Houses all important Splunk configurations – Indexer configurations – Configuration push to 2000+ servers (For a team that never touches the GUI) Deployment Forwarders
- 24. Deployer Deployment License Master Forwarders Department A Forwarders Department B Forwarders Department C Indexing Search Cluster
- 25. Data Silos by Tenant Each Department Has a Unique Index Department A Department B Department C
- 27. Simplify Onboarding Using an Onboard Script to clone our apps • Checks user logon • Asks for location, OS type (user input phase) • Adds server classes and indexes • Copies source application template • Replaces department variables • Assigns roles (authorize.conf, authentication.conf) • Assigns a default_namespace
- 28. Overcoming Distributed IT Administration NO ADMIN RIGHTS – Solution: Each department gets a shared Box folder Splunk Universal Forwarder Installation scripts How to docs MS ConfigMan (Windows) or Ansible (Linux) helps a great deal The magic of deploymentclient.conf
- 29. Script Details • Executes the .msi silently / stops the splunk service Defines location, OS, Department & Host –injects them into the deploymentclient.conf file - Assigns the deployment server Starts the service so it can phone home Verifies if Splunk is installed Also configures Deploymentclient.conf Modifies auditd.conf for splunk access Sets ownership and permissions Configures splunk to start on boot Differences • Dropping all unnecessary event IDs • Ingesting full auditd
- 30. Script to deploy to many One script to install them all Mass deployment of forwarder to 600+ servers remotely.
- 31. App Dev Phases
- 32. VERSION 1
- 33. VERSION 1
- 34. Training I2 Training credits* Optimizing Searches Data Inputs Reporting Lessons Learned
- 35. VERSION 2
- 36. VERSION 2
- 38. Down the Road Further development IT-12 application for departments – Devices not checking in – Alerting for failed user attempts – Filter for network access vs file access – More Linux distros Expand UITS I.T. Ops. – Kuali – VMware – Active Directory – IIS / Apache for IU Central Web Services – Multiple Security Apps
- 39. CAS & Shibboleth
- 42. Questions?
Editor's Notes
- #3: How many in attendance are in Higher Education?
- #7: Large organization: $3.3B enterprise 115,000 students 10,500 faculty and staff. 8 campuses 882 separate buildings over 8000 acres Given the scale, what does the I.T. landscape look like?
- #8: Central enterprise I.T. organization called UITS that handles management of core services Active Directory, Exchange, Networks, Wireless, Web Hosting, etc. Departmental I.T. providing hands on support directly to individual departments. 109 Separate IT groups Total servers = 5213 servers (UITS and Departmental) Obviously this means that IU has a large cyber security attack surface. Because of this, IU has implemented some safeguards to combat these issues.
- #9: IT12 – Best Practices Admin rights, network security, firewalling, as well as log management There are also Internal Audits that take place. Very public results (deans, directors, CIO, President, Board of Trustees) unfortunately, a consistent finding is log management.
- #10: Manual log review is very costly Time consuming Costly to build a utility to automate it Staff time is at a premium where they could be more helpful by direct support and innovation in their direct department. Departments - “If we have to review our logs because of IT policy, UITS should provide enterprise utility to make it easier” - We set out to provide a clear cut time saving utility that directly aligns with IT policy in order to wipe the slate clean on log review being an issue.
- #12: 3 years ago, HELPnet brought hardware log management online for internal use to meet IT12 requirements Started offering it at a small scale to other departments. Proposal approved in August by CIO Brad Wheeler to hopefully reduce these findings from Internal Audit.
- #15: Indiana University has a world class Storage and Virtualization team - vmware innovation award. - pushing the boundries of what virtualization can do
- #16: Indiana University has a world class Storage and Virtualization team that has earned vmware innovation awards. Multi-tier storage (low tier)
- #17: This is our initial infrastructure based on the initial service offering and based off of our current deployment needs Discuss search heads / Indexers Discuss locations
- #18: Explain what IOPS are and what Bonnie ++ is
- #19: With dozens and dozens of configuration files and the ability to expand with virtualized hardware, what does Splunk offer to help manage these servers?
- #21: So I am going to talk about the things we did to make our admin of splunk easier How we: Save Countless hours of staff time (primary example being an onboarding script) Reduce the administration of end points (no need to touch every server or forwarder in the environment) Quickly recover servers and stand up new servers --be it an additional SH or IDXR Take advantage of this extra time? --More hands on time with departments and development of apps ( spend time getting data into Splunk) esp non it12 data --Training our users to be power users So what features did we decided to introduce? first up our search head cluster- next slide
- #22: The star of your cluster is the deployer Single point of administration It’s the best place to back up all your configurations To deploys all your apps & configurations to the cluster It’s the job scheduler ….Brains! KO replication Joe power user…. (Senario) (Gotcha: Local configurations) Magic of this…? File precedence … GUI configurations... plan the role of your deployer carefully Scripting a new search head member gets a lot easier when all you have to do is spin up a new VM and assign it to the cluster (and the LB) and the let the deployer push base line configurations. Stuff replicates it’s a thing of beauty. Don’t confuse this cluster technology with MS clusters… Next up Deployment server….next slide
- #23: Deployment server is optimal BUT its a must It is a centralized administration point like that of the cluster deployer, however it pushes configs to indexers, and universal forwarders. Without it you would have to administer the configurations of your splunk components one system at a time. No one has time for that! -Adding indexers- The real beauty is the management of forwarders. Say you have a forwarder already on a system….you can increase/decrease the amount of data coming to the indexers by just editing the settings on the deployment server, once the forwarder checks in it picks up the changes.
- #24: So what happens when an IU department comes to us to start using Splunk? Most are seeking basic alignment services for IT12 policy alignment Some are seeking additional splunk apps and data collection…next slide
- #25: I want to revisit this architecture slide to discuss departments at IU. We have over 50 and any of them could be potentially sending different types of data not just server security logs for it12) and they don’t want to see each others data. To facilitate this we siloed data into indexes. WHY? Next slide
- #26: SECURITY SECURITY SECURITY Role Based Administration- AD groups. User A from dept A logs in….(scenario) Since we are pushing this configuration from the deployment server Indexer replication (and the lack there of) Reporting tool not a repository tool
- #27: So how do we make deployment easier?
- #28: For IT12 -- It starts on the back end getting our app copied and configs in place for the forwarder to phone home for the first time and sync So Dainel built a script to do the heavy lifting for us. Confirms your logged on as Splunk Asks for location (user input) Asks for OS type (user input) Adds the needed server classes (assign deployment apps in serverclass.conf) Adds the needed indexes (assign indexes in indexes.conf- homepath, coldpath, thawedpath etc.) Copies our application template Replaces department variables Assigns roles (authorize.conf) Assigns an AD group reference (authentication.conf) Assigns a default_namespace Gets fancy with the cron job wiz (looks to Daniel) Each night we push the splunk shculster-bundle cmd to prevent restarts during the day Okay so how do we get the forwarder to the departments??
- #29: Then we have to get the forwarder to the servers….and one does not simply walk into Mordor and just receive admin rights. Linux packaged tar.gz PowerShell Launches Splunk Forwarder msi
- #31: No tool to deploy … no problem Script the script deployment!
- #33: Reports saved every 2 hours Provide a high level overview for all four policy alignment pieces Limited access
- #34: Typical drilldown functionality takes you to a search bar Note multi-select File Path box
- #35: Training – Get some! Optimizing Searches – Refine everything at the beginning, avoid heavy utilization items like transactions appends, and additional searches to searched data Data Inputs – Make sure to use search time data manipulation vs index/forwarder time data manipulation Reporting – Initially we tried to set up saved searches to show data, and update it every 2 hours. Use report acceleration without a schedule.
- #36: Removed the activity dropdown from the splunk bar Added a new drop down menu with all for drilldown dashboards Added tables Removed all scheduled searches, and set the searches to on demand with report acceleration Emailed reports
- #37: Added in additional exclude multi select boxes Added behind the scenes options like optimized searches, auto populated drill down functionality, on demand searches
- #39: Main two areas for items down the road. Usability expansions of IT12 App Devices not checking in, Alerting, Filtering of network / console traffic Expand IT OPS Kuali, VMware, AD, Web Services, Security App, First development outside of IT12 is for CAS and Shibboleth.