How to optimally resource your Netezza team

IBM’s Netezza Performance Server systems require far less effort to administer than a traditional RDBMS thanks to being very fault tolerant. Not having frequent outages can, however, lead to a false sense of security, meaning that some businesses may hand over the day to day running of Netezza to a non-specialist team in a bid to reduce costs. When unforeseen hardware, software, systemic, or environmental problems occur, what once was the well-behaved kid in the corner becomes the problem child that needs a lot of attention.

Why is this? Well, in a nutshell, Netezza is a complex integrated system comprising many distinct pre-configured parts which traverse traditional departmental boundaries. Unless specifically trained, a DBA can’t be expected to know what the settings, standards and procedures are to ensure the system operates at maximum performance.

It’s not just the proprietary nature of Netezza that requires specialist knowledge, but also the requirement to keep the system at high availability owing to its mission-critical status, often with Board level visibility. Netezza handles large volumes of commercially sensitive data, with a variety of different concurrent users and workloads, some of which are extremely performance dependent, and all of which are contending for a finite set of resources. So, making sure you have the right skills on hand is essential as downtime has a big business impact.

It’s a Case of Not Knowing What You Don’t Know

When managing a sophisticated appliance, it’s important to know what you don’t know, and that is where augmenting your Netezza resources with on-demand skilled ones is important. I recall a client that handed their Netezza administration over to their dedicated Unix team that took very little time to bring the system to its knees.

Shortly after the handover their Unix team decided to upgrade the version of Red Hat being run on the Netezza appliance so that all their platforms would be standardised. The problem was they didn't realise that Netezza is an appliance and had been set up that way with a Postgres database acting as a repository for all the metadata. So, they broke it and it took quite a while to recover the system.  The Unix team did not know that the only way to upgrade Netezza is by using IBM authorised patches and upgrades that they have tested to make sure don’t compromise the system in any way.

Danger also lurks when non-Netezza DBAs assume responsibility for Netezza. An Oracle DBA, for example, will do Oracle-type things and they won’t know to do the Netezza-type things to keep the system running at peak performance. Without critical Netezza housekeeping tasks being performed, the performance deteriorates, and the problem is actually magnified. Arguably, the more non-Netezza DBAs you throw at the problem, the worse it can become and eventually you have to call in the experts anyway.

A Broad Spectrum of Team Structures

Getting to the crux of this article, there are many different resourcing models that can determine how much Netezza expertise exists on-site. At one end of the spectrum are companies that have highly compartmentalized teams taking responsibility for different aspects of the system e.g., Linux, Database, Network, Operations, Security, Backup & Recovery. These teams are responsible for very discreet areas, none of which can be expected to be Netezza experts, but they may still all get involved to solve a Netezza problem. This comes at a cost, especially if they are not actually able to contribute anything to the solution.

At the opposite end of the spectrum are companies with a dedicated Netezza team. However, providing a whole team of FTEs is a costly exercise for a system that is 99% plus available. If you start scaling that back, such as relying on just one or two Netezza specialists, big gaps will appear in skill coverage – after all, they have to sleep, have to go on holiday, do training, and sometimes get sick. So, when any of those things occur basically the whole system is put at risk. 

Our Recommendations for Ensuring Sufficient Netezza Expertise

Clearly neither extreme is optimal. They have their own relative advantages and disadvantages. Our recommended approach is to have a single team responsible for the Netezza platform overall, and this team should consult with other internal functional teams for best practices and in-house standards to be implemented, as well as define communication and workflow interfaces between the teams. 

We also suggest automating as many systems management functions as possible - e.g. backups, genstats, groom tables/versions, manual vacuums, AD/LDAP synchronization for users, groups, and permissions and monitoring and alerting, etc.

To overcome the problem of needing 24x7 availability of your mission critical system, we also suggest augmenting your in-house resources with external ‘experience on-demand’ type services to provide 24x7 cover and assist with diagnosing/resolving any issues that arise.

The Downside of DIY Automation

Many of the tasks that are routinely done by Netezza DBAs can be automated and many companies have done just that. Herein lies another problem, though. If the automation tasks are written to perform on the Netezza host, as they usually are, then as soon as the customer wants to upgrade to the NPS Cloud Pak for Data system, any process they have running on the host currently will have to be migrated off to somewhere else. So that is a pretty significant effort that may have to be repeated each time a major system upgrade occurs. With this in mind, we developed Smart Management Frameworks which takes all the automations and housekeeping tasks off the host and as such regardless of what changes are made to the architecture, nothing is going to fall over. 

Automation can't do everything of course. If you have a performance problem, you will need someone to investigate what the underlying cause of the problem is. It could be badly written SQL, a poor database design or workload management settings that need to be adjusted to alter the priority of different queries being run by different users within the organization. This type of investigative work is almost impossible to automate, as is formulating a plan on how to fix it.  For that reason, we have our Virtual DBA offering which provides performance tuning, workload management, query optimization and so forth on a 24 x 7 basis. So rather than paying for round the clock Netezza resources, this service allows customers to augment their in-house team for such occurrences and/or when their dedicated resources are rostered off.

Achieving the Best Possible Outcome

In summary, Netezza is a complicated system that requires very niche skills to keep running at peak performance, especially now that it is hosted on IBM’s Cloud Pak for Data System with its containerized, modularized architecture. Retaining highly trained resources in a niche field has always been expensive, which is why we have developed a suite of products and services that allow companies to augment their Netezza skills with automation tools and on-demand expertise.

Furthermore, if you want to migrate to NPS on your own private cloud, we can supply you with a bundled solution that includes not just the NPS software all installed and configured, but also migration to the new platform; safety, security, and surety automation via Smart Management Frameworks and Virtual DBA support all included. 

If you’d like to find out more about this, or how to optimize your Netezza team structure, please get in touch.

A full version of this article can be found on our website https://smart-associates.biz/blog/resourcing-your-netezza-team-optimally.