Cloud Automation Integrations for Azure from Automation by Broadcom

Written by: Richard Kao

For about as long as there’s been what we’ve known of as “the cloud,” there’s been Microsoft Azure. The technology was first unveiled in 2008, and the number and nature of the offerings available have continued to evolve. Today, the Azure cloud platform features 200 products and services, including virtual computing, storage, networking, and analytics offerings. In 2021 alone, through its Azure and other intelligent cloud offerings, Microsoft has generated more than $60 billion in revenues.

Limitations of managing workload automation in Azure

While teams are gaining efficiencies by moving workloads and automation to the cloud, significant challenges remain.

Lack of business process visibility

These days, organizations will typically have business services that rely upon workloads running in Azure and other environments. While automation operators can use Azure tools to manage their Azure-based workloads, these tools don’t provide complete visibility of the end-to-end automated process, they don’t track dependencies between jobs, and they don’t provide insights into potential SLA breaches.

Limited scheduling capabilities

In Azure environments, teams can employ basic, time-based schedulers. However, these technologies can’t intelligently accommodate dependencies—and these workflows typically have multiple upstream and downstream dependencies. The only option is to create hard-wired time delays, that is, scheduling subsequent tasks to start at a time after which prior tasks are expected to have been completed.

Following are just a few of the challenges these approaches present:

• Reliability and data quality issues. Relying on these hard-wired schedules means that if one task takes longer than the forced time delay established, a subsequent task will kick off, typically with old, inaccurate, or incomplete data. This means the job sequence may fail or return suboptimal or unusable data. These issues get magnified in many large-scale environments, where dozens of data sources may be used. If one source doesn’t come across in time, workflows may encounter cascading failures.
• Connectivity-related failures. When running Azure, users are highly reliant upon a range of dispersed, distributed networks. At any given moment, automation jobs can fail, simply due to a call to an API returning an error message. When this type of downtime occurs, scheduled jobs will fail, creating issues for subsequent downstream jobs.
• High costs and poor resource utilization. To prevent these issues, operators can opt to add buffers, that is, delaying the start time of subsequent jobs to accommodate potential delays. While this approach can help minimize failures, Azure instances will need to be kept idling, often unnecessarily. This can be very costly. In many environments, these idling resources may account for fees of thousands of dollars, and these costs are accrued frequently.
• Labor-intensive follow-up and mitigation. If a failure is discovered while a workstream is underway, the administrator will have to disable the schedule, potentially in multiple products, and manually troubleshoot and address any issues that have arisen.

The solution: Cloud Integrations from Automation by Broadcom

Automation by Broadcom offers robust scheduling that enables users to manage dependencies across pipelines, integrations, applications, and processes. These solutions deliver end-to-end visibility across on-premises deployments and various cloud environments, including Azure.

Today, Automic Automation, AutoSys, and dSeries offer integrations with these Azure solutions:

• Azure Blob Storage. Azure Blob Storage is a service for storing large amounts of unstructured data in the cloud. With our integrations, you can easily use your enterprise workload automation solution to monitor and manage your Azure Blob Storage objects.
• Azure Data Factory. Azure Data Factory is a popular cloud-based ETL and data integration service that allows you to create scalable workflows for moving and transforming data. Automation by Broadcom enables you to easily integrate Azure Data Factory pipelines with your existing enterprise workload automation. Once you’ve defined your pipelines within Azure Data Factory, use your enterprise workload automation solution to manage the scheduling of these workflows. (See my recent blog post for more on how we streamline Azure Data Factory workflows.)
• Azure Event Grid. Azure Event Grid provides a scalable publish/subscribe message distribution service. With this fully managed service, you can build data pipelines, integrate applications, and create event-driven serverless architectures. Azure Event Grid offers flexible message consumption patterns using the MQTT and HTTP protocols. With the Broadcom Azure Event Grid integration, you can easily include the publishing of Event Grid events, custom events, and cloud events in your enterprise's automated workflows.
• Azure Synapse. Azure Synapse Analytics offers advanced data warehousing and big data analytics capabilities, combining SQL, Spark, and Azure Data Explorer. With our integrations, you can efficiently integrate Azure Synapse jobs with your existing enterprise workload automation. Simply define your automated pipelines within Azure Synapse and use your enterprise workload automation solution to manage the scheduling of these workflows.

Benefits of Enterprise Automation by Broadcom

There are many benefits to extending enterprise automation to cloud processing. With Automation by Broadcom, you can effectively establish a “manager of managers,” employing a unified tool to govern several disparate cloud scheduling and workflow orchestration solutions. In this way, you can gain end-to-end business process visibility across all your cloud and on-premises workloads.

By extending your existing enterprise automation to Azure-based business processing, you maintain end-to-end visibility and you regain centralized command and control. The solution gives you unified automation observability of your cloud processing. Automation by Broadcom enables you to harness the power of predictive analytics with smart alerting, as well as advanced SLA management, reporting, and audit capabilities.

Learn more About Broadcom’s cloud integrations

Broadcom’s Automation Marketplace makes it easy to browse and search for cloud integrations available for Automation by Broadcom. The site features information on all Broadcom cloud integrations, including those for Azure, as well as Cloud Foundry, Databricks, Google Cloud Platform, and many more.

Visit the Automation Marketplace to get access to integration details, links to technical documents, and software downloads.