What are the design patterns for cloud-based event-driven architecture?

One benefit of this approach is that the major cloud providers have a service that supports the “pub/sub” pattern so teams don’t need to build their own. Google Cloud’s Pub/Sub, AWS SNS, and Azure Service Bus can provide teams the ability to implement this pattern depending on their preferred cloud infrastructure.

Pub/Sub addresses the challenge of decoupling Senders and Consumers by employing Asynchronous messaging. This approach offers multiple advantages high scalability, availability, and fault tolerance. However, employing this pattern may pose certain challenges: 1. The message broker's failure could disrupt the entire communication process. 2. Implementing bidirectional communication might prove challenging within this pattern. 3. Handling failed messages: Unprocessed messages can accumulate, burdening the subscriber. Implementing dead letter/posiend queues can mitigate this issue. It's advisable to avoid this pattern if you aim to build a near real-time application.

Event-driven architecture in the cloud relies on events triggering actions. Events are generated by changes in the system, and services communicate asynchronously. It enhances scalability, decouples components, and allows for real-time responsiveness to changes in the system.

In addition to different models, there are also unique approaches to processing the events once they reach a consumer. Simple event processing: Consumers process each event as it is received, such as in the previous eCommerce site example. Complex event processing: Consumers process a series of events to detect patterns in the event data. Event stream processing: Data streaming platforms ingest events and create a pipeline to stream processors that transform and consume the data.

In cloud-based event-driven architectures, various design patterns are often used to handle asynchronous events efficiently, enable decoupling of services, and improve scalability and maintainability. Here are some key design patterns relevant to such architectures: Event Sourcing Description: Capturing changes to application state as a sequence of events. This allows the system to reconstruct past states and enables audit trails. Use Cases: Systems where historical information is crucial, like financial transaction systems.

Event Sourcing differentiates between data and business (aka domain or application) events. Data events are changes in databases like SQL or NoSQL. For instance, when a new entry is added to a SQL table, this can be captured using Change Data Capture (CDC). Using these events directly can cause coupling, making downstream applications rely on your REST APIs for more details. Instead, within your microservice boundaries, transform data events into business events that fit a specific contract, then send them to an event bus like Kafka. Ideally, the application should emit business events directly to integration environments.

Event Sourcing offers a method to conduct data operations guided by events, establishing the event source as the sole source of truth. This fosters efficient auditing and action tracing through action replay. However, challenges may arise with this approach: 1. The system achieves eventual consistency, making it unsuitable for real-time data-dependent applications. 2. In multi-threaded or multi-instance setups, different instances of events may be stored, potentially resulting in discrepancies where the latest write might not accurately represent the application's true state.

The event sourcing pattern records application state changes as a series of events in an append-only log. This log is the application's source of truth, used to reconstruct component states, support auditing, analytics, and scenario replays 🔍. The pattern ensures high performance, consistency, resilience, and eases integration with event-driven components, enhancing overall system robustness and traceability. 🔄💪

Adopting a secure-by-design approach, the event sourcing pattern proves invaluable for cloud-based EDA. By capturing state changes in a secure, append-only event log, this pattern establishes a tamper-evident source of truth. Storing events ensures reconstructability of the application's current or past states, facilitating auditing, analytics, and scenario replay. This not only enhances performance, consistency, and resilience but also aligns with security principles by providing a verifiable and immutable record. In addition, the event sourcing pattern facilitates seamless integration with other event-driven components, forming a robust foundation that prioritises both functionality and security within the cloud-based EDA landscape.

The Command Query Responsibility Segregation (CQRS) pattern divides an application's read and write operations into distinct components. The write component processes commands, alters the application's state, and logs events 📝. The read component manages queries and updates its data store by subscribing to the event log 📊. CQRS enhances scalability, availability, responsiveness, and ensures a clear separation of concerns, optimizing each component's functionality. 🚀⚖️

CQRS creates a strong separation of concerns by distinguishingly dividing read and write processes into separate components, each with its own data store and logic. The write component, which is in charge of changing the state of the programme, makes sure that security is maintained by sending events to an event log, which is a safe and impenetrable record. In the meanwhile, isolation is strengthened by the read component, which responds to inquiries and subscribes to the event log. This pattern conforms to best practises for protecting data integrity and reducing attack surfaces within the cloud-based EDA paradigm, while also optimising scalability, availability, and responsiveness and fostering a security-conscious architecture.

Separation of Concerns: Traditional CRUD systems often intertwine the model responsible for reads (queries) with the one for writes (commands). In CQRS, these two are decoupled, leading to distinct models for each. This separation can lead to cleaner, more maintainable code. Scalability: Since read and write operations are separated, they can be scaled independently. For instance, if an application has a high read-to-write ratio, more resources can be allocated to handle queries without over-provisioning the command side

CQRS is a time-tested pattern with strong applicability to modern cloud architecture. It lends itself well to microservice architecture and scalable data stores by allowing each microservice domain to also define the "write" operations and "read" operations into different areas. This allows you to optimize writes in a transactional manner (and event-driven for cross-domain operations), but then optimize reads using read-only replicas or eventual consistency patterns. CQRS gives you the patterns you need to do these things in a managed and repeatable way.

There are two way to implement Saga Pattern, choreography and orchestrator. In choreography, services communicate directly, each handling its local transactions. On the other hand, orchestration involves a central controller (the orchestrator) that coordinates participants’ actions. The orchestrator interprets states, ensures data consistency, and handles failure recovery using compensating transactions. Use Saga Pattern when, 1. Maintaining data consistency across a distributed system while avoiding strong dependencies. 2. Implementing a fallback or compensatory mechanism in case a step within the sequence encounters a failure.

The saga pattern manages long-running or distributed transactions as sequences of events, denoting local or compensation actions. A local action alters a component's state and triggers the next step 🔄, while a compensation action reverses a prior action's effect ⏪. This pattern ensures high reliability, consistency, recoverability, and adeptly handles complex business processes and workflows, promoting streamlined and resilient systems. 🛠️💼

The saga pattern assures dependability and consistency by organising transactions as a sequence of events, each of which either a local action or a compensatory action. Local activities, changing the states of component parts, emitting events that start other actions or signal success or failure. Reversing previous local acts with compensation actions improves recoverability even further. By preserving a transparent audit trail through emitted events, this design not only complies with security standards but also strengthens the integrity of intricate business processes and workflows. Thus, in the complex world of cloud-based EDA, the saga design prioritises consistency and reliability while establishing a strong and secure foundation.

In my understanding event storming is a way of designing an app that uses events to communicate between different parts. Events are things that happen in the app, such as a user clicking a button, a payment being processed, or an order being shipped. Event storming helps you find out what events are important for your app, and how they affect each other. You can do event storming with a group of people who know different aspects of the app, such as developers, designers, customers, or managers. You can use sticky notes or a whiteboard to write down the events and draw connections between them. Event storming helps you understand the app better, and make it faster and easier to build.

The event storming pattern aids in discovering and modeling event-driven applications through collaborative brainstorming. Stakeholders map out events, commands, queries, aggregates, and policies, defining the application's behavior 🤝🗺️. This pattern ensures alignment, clarity, and creativity, accelerating the design and development of robust event-driven applications with precision and efficacy. 🚀💡

The event storming pattern, is crucial to designing a cloud-based EDA that is both safe and durable. In order to define the behaviour of the application, a variety of stakeholders are involved in a collaborative and visible process that maps out events, instructions, queries, aggregates, and policies. The focus on cooperation not only promotes alignment but also improves security by guaranteeing a thorough comprehension of the nuances of the application. The pattern strengthens the design and development process and fosters clarity and creativity by bringing together stakeholders from different areas. This method, is built on cooperation and security, lays the groundwork for building a solid and well-architected event-driven application

This is *gold standard* of what good looks like for large-scale distributed systems. But it requires focused and aligned leaders driving teams to work together with full and complete views of the entire system. This architecture is the most flexible, sustainable, and maintainable because the entire organization will be in lockstep about design patterns, event payloads, and data lifecycles so all major changes are agreed upon and peer reviewed by other engineering teams. When designed intentionally, this architecture can also be highly observable with modern tracing techniques, allowing operators to have deep visibility into multiple layers, making it easy to understand, debug, and fix on the fly.

It's crucial to determine event storage strategies, including retention policies and considerations for replayability. Storage optimization for efficient querying is essential. Managing transactions and meeting eventual consistency requirements are pivotal aspects. Implementing a mechanism to prevent event tampering is also imperative. CloudEvents has emerged as a widely adopted standard for standardizing event formats, promoting interoperability, and ensuring vendor neutrality. This standard is now embraced by various ecosystems such as AWS, Azure, GCP, Kubernetes, among others. Therefore, it's vital to ensure that your approach aligns with and adheres to this standard.

There are several design patterns commonly used in cloud-based event-driven architecture. Some of these patterns include: 1. Event-driven architecture 2. Event sourcing 3. CQRS (Command Query Responsibility Segregation) 4. Event-driven messaging 5. Event-driven integration 6. Saga pattern 7. Reactive architectures These are just a few examples of design patterns for cloud-based event-driven architecture, and there are many more patterns and variations depending on the specific requirements and use cases of an application.

Most enterprise cloud architectures are going to leverage all of the patterns described in this article. The goal is to find what's the best fit for the application you're building today and how it needs to scale in the foreseeable future. Don't over engineer and incorporate these patterns just for patterns sake - you'll create something even worse in the end.

Data loss is a common challenge with event driven architecture. If any of the components crashes before successfully processing and handing over the event to its next component, then the event is dropped and never makes it into the final destination. To minimize the chance of data loss, persist in-transit events and remove / dequeue the events only when the next component has acknowledged the receipt of the event. These features are usually known as client acknowledge mode and last participant support