Extending D365 with Azure

Editor's Notes

• #2: Note: This presentation was developed in October 2017. The services in the cloud are constantly changing.
• #6: There are a lot of good reasons for using Azure, such as reliability. Cost should not be the only consideration. Consider the learning curve to maintain apps in house vs in the cloud, and the cost of keeping updates in lock-step with the platform. Inconsistent workload demands – Web Portal with usage spikes or scheduled business process that does not need to run all the time Good for offload batch processing – Asynchronous process to create 20k records, such as monthly billing Use it to consolidate business logic – such as an integration with Microsoft Graph services, machine learning, or handling complex calculations like linear equations for product mix.
• #8: When you build extensions to D365, you have many options available to run your code in Azure. These are listed in the Compute column. We will discuss the first 5 of these because they provide a diverse set of features that work for many different applications. But the code does not run itself. Somehow the program you have hosted in Azure must determine when to run and which records in CRM they apply to. This is where we introduce the integration options because they control which programs will run in response to user activity in CRM. The most common integration works through defining an endpoint in D365 that connects to a Service Bus. An optional component to consider when extending your D365 to Azure involves using one of several storage options. Lets start with the Integration part first.
• #9: The Service Bus is the most common way to communicate events in D365 out to your Azure programs. Service Bus is a first-in-first-out message queue. It is like a database pipe – messages go in one side and come out the other. When you create a Service Bus Namespace in Azure, it gives you a connection string to connect to it. Your custom code would be the message receiver, and that code must retrieve each message (in effect lock it) run a process based on what is in the message, then mark it as complete. Why would we use it? D365 is super simple to send messages to it, it can queue up 1M messages, and it handles failures for us without writing any logic.
• #10: In general, D365 is easily configured to send messages through a Service Bus to trigger your programs in Azure to start running. When an event happens in D365 that we want to trigger some Azure code to process, we have a workflow that creates a record in our custom Service Bus Entity It is configured to push messages into separate queues or topics depending on what the event was. The event message sits in the Service Bus queue until the Service Bus Manager picks it up and calls your Line of Business (LoB) service. LoB Service connects to CRM and performs work Errors are logged for an operator to fix and restart the process
• #11: Do you need to use the Service Bus? No. You could use Webhooks if you are using D365 CD 9 or later. Here are some reasons you should use the Service Bus: Consider the scenario when you need to update a back-end service. By using a service bus, you can stop and start the back end service without disrupting the other business processes.
• #12: Create a custom entity Register the endpoint Register the step (CRM event message)
• #14: How much control do you want? You might need to give up some control to get scalability, quick deployment, security updates Can you reassign your team to higher value tasks?
• #15: Which service is right for you? There is a tradeoff between what is quick and easy to code and the need for robust enterprise features. I would categorize the services this way:
• #16: “Server-less” just means you don’t own the server. Languages: JavaScript, C#, and F#, and scripting options such as Python, PHP, Bash, Batch, and PowerShell. Triggers on Microsoft Graph Events (Excel Table events, One Drive file events, Outlook), Azure Cosmos DB, Blob storage, Events, HTTP REST or Webhook, Service Bus, and Timer. Integration with: Logic Apps, Storage tables, SQL tables, Twilio, SendGrid, Push Notification Hubs, and Auth Tokens. Complements Azure Logic Apps Run this with service bus topics and subscriptions
• #17: Another “Serverless” option Technology that Flow is built on Example: Poll the database every minute for 10 days + execute 500 workflows = $15/10 days Glue between applications Triggers: recurring schedule, on demand, HTTP, API News features released in frequent updates
• #18: Easier for a developer to get up to speed PCI, SOC, ISO compliant Triggers: Blobs, Service Bus, Queue, Timer, continuous, custom No remote desktop
• #19: Big learning curve. Better choice if you plan to migrate legacy applications away from an enterprise data center Stateful microservice removes the need for queues and caches. Stateful services are high availability and low latency. Auto-scaling built in to clusters. With the Fault Analysis Service, you can induce meaningful faults and run complete test scenarios against your applications
• #22: Reference architectures – Network DMZ, Hybrid, Identity Management, Managed web apps Performance Antipatterns – Busy Database, Chatty I/O, Extraneous Fetching (too much data),
• #23: Shows some example cloud architecture design patterns. There are over 30 to choose from. The Sidecar pattern is for features like logging, and security that may be part of your framework to provide orthogonal services. Anti-corruption layer deals with exposing views of data from the legacy system in a manner consistent with the new system and allows the legacy system to function without modification. The Strangler is used to direct data requests to specific pieces of functionality that is being incrementally moved form the legacy system to a microservice architecture. Ambassador patterns act as proxy to services to allow control over routing, resiliency, security features, and to avoid any host-related access restrictions.
• #24: Your on-premesis systems typically are using a SQL technology that guarantees data consistency, (also called ACID: Atomicity, Consistency, Isolation, Durability) Dynamics 365 lives in the BASE world. Basically Available, Soft state, Eventual consistency means that transactions are distributed and they are eventually updated in the database(s). This means that D365 may not be 100% accurate all the time and you must design processes that will catch inconsistencies and trigger a process to fix it. Event Sourcing is used to keep object state history so that it can be re-evaluated and corrected Command Query Responsibility Separation is used to de-couple the recording of the transaction from an actual update Use Materialized Views design pattern to present the result of the transactions Retry Pattern depends on what the fault is – cant help you if your code is broken.
• #25: Example of what could go wrong if you create a system with ACID in mind, but you are really using a BASE system: You start with a known account balance But you end with an incorrect balance What happened? What was the balance before the transactions? If this is milliseconds apart, it would be very hard to tell what is going on. You could use logging, but then someone has to monitor it, and how do you fix it?
• #26: By keeping the entire history of changes, we no longer really need the Balance (State) of the account We can completely recalculate the balance, or re-play a series of events (think debugging). We can also see the balance at any point in time. Eventually all accesses to an item will return the last updated value. By pushing updates into a transaction log, process does not have to wait for CRUD operations The cost: storing more data, overall system is more complex because it is many smaller parts, overhead to complete the task is more
• #27: For each transaction, keep track of steps taken If the process is not completed, have a process that will walk back through and undo what was done You are effectively creating what you might have done If you have used transactions in SQL Server
• #29: None of these reasons should stop you from using it, unless you are selling D365 licenses under a CSP model Microsoft has not been putting any effort into this area – seems like it is going away Our customers only get about 5% of CPU during peak periods, but pay all month for it
• #30: Location, location, location – test the latency because it can really cost you
• #31: My profile was for East coast time, but customer was in Houston and they saw my changes as having the wrong time. Keeping track of your services using ARM is really important to containing costs - Make a governance policy on who owns what using tags! TDD costs a bit more up front, but really pays for itself after sprint 1
• #32: If you include the Service Endpoints in the solution file that you promote from dev to production, you will be connecting the production instance back to the development code The message on the service bus has the domain of the system that created the message. Compare it to the connection string domain to make sure they are the same. You can differentiate which messages belong to which programs by using queues and topics – don’t use multiple service buses In D365, check the “non-interactive” box on the user account you use for integration.
• #33: These are the classes you need to combine to into creating a transaction The benefit is to reduce the amount of chatter with the API, and it is an all-or-nothing, which saves you from having to clean up partial transactions
• #34: Some key elements are color coded to make it easier to follow where they are initialized and used. This sample is a membership dues Header / Detail A key point – the duesHeaderId is manually assigned in order to build the link between the parent and child record Typically, you would have more than one detail (child) record.