Orchestra

Analytics engineers! Are you looking for a DEVELOPMENT Environment for dbt-core or a DEPLOYMENT Environment? 🤔 Orchestra offers a lightweight deployment environment for dbt-core WITHIN the Orchestration Control Plane. Super-easy to set-up, lightweight pricing, full visibility. Check it out below. https://lnkd.in/eMvx-6Q8 #dataengineering #analytics #dbt

Three big fundraises / acquisitions in the data space last week 💡 We were actually LATE to notice one. Not sure how it slipped our radar.. Find out more in link below. Congratulations Databricks and bauplan! #dataengineering #ai #datacontracts

Are you still stuck creating Data and AI Workflow Pipelines solely in AWS? Are you wondering why your manager insists on making you make the same mistakes over and over again? You're not alone. Show them this.💡 -------------- ☑️ Speed of implementation: AWS Lambda and Step is not a solution to orchestrating data pipelines or running heavy compute workloads / dbt on its own. Implementing this ourselves could take months vs. Orchestra which is ready off-the-shelf IMPACT: 6-12 Months vs. Now ☑️ Maintainability: AWS services would require a complex in-house build to work, and on its own is not an Orchestration solution. This means pipelines would take weeks to build instead of minutes IMPACT: Orchestra takes 90% less time to maintain ☑️ Usability: AWS Lambda / STEP forces anyone interested in Data, Analytics, or AI workflows to be registered in AWS and subscribe to AWS Services. AWS Is typically designed for highly technical team members, and would create issues with scalability in the future and usability today IMPACT: Orchestra removes bottlenecks, increases time to live of new Data Products by 50%+ ☑️ Complete solution: AWS Services on their own cannot handle out-the-box data processing, dbt, metadata aggregation, alerting, integration to tools (Snowflake, Power BI), error-handling, and provides no UI for viewing or structuring metadata IMPACT: AWS Lambda introduces enormous complexity; Orchestra is simple and scalable #aws #awslambda #data

Do you use a tool like Fivetran to land data in Snowflake? Are you worried your stakeholders sometimes produce dodgy data? Do you have stakeholders that want access to the raw data? ❌ If you don't test this data when it lands, stakeholders are going to instantly see dodgy data 💸 Creating another layer of data transformations as a staging area can be costly 😕 Data Quality Tools struggle to deal with tests in-flight easily, and only excecute after the fact Fortunately you can get around this easily with the CLONE command and DQ tests in an orchestrator -- Land the data in a "Raw" Schema in Snowflake -- Execute required data quality tests from the Orchestrator -- Use a "clone" command to create a zero copy clone of the data from a "Raw" to a "Trusted" schema in Snowflake Video of how you do this in Orchestra below. #dataengineering #stagin #snowflake tips

Struggling to see how to get around building your data pipelines as one massive DAG? You're not alone! Check out some helpful patterns in the video below, using Orchestra https://lnkd.in/ec9Ei4qh #dataengineering #datapipelines #ai

5 reasons it's time for a Modern Orchestration Platform https://lnkd.in/dkSYfF-w Follow Hugo Lu for more insights. #dataengineering #orchestration #orchestra

Some big news in the roundup this week around Temporal Technologies, ziggiz and Microsoft Fabric #MSFabric ! Check it out below. 🧠 Automated Schema Drift Management in Snowflake Using Cortex LLM 🧠 Globalizing Productions with Netflix’s Media Production Suit (link) 🧠 Linear Programming: Managing Multiple Targets with Goal Programming Editor’s Pick 🧠 The Rise of the Open Lakehouse made Databricks. It could bring it down 🧠 The Evolution of Data Engineering: From Flat Files to AI-Driven Pipelines 🧠 Announcing Orchestra Partnership with Enzo Unified 🧠 Apache Spark : Tungsten a detailed view 🧠 Are We Watching More Ads Than Content? Analyzing YouTube Sponsor Data #dataengineering #fabric #temporal #analytics

Existing tools create massive silos in organisations. Without workflows that are easy to manage, the data team will always be a bottleneck that is insurmountable for organisations try to use AI. https://lnkd.in/eYkdcQ22 The Silo Trap is everywhere. #data #ai #genai #agents

Super awesome to see use-cases for event-driven workflows and parameterising dbt commands on the fly! https://lnkd.in/eYwqRAVf #faster #cheaper #orchestra

You can use Orchestra to run different parts of your pipelines on demand! 🥳 This means you can - Add a new Task in a Development Environment - Run the Task in the Development Environment - Run the downstream dependencies - Run the single task in Production after publishing And of course this functionality is exposed as an endpoint so it can be automated in CI/CD pretty neat huh? #dataengineering #orchestra #cicd