Sigmoid reposted this
Data Engineering Intern @ Sigmoid Analytics | Former SDE Intern @ Dell Technologies | IT'25 @ IIIT-Bh
Hey folks! A couple of weekends back, I had the opportunity to participate in my first corporate hackathon – a 𝟮𝟰-𝗵𝗼𝘂𝗿 internal 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗛𝗮𝗰𝗸𝗮𝘁𝗵𝗼𝗻 at Sigmoid, with 95+ interns across 17+ teams. I teamed up with Nilesh Phapale, Akshat Agrawal, Atharva Kale, and Ankit Debroy as 𝗧𝗲𝗮𝗺 𝗦𝘁𝗿𝗮𝘄𝗯𝗲𝗿𝗿𝘆🍓 , where we took on the challenge of building a 𝙨𝙚𝙡𝙛-𝙨𝙚𝙧𝙫𝙚𝙙 𝙙𝙖𝙩𝙖 𝙥𝙞𝙥𝙚𝙡𝙞𝙣𝙚 𝙪𝙨𝙞𝙣𝙜 𝙈𝙞𝙘𝙧𝙤𝙨𝙤𝙛𝙩 𝙁𝙖𝙗𝙧𝙞𝙘. Self-served data pipelines have two key goals: 1. Enable non-technical users to build pipelines easily using no-code tools 2. Help technical users automate the creation of many similar pipelines, reducing repetitive and error-prone work These are foundational capabilities that most data teams aim for — yet there’s no widely adopted, general-purpose self-served data pipeline solution available today. For the hackathon, we focused on implementing these two core features in a simplified, functional way. Using Microsoft's Fabric Lakehouse and Data Factory, we built a tool that lets users define transformations and automatically generate pipelines — aiming for flexibility and ease of use. 💡 Challenges we faced: - Limited API endpoints in Microsoft Fabric, and evolving documentation - Very little community support – we relied heavily on trial, error, and creative problem-solving - Implementing dynamic user-selected transformations without using any server or backend database — we eventually tackled this by leveraging parameterized notebooks within Fabric to run transformation logic on the fly This was a huge learning experience. When we started, we didn’t even had a clear understanding of what a data pipeline really meant — and by the end, we had a working tool that could automate creating one. It was an intense but fun sprint (including an all-nighter!), and we were thrilled to see our efforts recognized — we got to showcase our project to our CTO and were awarded 𝗳𝗶𝗿𝘀𝘁 𝗽𝗿𝗶𝘇𝗲 (𝗚𝗼𝗹𝗱) by our founders during our company-wide all-hands meet! A big shoutout to our team mentor Sonia Das for crafting such thoughtful problem statements and guiding us throughout the hackathon. Looking forward to more such experiences! 🚀 Live long and Prosper🖖 #DataEngineering #Hackathon #MicrosoftFabric #SelfServedPipelines #LearningByDoing #SigmoidAnalytics