From the course: Apache Airflow Essential Training
Join today to access over 24,800 courses taught by industry experts.
Introducing the FileSensor - Apache Airflow Tutorial
From the course: Apache Airflow Essential Training
Introducing the FileSensor
- [Instructor] In this demo, we'll work with a file sensor in Apache Airflow. Now, sensors are a special kind of operator that are designed to do exactly one thing, wait for something to occur. Sensors wait for a certain condition, or an external event to occur, before proceeding with the execution of a task or a workflow. Sensors are very useful when you want to pause the execution, until a specific condition is met. Thus, sensors provide a way to react to changes or events from various sources, such as files, databases, APIs, and so on. First, we'll understand the operation of a sensor. And then we'll use a sensor in a pipeline. Notice I've created a tmp, tmp sub folder under my Airflow folder. This is where my sensor is going to look for a specific file. Please note that the tmp folder is initially empty. Also, on line eight of my code, you can see that I have an import for the file sensor. Let's take a look at our…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.