ETL Developers

The ETL process has three steps: Extract, Transform, and Load.

In the extraction phase, raw structured or unstructured data from a variety of sources (e.g., customer relationship management systems, enterprise resource planning systems, application programming interfaces, SQL, CSV files, Web pages, XML, and JavaScript Object Notation) is copied or exported and temporarily stored in a staging area. Validation rules are applied during this step, and only data that meets the requirements of its destination is moved to the next step in the ETL process.

In the transformation phase, raw data is filtered, cleansed, formatted, or otherwise optimized for quality to fit within a uniform schema and is easier to use for business functions, analytics, and other downstream activities.

In the load stage, the transformed data is loaded—often in batches—from the temporary staging area to a target database, data warehouse, data hub, data lake, or data store. Loading is done on-premises or in the cloud. The ETL process is then complete.

Job duties for ETL developers vary based on their employer, training, and job title, but their “big picture” duties involve designing, creating, automating, troubleshooting, and supporting ETL applications to extract and transform large amounts of data before building data pipelines to transfer it into new data storage applications. For a specific project, developers meet with executives, clients, and other stakeholders to determine what type of data will be moved through the ETL process, the data storage format and other requirements that will be needed, and the expected project budget and timeline. They create the data warehouse or other data storage application, then work with testers, business analysts, data experts, and others to test and debug the product to ensure it functions correctly. They also enhance and repair existing data applications. Many developers oversee the entire ETL process to ensure that it works as designed.