Data Engineer

Job Title: Data Engineer Work Location: Irving, TX Required Skills: Expertise in Designing and developing scalable Apache spark ETL based Data processing pipelines. Strong command line knowledge in Unix Linux with Shell scripting using Bash Korn shell or Perl and File processing using awk scripts. Expertise in SQL querying and complex joins. Implementing comprehensive Spark based Data validation frameworks transforming large volumes of financial data within the Project lifecycle. Expertise with complex Data workflows with Apache Air Flow managing task dependencies SLAs etc to ensure timely data delivery and corresponding automated validation controls. Strong Analytical skills and expertise on Spark SQL for Data analysis and validation ensuring the delivery of clean query ready datasets for business consumption. Expertise in Data quality checks and monitoring.

Mandatory Skills : Apache Spark, Databricks Python, Spark SQL