AWS Databricks Lead

AWS services and databrick Location Hartford CT on-site Proficient in leveraging Databricks features such as o Delta Lake o Unity Catalog o Scalable data processing frameworks Strong understanding of AWS cloud services including o Amazon S3 o AWS Lambda o Amazon SNS o AWS Step Functions o AWS Glue Skilled in job cost optimization techniques to o Minimize runtimes o Improve system performance Experience with o Python core libraries o PySpark for data engineering tasks Good analytical and problem-solving skills with expertise in o SQL query development o Stored procedures o Creating data marts and views in RDBMS Strong understanding of o ETL concepts o End-to-end data pipeline architecture o Data warehouse and reporting layers Familiarity with CICD pipelines and deployment tools such as o Terraform Experience with version control systems o GitHub or equivalent platforms Understanding of o Agile methodology o Software Development Life Cycle SDLC Excellent communication and presentation skills with the ability to collaborate effectively with stakeholders Roles Responsibilities Collaborate with business users and stakeholders to understand challenges and provide effective Databricksbased solutions Research and evaluate new Databricks features conduct Proof of Concepts POCs and develop best practice documentation Provide support for Sparkrelated issues and participate in code reviews offering recommendations for improvement Lead and mentor junior team members guiding them in development and troubleshooting activities Manage midsized teams and oversee delivery of work ensuring quality and timely completion Take ownership of assigned tasks and contribute effectively in an Agile environment Participate in solution design discussions and prepare designdocumentation artifacts based on requirements Drive job cost optimization and performance tuning initiatives to enhance system efficiency Support CICD processes and infrastructure automation to streamline deployments Actively participate in Agile ceremonies and contribute to continuous improvement of team collaboration and productivity

Mandatory Skills : AWS AI Services, AWS EMR, AWS Glue