ETL- Data Engineer

Job Category: WFO
Job Type: Full Time
Job Location: Gurgaon

Experience: 6 to 12years

Notice Period: Immediate to 30 days.

Job Description:

Responsibilities

  • Design, Develop and maintain ETL pipelines using Pyspark in Azure Databricks using delta tables.
  • Create build from Github and release pipeline for Ingestion and Databricks using Azure Devops / Harness
  • Monitor Performance of ETL Jobs, resolve any issue that arose and improve the performance metrics as needed.
  • Diagnose system performance issue related to data processing and implement solution to address them.
  • Collaborate with other teams to ensure successful integration of data pipelines into larger system architecture requirement.

Skills

  • Should have Strong background in Software development with experience in ingest, transform and store data from large datasets using Pyspark in Azure Databricks with strong knowledge on distributed computing concepts.
  • Must have hands on experience in designing and developing ETL Pipelines in Pyspark in Azure Databricks with strong python scripting exposure like list comprehensions, Dictionary variables etc.
  • Must have minimum 5 years of exposure and good proficiency in data warehousing concepts.
  • Proficient in SQl and database Design concepts.
  • Hands on experience with good proficiency in Delta table and delta file operations like merge, Insert override, Partition overrides etc.
  • Hands on Experience in CICD in Azure DevOps/Harness and ADF/Stone branch for orchestration.
  • Knowledge of Azure cloud computing platform with Azure Synapse, ADLS.
  • Knowledge on GitHub and build management.
  • Passion for data and experience working within a data driven organization
  • Excellent presentation, communication (oral & written), and relationship building skills, across all levels of management

 

 

 

 

Apply for this position

Allowed Type(s): .pdf, .doc, .docx