Senior Data Engineer

  • Metro Vancouver Regional District
  • Insight Global

Must-haves

Masters Degree in Business Administration, Information Systems or equivalent.

Five years' recent, related experience including technical experience in working with databases and decision support tools, preferably in a health care setting or equivalent industry/certification experience (e.g. Databricks Data Engineer Associate, Databricks Data Engineer Professional, Microsoft Azure Data Engineer Associate).

Five years experience in Information Technology with focus in database development, warehousing, and/or data engineering.

Expertise using the Databricks platform and understanding the concepts of Databricks SQL, Delta Lake, cluster management.

Experience in optimization of pipelines and processes of Databricks platform.

Strong proficiency in Databricks SQL, Python, Scala with an ability to write efficient and scalable code for data processing/transformation tasks.

Experience building and optimizing data pipelines (ETL/ELT).

Strong understanding of DevOps and MLOps practices focusing on version control (Git), CI/CD pipelines, and automated testing frameworks.

Experience using big data processing frameworks such as Apache Spark & Hadoop.

Strong understanding of Spark RDD's, data frames, and API's for data manipulation.

Knowledge of data modelling and data warehousing principles.

Knowledge using cloud data warehousing solutions (e.g. Azure SQL, Synapse Analytics, Snowflake).

Advanced analytical and problem solving skills.

Plusses

Day-to-Day

Insight Global is looking for a Data Engineer to join the Data & Solutions Infrastructure team for one of BC's leading health authorities. This resource will be instrumental in maintaining and advancing data infrastructure. Take charge of both cloud-based and local data systems, ensuring their continuous improvement and operational stability.

This resource will be responsible for engineering scalable and efficient data integration systems that support extensive analysis, model crafting, validation, and deployment. Work alongside diverse teams to pinpoint and realize forward-thinking solutions that cater to the informational requisites of the health authority.