Search Jobs

*Remote Contract-to-Hire Opportunity*

Lighthouse Technology Services is partnering with our client to fill their Senior Data Engineer (AWS / Python / SQL / Snowflake) position! This is a 6+ month contract-to-hire opportunity. This role can be remote within the United States with quarterly travel to Charlotte, NC. This role will be a W2 employee of Lighthouse Technology Services. 

Position Overview

We are seeking a Senior Data Engineer with 10+ years of experience to design, build, and optimize scalable data ingestion pipelines within a modern cloud data platform. This role will focus heavily on Python development, advanced SQL, and Snowflake data warehouse administration, ensuring data pipelines are performant, reliable, and cost-optimized.

The ideal candidate has deep hands-on experience building enterprise-scale data ingestion pipelines, integrating multiple source systems (databases and APIs), and implementing data quality validation. This individual will play a key role in improving existing ingestion frameworks while helping design and implement future data architecture solutions.

What You'll Be Doing:

Build & Optimize Data Pipelines

• Design, develop, and maintain scalable data ingestion pipelines that move data from various source systems into Snowflake.
• Build ingestion pipelines using Python, SQL, and modern ingestion tools such as Informatica IDMC and Fivetran.
• Connect to external APIs using Python, designing custom ingestion frameworks where pre-built tools are not sufficient.
• Implement data validation logic and quality checks within ingestion pipelines to ensure the integrity of replicated data.

Snowflake Data Warehouse Management

• Administer and optimize the Snowflake data warehouse environment, including performance tuning and cost optimization.
• Analyze and optimize query performance and data models.
• Apply best practices around Snowflake scaling strategies, including understanding and implementing vertical vs. horizontal scaling approaches.
• Monitor and improve warehouse performance to ensure pipelines run efficiently with minimal runtime and cost.

Data Replication & Integration

• Build solutions that replicate data from SQL Server and other enterprise source systems into Snowflake.
• Design ingestion frameworks to replicate data from external APIs into Snowflake using Python and external access networks.
• Leverage ingestion tools such as Fivetran where appropriate while also developing custom Python-based pipelines when needed.

Cloud Data Engineering

• Build and orchestrate data workflows using AWS services, including:
• AWS Airflow for pipeline orchestration
• S3 for storage
• CloudWatch for monitoring and debugging
• Implement robust deployment practices and integrate pipelines with CI/CD processes.

Engineering Best Practices

• Write clean, maintainable, and well-documented Python code including clear functions, comments, and modular design.
• Ensure pipelines are scalable, performant, and fault tolerant.
• Document all pipeline designs, ingestion logic, and system integrations.

What You’ll Need to Have

• 10+ years of professional experience in Data Engineering
• Strong Python development expertise with the ability to design, build, and optimize ingestion frameworks
• Advanced SQL skills, including writing efficient and performant queries
• Deep experience working with Snowflake, including administration, performance tuning, and cost optimization
• Hands-on experience building scalable data ingestion pipelines

Data Platform & Integration

• Experience with Informatica IDMC for data ingestion
• Experience using Fivetran or similar ingestion tools (must be able to explain hands-on usage)
• Experience replicating data from SQL Server or similar relational systems into Snowflake
• Experience ingesting data from external APIs using Python

Cloud & Pipeline Orchestration

• Experience working within AWS cloud environments
• Experience using AWS Airflow for scheduling and orchestration
• Familiarity with AWS CloudWatch for monitoring and debugging
• Experience with CI/CD pipeline implementation and deployment strategies

Data Engineering Expertise

• Strong understanding of ETL workflows and pipeline design
• Expertise in performance tuning for pipelines and data warehouses
• Knowledge of Snowflake scaling strategies (vertical vs. horizontal scaling)
• Experience implementing data quality validation within ingestion pipelines
• Experience documenting engineering solutions and maintaining technical documentation

Preferred / Nice to Have

• Hands-on experience with Fivetran ingestion architecture
• Experience with Lambda or additional AWS services
• Experience improving existing ingestion frameworks and platform architecture

Important: The environment is AWS-based. This role does not use Azure or Hadoop-based ecosystems, and those backgrounds alone will not translate directly to this role.

Pay Range: $84-$89/hr +

Questions about any of our jobs? Email us at [email protected] 

View all of our open jobs here: jobs.lhtservices.com