Software Engineer (3) - Core Engineer 3 #1034911
Job Description:
GCP AI Data Engineer
 
Skills Required:
TERRAFORM, Big Query, AIRFLOW, POSTGRES, Data Flow, Data Fusion, Dataproc, Cloud Storage
 
Experience Required:
2+ years' experience
 
Education Required:
Bachelor's Degree
 
Additional Information :
Design and build GCP data driven solutions for enterprise data warehouse and data lakes
Design and implement scalable data architectures on GCP, including data lakes, warehouses, and real-time processing systems.
Utilize ML Services like Vertex AI.
Develop and optimize ETL/ELT pipelines using tools like Python, SQL, and streaming technologies (e.g., Kafka, Apache Airflow).
Architect Historical and Incremental Loads and Refine Architecture on an ongoing basis
Manage and optimize data storage, partitioning, and clustering strategies for high performance and reliability, utilizing services such as BigQuery, Spark, Pub/Sub, and Object Storage.
Collaborate with data scientists, Data engineers, and other stakeholders to understand data needs and deliver solutions aligned with business objectives, security, and data governance.
Automate infrastructure and deployments using Infrastructure as Code (IaC) using tools like Terraform and CI/CD practices (e.g., Tekton) to ensure reliability and scalability.
Operationalize machine learning models by building data infrastructure and managing structured and unstructured data, supporting AI/ML/LLM workflows, including data labeling, classification, and document parsing.
Monitor and troubleshoot data pipelines and systems to identify and resolve issues related to performance, reliability, and cost-effectiveness.
Document data processes, pipeline designs, and architecture, contributing to knowledge transfer and system maintenance.