Data Engineer

Responsabilidades e Atribuições
• Build and maintain data, reporting and analytics processes for an AWS data lake
• Manage AWS solutions and infrastructure for data, supporting developments
• Aut as an advisor to senior management on technical questions and research
• Develop auxiliary routines to monitor the environments (jobs, applications, queries)
• Help on architect decisions and in the data pipeline construction
• Develop and duplo ETL/ELT jobs, consuming data from different types of data sources
• Develop, run and maintain load tests
• Bugs and data fixes, addressing root cause analysis for continuous improvement
• Evolution of data schemas and documentation to delivery business value

REQUISITOS E QUALIFICAÇÕES
• Apache NiFi for data ingestion and data flow automation.
• 2+ years of working experience using AWS Cloud, configuration and monitoring knowledge (EC2, Security,
CloudFormation), and specially for Big Data Services SQS, S3, RDS, EC2, Athena, Glue, EMR;
• 2+ years of working experience in Datalake or DW projects
• Experience with cloud computing, as well scalability and performance problems
• Experience with Unix/Linux environments, data pipelines and database modelling
• Working experience with BigData, open standards and open source technologies
• Expert in SQL and RDBMS including performance optimizations, preferable PostgreSQL
• Knowledge on NoSQL databases like MongoDB or AWS DocumentDB.
• Practical knowledge of functional programming.
• Practical knowledge of Data Lake, S3 or Hadoop.
• Practical knowledge of Unstructured and Structured Data formats.
• Good knowledge of programming in Python.
• Distributed data processing using Apache Spark on AWS EMR.