Passenger ticket sales service
Role
Python Data Engineer
Description of Project
Implementation of ETL processes, microservices, integration with API partners
Responsibilities
Development of ELT processes for Airflow.
Development of pipeline for data processing and saving to BigQuery column database. Development of data retrieval and sending to various sources: REST API, Google Cloud Storage, BigQuery, Kafka, Postgres. Implementation of data processing microservices for ML models, notification and data monitoring services using prometheus. Integration with partner services, setting up and receiving webhook notifications.
Technology Stack
Python, FastAPI, Apache Airflow, Asyncio, Multiprocessing, Pandas, Docker-compose, Google Cloud Storage, Google BigQuery, Postgresql, Confluent Cloud, Kafka
Period
09.2022 —
until now
(2 years 9 months)