Data Engineer, but I mainly work with/as Data/Dev/MLOps, so am I really a data engineer? Idk.
- NY Taxi Data & MLOps Pipeline: Automated data & MLOps pipeline leveraging Kubernetes and Apache Airflow. Integrates Spark, Kafka, and DBT with a focus on data quality. Tailors solutions for diverse user needs.
- Xbox Data Scraping & Analysis Pipeline: Automated data-driven project leveraging Python, Airflow, and GKE. Scrapes diverse data sources, providing insights into Xbox hardware and game data.
- Easy Expectations: A python package that abstracts away the complexity of
Great Expectationsand allow for easy no-knowledge-required implementation for basic use cases. - SchemaDiff: A python package that efficiently detects files with inconsistent schemas amidst thousands of files by reading the parquet files metadata.
- Order of The Template: A Python toolkit for parsing and processing YAML templates, capable of resolving Bash syntax environment variables and Jinja templating. It also offers schema validation functionality.