/projects

IoT Sensor Data Pipeline with CI/CD

End-to-end streaming pipeline for IoT sensor data. Includes Docker/Docker Compose setup for local development.

IoT Sensor Data Pipeline with CI/CD

Highlights

  • Implemented a complete medallion architecture (bronze → silver → gold) for IoT sensor data streaming
  • Kafka single-node broker in KRaft mode as the message bus with synthetic data producer
  • SQL Server (T-SQL) persistence layer with pyodbc connection handling
  • Streamlit dashboard with auto-refresh (5s) displaying KPIs, per-sensor metrics, watermarks, and latency charts
  • Apache Airflow DAG orchestration with automated data tests between pipeline stages
  • Docker & Docker Compose for local development and containerized services (Kafka, Airflow, SQL Server)
  • CI/CD pipeline with GitHub Actions (linting, tests, automated deployment to Oracle Cloud)

Technologies

PythonKafkaSQL ServerAirflowDockerGitHub ActionsCI/CDDataOps
View on GitHub