/projects
Watchtower – Real-Time Build-Farm Log Pipeline
Real-time log pipeline for a build farm. Streams logs through Kafka and Flink into ClickHouse, with Airflow orchestration and LLM-based anomaly classification.
Highlights
- Designed a real-time streaming pipeline ingesting build-farm logs with Kafka and processing them with Apache Flink
- Stored and queried processed logs in ClickHouse for fast analytical queries
- Orchestrated pipeline workflows with Apache Airflow
- Integrated an LLM to classify anomalies detected in the log stream
- CI with GitHub Actions to keep the codebase tested and reliable (DataOps practices)
Technologies
PythonKafkaFlinkClickHouseAirflowLLMGitHub ActionsCI/CDDataOps