Course Wiki
Using an Orchestrator — MLOps Zoomcamp Module 3
Now that we converted the notebook into a python script, we can use an orchestrator to turn the script into a production pipeline.
Related Pages
MLOps Zoomcamp › Module 3: Orchestration & ML Pipelines › Using an Orchestrator
Notes
Now that we converted the notebook into a python script, we can use an orchestrator to turn the script into a production pipeline.
There’s no video for this unit, but you can use ChatGPT to help you with this.
Step 1: Choosing the Tool
For that you first need to choose an orchestrator. For example:
- Airflow
- Prefect
- Dagster
- Kestra
- Mage
- or some other tool
Step 2: Running the Tool
- Configure the tool to run locally
- Run the simplest “hello world” workflow
Step 3: Orchestrating the Workflow
- Get the code from the previous unit (see code)
- Use the tool to orchestrate the steps in the pipeline
Step 4: Parametrizing the Workflow
- Schedule the workflow to run monthly
- The train data should be from two months ago
- The validation data - one month ago
Step 5: Backfilling
- Learn to run the workflow for some of the past months
Step 6: Deployment (optional)
- Learn to deploy the tool to the cloud
Resources
For guidance, you can refer to past cohorts of the course:
- Prefect - 2022 and 2023
- Mage - 2024
You can also rely on ChatGPT or similar tools. They are very helpful.
Key concepts
Related notes
- mlops-m01-homework
- mlops-m01-optional-training-a-ride-duration-prediction-mod
- mlops-m02-getting-started-with-mlflow
- mlops-m02-homework
- mlops-m03-homework
- mlops-m03-introduction-to-ml-pipelines
- mlops-m03-turning-the-notebook-into-a-python-script
- mlops-m04-mlops-zoomcamp-4-6-batch-scoring-with-mage
- mlops-m04-optional-streaming-deploying-models-with-kinesis
- mlops-m04-three-ways-of-deploying-a-model
- mlops-m04-web-services-deploying-models-with-flask-and-doc
- mlops-m05-debugging-with-test-suites-and-reports
- mlops-m06-homework
- mlops-m07-course-project