Metaflow

Metaflow is an open-source framework for real-life ML, AI, and data science projects. It lets developers use any Python libraries for models and business logic, with library dependency management locally and in the cloud. Workflows are written in plain Python and can be developed and debugged locally, then deployed to production without changes. Metaflow automatically tracks and stores variables inside flows for experiment tracking and debugging, and can scale execution to the cloud using GPUs, multiple cores, and large memory. It can be deployed on AWS (EKS and S3, or AWS Batch and AWS Step Functions), Azure (AKS and Azure Blob Storage), Google Cloud (GKE and Google Cloud Storage), or a custom Kubernetes cluster. The page states it was originally developed at Netflix, open-sourced in 2019, and is used by hundreds of companies. Recent releases mentioned include the spin command, recursive and conditional steps, custom decorators, uv support, a one-click local development stack, the @checkpoint decorator, and a Config object.

Key features

• Use any Python libraries for models and business logic with dependency management locally and in the cloud
• Deploy workflows to production with a single command and integrate with other systems through events
• Automatic tracking and storage of variables inside flows for experiment tracking and debugging
• Create workflows in plain Python, developed and debugged locally and deployed without changes
• Scale execution to the cloud using GPUs, multiple cores, and large amounts of memory
• Access data from data warehouses and version data as it flows across steps
• Deploy on AWS, Azure, Google Cloud, or a custom Kubernetes cluster, including on-premise
• Make flows react automatically to updating data and other events
• Recent releases include the spin command, recursive and conditional steps, custom decorators, uv support, a one-click local development stack, @checkpoint decorator, Config object, and AWS Trainium support

Best suited for

• ML/AI engineers
• Data scientists
• Teams running experiments that need to move from laptop to cloud or production
• Organizations with existing cloud or Kubernetes infrastructure

Pricing

Not stated on the website