101: Interesting Features of Kubeflow Pipeline in GCP

What is Docker and Kubernetes?

Hypervisor makes an operating system independent of the underlying hardware. So, on a single hardware, multiple operating systems can be hosted. It stands for the concept of Virtual machines. Container bundles the application code, runtimes, dependencies, and any other files system object required for execution of the code. Multiple containers dedicated to individual applications can be executed on a single operating system (which can be a physical machine or virtual machine).

Docker is a software that is responsible for building, running, and managing containers on machines or on the cloud. A docker container image or docker image is a template or package that actually includes application code and its dependencies. These images are distributed through different registries such as Docker Hub, Container Registry by GCP, etc. The container is the running instance of an image. So, the docker image is created once, but using that image, multiple containers can be created wherever an application needs to be deployed or executed. Docker is responsible for packaging and distributing applications in the form of containers. A set of instructions required for creating a docker image is mentioned in the Dockerfile. Kubernetes is a container orchestration tool. When multiple containers run in a cluster of machines, they are managed by Kubernetes. Kubernetes uses docker for deploying containerized applications.

MLOps

MLOps is a lifecycle management for developing machine learning models. It focuses on managing resources, code, data, time, and quality to successfully create and serve the ML model. MLOps involves simplifying and automating the complete machine learning workflow, fostering cooperation between data scientists and operations teams, and upholding the dependability and expandability of machine learning systems.

What is a Machine Learning pipeline?

A machine learning pipeline/workflow includes an organized and automated series of steps for processing data and training ML models, which simplifies the purpose of the development and deployment of ML models. The main objective of a machine learning pipeline is to optimize the entire process of building and deploying machine learning models, ensuring greater efficiency, reproducibility, and ease of management.

Three important phases of the machine learning lifecycle are the discovery phase, development phase, and deployment phase. The machine learning pipeline majorly focuses on preparing data, training and evaluating machine learning models, feature engineering, and deploying ML models. The machine learning pipeline is considered to be part of MLOps. MLOps is a broader term.

Developing ML workflow is an iterative process.

What is Kubeflow

While building an ML model, the developer is hardly aware of the infrastructure needed for developing and deploying the ML model. Kubeflow is an open-source machine-learning framework that makes it easy to develop, deploy, manage, and orchestrate a machine-learning pipeline on the Kubernetes cluster. As we know, Kubernetes stands as a widely adopted platform for container orchestration, and Kubeflow harnesses the functionalities of Kubernetes to establish a streamlined ecosystem for constructing and launching machine learning applications. Kubeflow is intended to be used by data scientists keen to build and experiment with ML pipelines. For having a flexible pipeline for all stages of the machine learning pipeline, Kubeflow is a great option. Kubeflow can be run anywhere the way Kubernetes clusters; thus, applications built on Kubeflow are portable across clouds and on-premise environments.

Kubeflow pipeline in GCP

GCP Veterx AI is a machine learning platform for training and deploying ML models. Vertex AI pipelines allow the user to automate and monitor ML systems by orchestrating ML workflow. Vertex AI pipelines support the execution of pipelines built using Kubeflow or Tensorflow Extended. While using Kubeflow pipelines(KFP) alone, the user is responsible for creating & managing the Kubernetes cluster. But Vertex AI pipelines is the serverless solution for the execution of KFP. Alternatively, Kubeflow pipelines can be easily deployed on Google Kubernetes Engine (GKE), too.

Conclusion

One of the biggest challenges in ML model development is continuously evaluating and updating the model due to environmental changes. An ML pipeline can be designed to be automated, reused, and easily scaled. Kubeflow Pipelines framework allows engineers to develop one of its kind. Vertex AI is the serverless solution for executing such ML pipelines designed using the Kubeflow Pipelines framework.

Go through the Exam Course Outline:

• GCP Professional Machine Learning Engineer certification

Professional ML Engineer Certification  |  Learn  |  Google Cloud

Propel Your Preparation with Exam Learning Paths

• For training guidance (GCP Associate Cloud Engineer)

Google Cloud Associate Engineer Certification Training | CloudThat Training

• For training guidance (GCP Professional Data Engineer)

Google Cloud Professional Data Engineer Certification Training | CloudThat Training

• For training guidance (GCP Professional Machine Learning Engineer)

Google Cloud Machine Learning Engineer Certification Training | CloudThat Training

Exploring the additional resources

• Introduction to Vertex AI  |  Google Cloud
• Getting Started with Kubeflow Pipelines | Google Cloud Blog
• code-snippets/ml/kubeflow-pipelines at master · amygdala/code-snippets · GitHub

Further Reading

Getting started with Kubeflow Pipelines | Google Cloud Blog

The most insightful stories about Kubeflow – Medium

About CloudThat