Machine Learning Pipelines: Setting Up On-premise Kubernetes

This multi-part tutorial guides you through building an on-premise machine learning pipeline using open-source tools. It's perfect for startups on a budget, emphasizing control and cost predictability.

Key Advantages:

• Cost-Effective: Avoids cloud service expenses.
• Customizable: Offers greater control over your ML pipeline.
• Accessible: Simplifies Kubernetes setup with Rancher Kubernetes Engine (RKE), requiring only basic Docker and Linux skills.
• Practical Approach: A hands-on guide for ML pipeline development, ideal for beginners.

Why Go On-Premise?

Many assume cloud storage (AWS S3, Google Cloud Storage), but this series demonstrates building a functional pipeline using existing servers, ideal for resource-constrained environments. This approach provides a safe learning environment without unpredictable costs.

Target Audience:

This guide is for software engineers or individuals building production-ready ML models, especially those new to ML pipelines.

Prerequisites:

Familiarity with Linux (Ubuntu 18.04 recommended) and basic Docker knowledge are helpful. Deep Kubernetes expertise isn't required.

Tools Used:

• Docker
• Kubernetes
• Rancher (with RKE)
• KubeFlow/KubeFlow Pipelines (covered in later parts)
• Minio
• TensorFlow (covered in later parts)

Phase 1: Easy Kubernetes Installation with Rancher

This section focuses on the challenging task of Kubernetes installation, simplified with RKE.

Step 0: Machine Preparation:

You'll need at least two Linux machines (or VMs with bridged networking and promiscuous mode enabled) on the same LAN, designated as 'master' and 'worker'. Note that using VMs limits GPU access and performance.

Essential machine details (IP addresses, usernames, SSH keys) are needed for configuration. A temporary hostname (e.g., rancher-demo.domain.test) will be used for this tutorial. Modify your /etc/hosts file accordingly on both machines to reflect this hostname and the IP addresses. If using VMs, add the hostname entry to your host machine's /etc/hosts file as well for browser access.

Step 1: Obtaining the RKE Binary:

Download the appropriate RKE binary for your OS from the GitHub release page, make it executable, and move it to /usr/local/bin. Verify the installation by running rke.

Step 2: Preparing Linux Hosts:

On all machines:

1. Install Docker: Install Docker CE (version 19.03 or later) using the provided commands. Verify the installation and add your user to the docker group. Log out and back in for the group changes to take effect.
2. SSH Keys: Set up SSH keys on the master node and copy the public key to all worker nodes. Configure SSH servers to allow port forwarding (AllowTcpForwarding yes in /etc/ssh/sshd_config).
3. Disable Swap: Disable swap using sudo swapoff -a and comment out swap entries in /etc/fstab.
4. Apply Sysctl Settings: Run sudo sysctl net.bridge.bridge-nf-call-iptables=1.
5. DNS Configuration (Ubuntu 18.04 ): Install resolvconf, edit /etc/resolvconf/resolv.conf.d/head, add nameservers (e.g., 8.8.4.4 and 8.8.8.8), and restart resolvconf.

Step 3: Cluster Configuration File:

On the master node, use rke config to create a cluster.yml file. Provide the necessary information (IP addresses, hostnames, roles, SSH key paths, etc.).

Step 4: Bringing Up the Cluster:

Run rke up on the master node to create the Kubernetes cluster. This process takes some time.

Step 5: Copying Kubeconfig:

Copy kube_config_cluster.yml to $HOME/.kube/config.

Step 6: Installing Kubectl:

Install kubectl on the master node using the provided commands. Verify the installation by running kubectl get nodes.

Step 7: Installing Helm 3:

Install Helm 3 using the provided command.

Step 8: Installing Rancher using Helm:

Add the Rancher repository, create a cattle-system namespace, and install Rancher using Helm. Monitor the deployment status.

Step 9: Setting up Ingress (for access without a load balancer):

Create an ingress.yml file (adapting the host to your chosen hostname) and apply it using kubectl apply -f ingress.yml.

Step 10: Accessing Rancher:

Access the Rancher UI at https://rancher-demo.domain.test (or your chosen hostname), create a password, and set the domain name.

Step 11: Installing cert-manager:

Install cert-manager (version v0.9.1) using the provided commands. Monitor the pods to ensure they are running.

This completes the Kubernetes cluster setup. The next part of the series will cover installing Kubeflow.

Frequently Asked Questions (FAQs):

The FAQs section provides comprehensive answers to common questions about on-premise Kubernetes setup, covering benefits, comparisons with cloud-based solutions, challenges, security considerations, migration strategies, hardware requirements, scaling, Kubernetes operators, machine learning workloads, and performance monitoring.

The above is the detailed content of Machine Learning Pipelines: Setting Up On-premise Kubernetes. For more information, please follow other related articles on the PHP Chinese website!