From EC2 to Kubernetes: A Practical Guide to Jenkins Migration

Migrating Jenkins from EC2 to Kubernetes can significantly improve scalability, resource utilization, and deployment flexibility. This guide walks through the entire migration process, from planning to execution.

Why Migrate to Kubernetes?

Running Jenkins on Kubernetes offers several advantages:

Dynamic Agent Scaling: Automatically spin up and down build agents based on demand
Resource Efficiency: Better utilization of compute resources
High Availability: Built-in redundancy and self-healing capabilities
Infrastructure as Code: Declarative configuration management
Cost Optimization: Pay only for what you use

Prerequisites

Before starting the migration, ensure you have:

An existing Jenkins instance running on EC2
AWS EKS cluster set up and configured
kubectl configured to access your EKS cluster
Helm 3.x installed
Understanding of Kubernetes concepts (Pods, Services, PersistentVolumes)

Migration Strategy

1. Assessment Phase

First, audit your current Jenkins setup:

# List all installed plugins
java -jar jenkins-cli.jar -s http://your-jenkins-url list-plugins

# Export Jenkins configuration
java -jar jenkins-cli.jar -s http://your-jenkins-url get-job <job-name> > job-config.xml

Document:

Installed plugins and versions
Job configurations
Credentials and secrets
Build agents configuration
Volume mounts and persistent data

2. Prepare Kubernetes Resources

Create necessary Kubernetes resources:

Namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: jenkins

PersistentVolumeClaim for Jenkins Home:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: gp3

3. Deploy Jenkins using Helm

Install Jenkins on Kubernetes using the official Helm chart:

# Add Jenkins Helm repository
helm repo add jenkins https://charts.jenkins.io
helm repo update

# Create values.yaml for customization
cat <<EOF > jenkins-values.yaml
controller:
  image: "jenkins/jenkins"
  tag: "lts"
  adminPassword: "your-secure-password"
  
  resources:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "2000m"
      memory: "4Gi"
  
  persistence:
    enabled: true
    existingClaim: jenkins-pvc
    size: 50Gi
  
  serviceType: LoadBalancer
  
  installPlugins:
    - kubernetes:latest
    - workflow-aggregator:latest
    - git:latest
    - configuration-as-code:latest

agent:
  enabled: true
  image: "jenkins/inbound-agent"
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "1000m"
      memory: "2Gi"
EOF

# Install Jenkins
helm install jenkins jenkins/jenkins -f jenkins-values.yaml -n jenkins

4. Configure Kubernetes Plugin

Configure Jenkins to use Kubernetes for dynamic agent provisioning:

Navigate to Manage Jenkins → Configure System
Scroll to Cloud section
Add a new Kubernetes cloud
Configure:
- Kubernetes URL: https://kubernetes.default
- Kubernetes Namespace: jenkins
- Jenkins URL: http://jenkins.jenkins.svc.cluster.local:8080

5. Migrate Data

Transfer Jenkins home directory data:

# On EC2 instance, create a backup
tar -czf jenkins-home-backup.tar.gz /var/lib/jenkins

# Copy to S3
aws s3 cp jenkins-home-backup.tar.gz s3://your-backup-bucket/

# From Kubernetes pod, download and extract
kubectl exec -n jenkins jenkins-0 -- bash -c \
  "aws s3 cp s3://your-backup-bucket/jenkins-home-backup.tar.gz /tmp/ && \
   tar -xzf /tmp/jenkins-home-backup.tar.gz -C /var/jenkins_home"

6. Update Pipeline Configurations

Update your Jenkinsfiles to use Kubernetes agents:

pipeline {
    agent {
        kubernetes {
            yaml '''
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: maven
    image: maven:3.8-jdk-11
    command:
    - cat
    tty: true
  - name: docker
    image: docker:latest
    command:
    - cat
    tty: true
    volumeMounts:
    - name: docker-sock
      mountPath: /var/run/docker.sock
  volumes:
  - name: docker-sock
    hostPath:
      path: /var/run/docker.sock
'''
        }
    }
    stages {
        stage('Build') {
            steps {
                container('maven') {
                    sh 'mvn clean package'
                }
            }
        }
        stage('Docker Build') {
            steps {
                container('docker') {
                    sh 'docker build -t myapp:latest .'
                }
            }
        }
    }
}

Challenges and Solutions

Challenge 1: Persistent Storage

Problem: Jenkins needs persistent storage for job configurations and build history.

Solution: Use EBS-backed PersistentVolumes with the gp3 storage class for better performance and cost efficiency.

Challenge 2: Plugin Compatibility

Problem: Some plugins may not work well in containerized environments.

Solution: Test all plugins in a staging environment first. Use Configuration as Code (JCasC) plugin to manage configurations declaratively.

Challenge 3: Network Connectivity

Problem: Build agents need to communicate with external services.

Solution: Configure proper network policies and use AWS VPC CNI for pod networking. Ensure security groups allow necessary traffic.

Challenge 4: Resource Management

Problem: Builds can consume excessive resources.

Solution: Set resource requests and limits for both controller and agents. Implement pod priority classes for critical builds.

Best Practices

Use Configuration as Code (JCasC): Manage Jenkins configuration through YAML files
Implement RBAC: Use Kubernetes RBAC for fine-grained access control
Monitor Resource Usage: Use Prometheus and Grafana for monitoring
Backup Regularly: Automate backups of Jenkins home directory to S3
Use Namespaces: Isolate Jenkins in its own namespace
Implement Pod Security Policies: Restrict container capabilities
Use Secrets Management: Store credentials in Kubernetes Secrets or AWS Secrets Manager

Performance Optimization

Use node affinity to schedule builds on appropriate nodes
Implement pod disruption budgets to ensure availability during updates
Configure horizontal pod autoscaling for agent pods
Use local SSD volumes for temporary build artifacts
Enable build caching to speed up builds

Monitoring and Troubleshooting

Monitor Jenkins on Kubernetes:

# Check Jenkins pod status
kubectl get pods -n jenkins

# View Jenkins logs
kubectl logs -n jenkins jenkins-0 -f

# Check resource usage
kubectl top pods -n jenkins

# Describe pod for detailed information
kubectl describe pod -n jenkins jenkins-0

Conclusion

Migrating Jenkins from EC2 to Kubernetes requires careful planning and execution, but the benefits in terms of scalability, resource efficiency, and operational flexibility make it worthwhile. By following this guide and implementing best practices, you can achieve a smooth migration and unlock the full potential of running Jenkins on Kubernetes.

The key to success is thorough testing in a staging environment before migrating production workloads. Start small, validate each step, and gradually migrate your pipelines to the new infrastructure.

Jenkins on Kubernetes: Migrating from EC2