Velero for Kubernetes: Backup & Restore Stateful Workloads with Restic for Velero

This article is part of a series of blog posts on using Velero for Kubernetes backup, restore, migration & disaster recovery.

All articles in this series explore Velero in the context of AWS Elastic Kubernetes Service (EKS).

Stay tuned as we publish more articles in the coming weeks & months. Here’s a sneak preview of what’s to come:

  1. An Introduction to Velero for Kubernetes Backup & Restore
  2. Velero for Kubernetes Backup: Install & Configure
  3. Backup & Restore Stateless Workloads with Velero for Kubernetes
  4. Velero for Kubernetes: Backup & Restore Stateful Workloads with AWS EBS Snapshots
  5. Velero for Kubernetes: Backup & Restore Stateful Workloads with Restic for Velero
  6. Monitoring Velero Kubernetes Backups & Automated Alerting for Backup Failures

Introduction

In the earlier post in this series we saw how to backup & restore stateful WordPress using AWS EBS snapshots.

In this post, we will see how to accomplish the same without snapshots, using file-system level backups alone.

But why would you want to do that?

File system backups instead of snapshots can be useful when:

  1. You are migrating workloads across AWS regions. Use S3 backups instead of migrating snapshots across regions.
  2. Migrating Kubernetes workloads across cloud providers, or from on-prem to the cloud.

Install Restic

Restic is another open-source project that provides file system backups for Velero.

Restic is not installed by default with Velero. Install it as follows:

velero install \
--backup-location-config region=ap-south-1 \
--bucket S3_BUCKET \
--default-volumes-to-restic \
--namespace velero \
--plugins velero/velero-plugin-for-aws \
--provider aws \
--secret-file ~/velero-credentials \
--use-restic true \
--use-volume-snapshots false \
--wait

Replace S3_BUCKET above with the name of your S3 bucket.

This command creates a number of resources in our cluster:

CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
CustomResourceDefinition/backups.velero.io: created

CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
CustomResourceDefinition/backupstoragelocations.velero.io: created

CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
CustomResourceDefinition/deletebackuprequests.velero.io: created

CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
CustomResourceDefinition/downloadrequests.velero.io: created

CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumebackups.velero.io: created

CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumerestores.velero.io: created

CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource client
CustomResourceDefinition/resticrepositories.velero.io: created

CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
CustomResourceDefinition/restores.velero.io: created

CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
CustomResourceDefinition/schedules.velero.io: created

CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
CustomResourceDefinition/serverstatusrequests.velero.io: created

CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
CustomResourceDefinition/volumesnapshotlocations.velero.io: created

Waiting for resources to be ready in cluster...

Namespace/velero: attempting to create resource
Namespace/velero: attempting to create resource client
Namespace/velero: created

ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: attempting to create resource client
ClusterRoleBinding/velero: created

ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: created

Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: created

BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: created

VolumeSnapshotLocation/default: attempting to create resource
VolumeSnapshotLocation/default: attempting to create resource client
VolumeSnapshotLocation/default: created

Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: created

DaemonSet/restic: attempting to create resource
DaemonSet/restic: attempting to create resource client
DaemonSet/restic: created

Waiting for Velero deployment to be ready.
Waiting for Velero restic daemonset to be ready.

Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

Install Velero from CLI

A quick aside.

Since this is the first time in this series we’re installing Velero using its own CLI, instead of our usual way of Helm chart, let’s take a minute to explore the different configurations of Velero installation from CLI.

No Restic. Only AWS EBS snapshots (with IAM user auth):

velero install \
--backup-location-config region=ap-south-1 \
--bucket S3_BUCKET \
--namespace velero \
--plugins velero/velero-plugin-for-aws \
--provider aws \
--secret-file ~/velero-credentials \
--snapshot-location-config region=ap-south-1 \
--use-volume-snapshots true \
--wait

No Restic. IAM role for auth:

velero install \
--backup-location-config region=ap-south-1 \
--bucket S3_BUCKET \
--namespace velero \
--plugins velero/velero-plugin-for-aws \
--pod-annotations iam.amazonaws.com/role=arn:aws:iam::<AWS_ACCOUNT_ID>:role/<VELERO_ROLE_NAME> \
--provider aws \
--snapshot-location-config region=ap-south-1 \
--use-volume-snapshots true \
--wait

Restic Components

Once Restic is installed & running, you should be able to see its pod(s) & DaemonSet:

> kubectl get all --namespace velero

NAME                          READY   STATUS    RESTARTS   AGE
pod/restic-dsmrm              1/1     Running   0          91s
pod/velero-75bcf46c6d-tq5lr   1/1     Running   0          91s

NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/restic   1         1         1       1            1           <none>          92s

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/velero   1/1     1            1           92s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/velero-75bcf46c6d   1         1         1       92s

Stateful Workload

As a quick refresher, here’s what our stateful WordPress workload looks like after its installed from a Bitnami Helm chart:

> kubectl get all --namespace wordpress

NAME                             READY   STATUS    RESTARTS   AGE
pod/wordpress-7ff458b78c-9d5dv   1/1     Running   0          81m
pod/wordpress-mariadb-0          1/1     Running   0          81m

NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP                                                                PORT(S)                      AGE
service/wordpress           LoadBalancer   10.100.42.227    aad879576dc5c441192560e71ce162d0-1013660622.ap-south-1.elb.amazonaws.com   80:31746/TCP,443:32297/TCP   81m
service/wordpress-mariadb   ClusterIP      10.100.217.251   <none>                                                                     3306/TCP                     81m

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/wordpress   1/1     1            1           81m

NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/wordpress-7ff458b78c   1         1         1       81m

NAME                                 READY   AGE
statefulset.apps/wordpress-mariadb   1/1     81m

For instructions on how to create this workload, see the previous article in this series.

Backup using Restic

Creating a Velero backup with or without Restic looks pretty much the same:

> velero backup create wordpress \
    --include-namespaces wordpress

Backup request "wordpress" submitted successfully.

Run `velero backup describe wordpress` or
`velero backup logs wordpress` for more details.

There is one obvious difference though. Instead of creating snapshots, Restic has uploaded all volume data to S3:

> velero backup describe wordpress --details

Name:         wordpress
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.21.5-eks-bc4871b
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=21+

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  wordpress
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2022-01-23 19:00:57 +0530 IST
Completed:  2022-01-23 19:01:16 +0530 IST

Expiration:  2022-02-22 19:00:57 +0530 IST

Total items to be backed up:  32
Items backed up:              32

Resource List:
  apiextensions.k8s.io/v1/CustomResourceDefinition:
    - backups.velero.io
    - deletebackuprequests.velero.io
    - downloadrequests.velero.io
  apps/v1/ControllerRevision:
    - wordpress/wordpress-mariadb-cd78bc67d
  apps/v1/Deployment:
    - wordpress/wordpress
  apps/v1/ReplicaSet:
    - wordpress/wordpress-7ff458b78c
  apps/v1/StatefulSet:
    - wordpress/wordpress-mariadb
  discovery.k8s.io/v1/EndpointSlice:
    - wordpress/wordpress-mariadb-hmvnb
    - wordpress/wordpress-nknxt
  v1/ConfigMap:
    - wordpress/kube-root-ca.crt
    - wordpress/wordpress-mariadb
  v1/Endpoints:
    - wordpress/wordpress
    - wordpress/wordpress-mariadb
  v1/Namespace:
    - wordpress
  v1/PersistentVolume:
    - pvc-37112caf-03ec-4ddf-b0e7-0b6a05ebb2be
    - pvc-b3837378-941f-4a3d-abe6-44aeadf0158b
  v1/PersistentVolumeClaim:
    - wordpress/data-wordpress-mariadb-0
    - wordpress/wordpress
  v1/Pod:
    - wordpress/wordpress-7ff458b78c-9d5dv
    - wordpress/wordpress-mariadb-0
  v1/Secret:
    - wordpress/default-token-jfg2q
    - wordpress/sh.helm.release.v1.wordpress.v1
    - wordpress/wordpress
    - wordpress/wordpress-mariadb
    - wordpress/wordpress-mariadb-token-tnm52
  v1/Service:
    - wordpress/wordpress
    - wordpress/wordpress-mariadb
  v1/ServiceAccount:
    - wordpress/default
    - wordpress/wordpress-mariadb
  velero.io/v1/Backup:
    - wordpress/wordpress
  velero.io/v1/DeleteBackupRequest:
    - wordpress/wordpress-wxjx9
  velero.io/v1/DownloadRequest:
    - wordpress/wordpress-67d3145e-bec2-4234-9690-4874a30d4098

Velero-Native Snapshots: <none included>

Restic Backups:
  Completed:
    wordpress/wordpress-7ff458b78c-9d5dv: wordpress-data
    wordpress/wordpress-mariadb-0: data

Next Steps

The steps to restore a Restic-based backup are the same as the steps we used in the last article for a snapshot-based backup.

Conclusion

In this article, we explored how & why one might use file system backups instead of snapshots with Velero.

About the Author ✍🏻

Harish KM is a Principal DevOps Engineer at QloudX & a top-ranked AWS Ambassador since 2020. 👨🏻‍💻

With over a decade of industry experience as everything from a full-stack engineer to a cloud architect, Harish has built many world-class solutions for clients around the world! 👷🏻‍♂️

With over 20 certifications in cloud (AWS, Azure, GCP), containers (Kubernetes, Docker) & DevOps (Terraform, Ansible, Jenkins), Harish is an expert in a multitude of technologies. 📚

These days, his focus is on the fascinating world of DevOps & how it can transform the way we do things! 🚀

Leave a Reply

Your email address will not be published. Required fields are marked *