January 22nd 2022 • 1 minute read

Arc SQL MI - increase storage size of PVC

Kubernetes supports the ability to increase the size of a PVC see here - this is great! Kubernetes does not support the ability to increase the size of a PVC that's attached to a StatefulSet see here - not so great.

That means if your SQL MI - which is part of a StatefulSet - mounted volumes run out of size, or need more size - you need to follow these steps:

Scale StatefulSet to 1 → 0
Change 4 PVCs from 5Gi → 20Gi
PVC will enter Resizing and then show FileSystemResizePending
Scale StatefulSet to 0 → 1
PVC resize completes

We start with a SQL MI GP that has 5Gi to begin:

We will also make a change to the CRD after the scaling (switch SQL Server Agent to Off) - to make sure Data Controller is continuing to work against the modified StatefulSet:

Get the names of PVCs used by MI

$mi = "sql-ad-yes-1-0"
kubectl get pvc -n arc | grep $mi

NAME                                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
backups-ks4has8em2wnhf9peu69c904-sql-ad-yes-1-0      Bound    pvc-608e416c-4a31-4b22-bda3-306ea5fcf06e   5Gi        RWO            managed-premium   13m
data-ks4has8em2wnhf9peu69c904-sql-ad-yes-1-0         Bound    pvc-8ec07daa-dc1c-4a31-9f9c-93c558bbb250   5Gi        RWO            managed-premium   13m
datalogs-ks4has8em2wnhf9peu69c904-sql-ad-yes-1-0     Bound    pvc-67467244-5cc2-4c34-858f-fcbaf2326bdf   5Gi        RWO            managed-premium   13m
logs-ks4has8em2wnhf9peu69c904-sql-ad-yes-1-0         Bound    pvc-25913f00-32eb-450e-9378-bf4a0c999476   5Gi        RWO            managed-premium   13m

Check Azure disks before scaling

We can check that the underlying Azure Disks serving the Volume are in Attached state:

$rg_aks="raki-ad-direct-1"
$aksName="aks-cni"
$rg_managed=$(az aks show --resource-group $rg_aks --name $aksName --query "nodeResourceGroup" --output tsv)

# backups
az disk show --name "kubernetes-dynamic-pvc-608e416c-4a31-4b22-bda3-306ea5fcf06e" --resource-group $rg_managed --query "[diskState, diskSizeGb]"
# data
az disk show --name "kubernetes-dynamic-pvc-8ec07daa-dc1c-4a31-9f9c-93c558bbb250" --resource-group $rg_managed --query "[diskState, diskSizeGb]"
# datalogs
az disk show --name "kubernetes-dynamic-pvc-67467244-5cc2-4c34-858f-fcbaf2326bdf" --resource-group $rg_managed --query "[diskState, diskSizeGb]"
# logs
az disk show --name "kubernetes-dynamic-pvc-25913f00-32eb-450e-9378-bf4a0c999476" --resource-group $rg_managed --query "[diskState, diskSizeGb]"

Disks are 5 GB each to serve current claim

Scale StatefulSet to 1 → 0

kubectl scale statefulsets sql-ad-yes-1 -n arc --replicas=0

Change 4 PVCs from 5Gi → 20Gi

$payload = '{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"20Gi\"}}}}'

kubectl patch pvc backups-ks4has8em2wnhf9peu69c904-sql-ad-yes-1-0 -n arc --type merge --patch $payload
kubectl patch pvc data-ks4has8em2wnhf9peu69c904-sql-ad-yes-1-0 -n arc --type merge --patch $payload
kubectl patch pvc datalogs-ks4has8em2wnhf9peu69c904-sql-ad-yes-1-0 -n arc --type merge --patch $payload
kubectl patch pvc logs-ks4has8em2wnhf9peu69c904-sql-ad-yes-1-0 -n arc --type merge --patch $payload

PVC will enter Resizing and then show FileSystemResizePending

kubectl get pvc backups-ks4has8em2wnhf9peu69c904-sql-ad-yes-1-0 -n arc -o=jsonpath='{.status.conditions}'
kubectl get pvc data-ks4has8em2wnhf9peu69c904-sql-ad-yes-1-0 -n arc -o=jsonpath='{.status.conditions}'
kubectl get pvc datalogs-ks4has8em2wnhf9peu69c904-sql-ad-yes-1-0 -n arc -o=jsonpath='{.status.conditions}'
kubectl get pvc logs-ks4has8em2wnhf9peu69c904-sql-ad-yes-1-0 -n arc -o=jsonpath='{.status.conditions}'

Observe Azure disks are unattached and scaled up now:

Scale StatefulSet Up

kubectl scale statefulsets sql-ad-yes-1 -n arc --replicas=1

Verify connectivity

Once the SQL MI pod comes back up - we’re able to connect once again - and note the SQL Agent remains on:

Change SQL Agent to off via CRD

$payload = '{\"spec\":{\"settings\":{\"sqlagent\":{\"Enabled\":false}}}}'
kubectl patch sqlmi sql-ad-yes-1 -n arc --type merge --patch $payload

kubectl get sqlmi sql-ad-yes-1 -n arc -o=jsonpath='{.spec.settings.sqlagent.Enabled}'

This triggers a restart of the Pod by the Controller:

Which shows that the CRD continues to be functional.

Arc SQL MI - increase storage size of PVC

Get in touch 👋