Race condition between dynamic PVC creation and deployment scheduling

What happened:

I have a 19 node Kubernetes bare metal cluster set up using kubeadm.
I installed the csi-driver-nfs 4.2.0 using helm according to the instructions provided in this repository.
Only the default values were used.

I then created a StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: nfs-csi
provisioner: nfs.csi.k8s.io
parameters:
  server: RETRACTED
  share: /raid/kube_storage/
  # csi.storage.k8s.io/provisioner-secret is only needed for providing mountOptions in DeleteVolume
  # csi.storage.k8s.io/provisioner-secret-name: "mount-options"
  # csi.storage.k8s.io/provisioner-secret-namespace: "default"
reclaimPolicy: Delete
volumeBindingMode: Immediate
#mountOptions:
# - nconnect=8 # only supported on linux kernel version >= 5.3
# - nfsvers=3

I then tried testing the setup with an example adapted from this repository for dynamic provisioning:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-deployment-nfs
spec:
  accessModes:
    - ReadWriteMany # In this example, multiple Pods consume the same PVC.
    # - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-nfs
spec:
  replicas: 1
  selector:
    matchLabels:
      name: deployment-nfs
  template:
    metadata:
      name: deployment-nfs
      labels:
        name: deployment-nfs
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
        - name: deployment-nfs
          image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
          command:
            - "/bin/bash"
            - "-c"
            - set -euo pipefail; while true; do echo $(hostname) $(date) >> /mnt/nfs/outfile; sleep 1; done
          volumeMounts:
            - name: nfs
              mountPath: "/mnt/nfs"
      volumes:
        - name: nfs
          persistentVolumeClaim:
            claimName: pvc-deployment-nfs

This results in the following error message in kubectl describe pod deployment-nfs-65bf4f7c4c-rrtv8

Warning  FailedScheduling  14m   default-scheduler  0/19 nodes are available: 19 pod has unbound immediate PersistentVolumeClaims. preemption: 0/19 nodes are available: 19 Preemption is not helpful for scheduling.
  Normal   Scheduled         14m   default-scheduler  Successfully assigned default/deployment-nfs-65bf4f7c4c-rrtv8 to conway013
  Normal   Pulled            14m   kubelet            Container image "mcr.microsoft.com/oss/nginx/nginx:1.19.5" already present on machine
  Normal   Created           14m   kubelet            Created container deployment-nfs
  Normal   Started           14m   kubelet            Started container deployment-nfs

If I put the definition of the PVC and the Deployment in seperate yaml files and apply the PVC first and then the Deployment, I get no error message and the deployment works fine.

This is especially a problem for installing helm charts, which also trigger the race condition (e.g. helm install --generate-name -f values.yaml -n authentik authentik/authentik).

What you expected to happen:

It should not make a difference if i apply the PVC and the Deployment in one file or in seperate files.

How to reproduce it:

Put the two definitons in one file to get the race condition.

Split the two definitions across two files to avoid the race condition.

Anything else we need to know?:

The NFS export is hosted on an FreeBSD server.

Environment:

• CSI Driver version: 4.2.0
• Kubernetes version (use kubectl version):
  Client Version: v1.24.1
  Kustomize Version: v4.5.4
  Server Version: v1.24.1
• OS (e.g. from /etc/os-release): Ubuntu 22.04 LTS
• Kernel (e.g. uname -a): 5.15.0-40-generic
• Install tools: nfs-common, kubeadm, kubectl, containerd
• Others:

[andyzhangx]

I think your agent node has some issues per below error msg, I don't think it's related to this driver:

preemption: 0/19 nodes are available: 19 Preemption is not helpful for scheduling.

if there is sth. wrong with the driver, pls get driver controller logs: https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/docs/csi-debug.md#case1-volume-createdelete-failed

The line you mention is exactly the problem.

If I split the PVC and Deployment in two distinct yaml files and apply them one after the other I do not get the warning about 0/19 nodes being availible.

This only occurs if I put both definitions in the same file, seperated by three dashes and apply them at once.

This is the log file of the controller.
csi-nfs-controller.log

If you are sure this is not related to the csi-nfs driver and you have an idea where the problem could stem from this would be helpful as well.

I'm pretty sure it's related to something in my local setup, thank you for the support.

I'm not sure what it is but if it's something relevant to the NFS CSI Driver I'll open up a more specific issue again.