Nvidia Driver – Cuda Mismatch

When I tried nvidia-smi, I got this error:

Failed to initialize NVML: Driver/library version mismatch

As message indicates this is a version mismatch. I think this happened because of some automatic upgrade. Let us check the versions:

# cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  510.73.08  Wed May 18 20:34:14 UTC 2022
GCC version:  gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~18.04)
# sudo apt list --installed | grep nvidia
libnvidia-compute-510/bionic-updates,bionic-security,now 510.85.02-0ubuntu0.18.04.1 amd64 [installed,upgradable to: 510.85.02-0ubuntu1]
// some more similar lines with version 510.85.02

The solution was to reinstall the CUDA toolkit from here: https://developer.nvidia.com/cuda-downloads