Slow execution time for CUDA initialization in Azure Batch VM
I have an issue of slow initialization time for running some CUDA program in one of the VM for Azure Batch.
After some troubleshooting, I made a simple test running this call as shown in the below code.
#include &lt;stdio.h&gt;
#include &lt;cuda.h&gt;
#include &lt;cuda_runtime_api.h&gt;
#include &lt;time.h&gt;

clock_t start, end;
double cpu_time_used;

int main()
{
    CUresult result;
    printf(&quot;CUDA version %d \n&quot;, CUDA_VERSION);    
    start = clock();
    result = cuInit(0);
    if (result != CUDA_SUCCESS) {
        printf(&quot;cuInit failed with error code %d: %s\n&quot;, result, cudaGetErrorString(result));
        return 1;
    }
    end = clock();
    cpu_time_used = ((double) (end - start)) / CLOCKS_PER_SEC;    
    printf(&quot;cuInit took %f seconds to execute \n&quot;, cpu_time_used); 
    return 0;
}

It takes about 1.9 seconds in average.
Some specs:

NVidia driver: 460.32.03
CUDAToolkit: 10.2
Azure Batch: nc6, Tesla K80

As a comparisons, the same code was running on my desktop (Windows) as well another custom Azure VM (nc6, not Azure Batch) giving similar result of 0.03 seconds. (cudatoolkit 10.2)
-- update 1 --
Calling CUDA initialization via Torch also shows a significant lag (for the first call) as shown from this test:
run: 0, import torch: 0.430, cuda_available: 4.921
run: 1, import torch: 0.000, cuda_available: 0.000
run: 2, import torch: 0.000, cuda_available: 0.000
run: 3, import torch: 0.000, cuda_available: 0.000
max time for import torch: 0.43 s,  max time for cuda_available: 4.92 s
torch.version 1.7.1+cu101 torch.version.cuda: 10.1

The import torch code is:
import torch

and the cuda_available code is:
torch.cuda.is_available()


My question is the time taken by the Azure Batch for CUDA initialization normal behavior ?

Azure Batch in VirtualMachineConfiguration mode allocates Virtual Machine Scale Sets internally. There is no difference in the underlying hardware Azure Batch allocates from. For further investigation:

How big is your sample set? Is your start time reproducible between different VMs within different Batch pools? Perhaps you got a bad or degraded VM or GPU.
What does nvidia-smi tell you on the compute node in question? Are there any potential hardware faults?
Are you using an Azure Batch task or are you remoting into the VM and executing your task interactively?

