Why do repeated calls to torch.cuda.is_available() all return True?
I'm reading code that makes multiple calls to torch.cuda.is_available(). Each time it prepares to query a net, calls a function (shown below) that uses torch.cuda.is_available() to set the device (cuda or cpu) used. I don't understand why calls after the first one don't return False, thus pushing computation to the cpu.
Is the GPU released when the code leaves the method? Or, does each call take up only a relatively small part of the GPU, so that the code would need to make multiple calls to this method, before computation was pushed to the CPU?
Code in question:
def computeProposals(imageName):
    app.config['args'].img = imageName
    print('ARGs --img = ', app.config['args'].img)

    # Setup device
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

    # Setup Model
    Config = namedtuple('Config', ['iSz', 'oSz', 'gSz', 'batch'])
    config = Config(iSz=160, oSz=56, gSz=112, batch=1)  # default for training

    model = (models.__dict__[app.config['args'].arch](config))
    model = load_pretrain(model, app.config['args'].resume)
    model = model.eval().to(device)

    scales_range = np.arange(app.config['args'].si,
                             app.config['args'].sf + app.config['args'].ss,
                             app.config['args'].ss)
    scales = [2 ** i for i in scales_range]
    meanstd = {'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}
    infer = Infer(nps=app.config['args'].nps, scales=scales, meanstd=meanstd,
                  model=model, device=device)

    print('| start')
    tic = time.time()
    im = np.array(Image.open(app.config['args'].img).convert('RGB'),
              dtype=np.float32)
    h, w = im.shape[:2]
    img = np.expand_dims(np.transpose(im, (2, 0, 1)), axis=0).astype(np.float32)
    img = torch.from_numpy(img / 255.).to(device)
    infer.forward(img)
    masks, scores = infer.getTopProps(.2, h, w)
    toc = time.time() - tic
    print('| done in %05.3f s' % toc)

   return masks, scores

Here is the code that makes the repeated calls to computeProposals:
@app.route('/')
def index():
    global cc_data

    base_dir = app.config['base_dir']
    img_name = app.config['img_name']

    img_dir = os.path.join(base_dir, 'images')
    img_path = os.path.join(img_dir, img_name)
    print(img_path)
    print('Loading image and proposals, please wait')

    img = skio.imread(img_path)
    img = img[:, :, :3]

    masks, scores = computeProposals(img_path)

    session['pos_wts'] = np.zeros(masks.shape[2], dtype=np.float64).tolist()
    session['neg_wts'] = np.zeros(masks.shape[2], dtype=np.float64).tolist()
    masks = np.transpose(masks, (2, 0, 1))
    dilated = dilate_proposals(masks)
    print('Loading done')

    img_h = img.shape[0]
    img_w = img.shape[1]
    print('Image height {} and width {}'.format(img_h, img_w))

    rendered_img = draw_buttons(np.copy(img), img_w)
    if app.config['DoneFlag'] == 1:
        rendered_img = draw_end(rendered_img, img_h)

    img_stream = embed_image_html(rendered_img)

    # Create dicts with session variables
    cc_data = {'img_h': img_h, 'img_w': img_w, 'masks': masks,
               'scores': scores, 'dilated': dilated,
               'orig': np.copy(img).tolist(), 'render': rendered_img.tolist(),
               'clicks': []}

    session['response'] = {'input_img': img_stream,
                           'im_width': img_w, 'im_height': img_h,
                           'show_error': False}

    return render_template('index.html', response=session['response'])


Function torch.cuda.is_available() does not return whether the Cuda device is being used or if there is a memory left on the (those) device(s). This means the returned value will not depend on the number of processes running or the memory which has already been allocated to those processes. It only returns whether one or more Cuda devices are accessible by PyTorch.
