Extract the data from Dicom files in an time efficient way I am trying to extract and save images from a Dicom dataset. It is taking approximate 4-5 hours to process 1000 files. (Also, I am reading and saving files in an external hard drive so there is a time lag due to this factor too)
Is there a way to expedite the program as I have more than 10k files and it'll take me forever to just extract the images.
Also, if there is any other suggestion or improvements, anyone could suggest then it is hugely appreciated as I am not from Computer Science background.
Below is my code -
import pandas as pd
import shutil
import os
import glob
import numpy as np
import pathlib
import torch
import random
import shutil
import pydicom
import matplotlib.pyplot as plt
from skimage.io import imread, imsave,imshow

data_dir = 'Path_to_dicom_files'
files = glob.glob(os.path.join(data_dir,'*.dcm'))
folders = []
counter = 1
Destination_path = 'Path_to_destination_folder_where_images_will_be_saved'
root_dirs = 'Path_from_where_files_are_being_read'
for i in files:
    print(i)
    dcm = pydicom.dcmread(i)
    name = dcm.AccessionNumber
    dest = os.path.join(Destination_path,name)
    if dcm.Modality == &quot;XC&quot;:
        if os.path.isdir(dest):
            img = dcm.pixel_array 
            name = dcm.AccessionNumber+'_'+str(counter)+'.png'
            counter+=1
            #img.save(name)
            imsave(name,img)
            shutil.move(os.path.join(root_dirs,name),dest)
        else:
            os.mkdir(dest)
            img = dcm.pixel_array
            name = dcm.AccessionNumber+'_0.png'
            #img.save(name)
            imsave(name,img)
            shutil.move(os.path.join(root_dirs,name),dest)
Big data requires big time. But there might be ways to speed up your processing somewhat. For each DICOM file you:

open the file
read metadata
read an image
write the image as PNG in the current directory
move the PNG file to it's final destination

Some complex file formats benefit from being read and written on a local drive, the software assumes drive access is fast and memory is expensive, so it reads small bits of data scattered around the file instead of reading larger chunks and using only the small bits it needs. These file types can benefit from being copied over in their entirety to a local drive for reading. However, DICOM is not a complex file format, and DICOM files are commonly stored on networked drives, so I don't think this causes any specific problems.
However, writing PNG files might be a bit faster if done on a local drive. You create the file in the current directory (wherever that is), then move it from root_dirs to Destination_path. So I presume that root_dirs is the current directory? Instead, you could create the file in the temporary directory, then move it to its final destination like you do:
import tempfile
temp_dir = tempfile.gettempdir()  # expected to be on the local drive
# ...
name = dcm.AccessionNumber + '_' + str(counter) + '.png'
name = os.path.join(temp_dir, name)
imsave(name, img)
shutil.move(name,dest)

Next, skimage.io.imsave is very flexible, but it just calls functions in other libraries:

By default, the different plugins are tried (starting with imageio) until a suitable candidate is found. If not given and fname is a tiff file, the tifffile plugin will be used.

Note also that imageio further knows a lot of formats, including two different PNG implementations. So this writing function again does a lot of logic to find out what format to write the file in. And it might end up using the PIL implementation of PNG, which I hear is not very fast.
Therefore, it might be a bit more performant to use a different file writer. I would experiment with different ones, even a small difference in timing will accumulate over so many images. For example, with OpenCV you'd do:
import cv2
# ...
cv2.imwrite(name, img)

Finally, since dcm.pixel_array is a property, not a function, it is likely that dcm = pydicom.dcmread(i) reads in all the data. If you have files for which dcm.Modality == &quot;XC&quot; is not true, you've read in data unnecessarily. Consider looking for a DICOM reader that can read only the metadata. I don't know anything about pydicom, so can't comment on its speed.

Since this is Code Review, I'll give you some pointers towards better code:
Python people have strict rules they follow for code style (encoded in PEP 8). You break some of these rules, but I won't go into that, I think code style is personal and as long as you're consistent, you're OK.
Repeating code is bad though. It is easy to fix a bug in one place, and then forget it needs fixing in the other copy too. It causes there to be more code to read, making the code harder to understand. And it makes you type more, giving you more possibilities to make errors. The following bit of code is repeated, with only a trivial change:
counter = 1
# ...
        if os.path.isdir(dest):
            img = dcm.pixel_array 
            name = dcm.AccessionNumber+'_'+str(counter)+'.png'
            counter+=1
            imsave(name,img)
            shutil.move(os.path.join(root_dirs,name),dest)
        else:
            os.mkdir(dest)
            img = dcm.pixel_array
            name = dcm.AccessionNumber+'_0.png'
            imsave(name,img)
            shutil.move(os.path.join(root_dirs,name),dest)

Instead, write:
counter = 0
# ...
        if !os.path.isdir(dest):
            os.mkdir(dest)
        img = dcm.pixel_array 
        name = dcm.AccessionNumber+'_'+str(counter)+'.png'
        counter+=1
        imsave(name,img)
        shutil.move(os.path.join(root_dirs,name),dest)

You could also put the os.mkdir(dest) call outside the loop. Having less code inside the loop makes the loop faster.