Computing adjusted p-values in batches
Can you explain further about how you are computing in batches? The entire p-value array itself should not be large enough to cause the memory problem. So it seems like your main problem is that you are trying to hold the cross-joined DF in memory and do the computation you need. Therefore these are my thoughts:

You might be able to solve this problem the way you are currently doing it if you have access to a cluster which has more memory.
There is another way to solve it that will use less memory and could work locally but will take a long time using nested loop (see below). You can also write results intermediately instead of doing a union at the end of each loop. Afterwards you can take the initial DFS out of memory, and then load in all of the intermediary results and create the final df.
You could use the method from prior bullet point in cluster which will be a bit better.
It would be even better if you could use a cluster and parallelize it. This problem would be well suited for Spark.
If you don't have access to a cluster, you can still parallelize it on your own computer which may help a little with speed. Again, I think if you use Spark and parallelize it on your computer it would be a bit faster.

Here I will explain the first bullet which I think will at least allow you to get your final data frame without the memory issue:
I would first try making this problem into a nested loop of the columns of the data frames where you find the pvals for each gene --&gt; all miRNAs and then subset this after computing q-values and using a cutoff. Then you will only be storing in memory the pairs which have met your q-value threshold. I would not suggest trying to create a joined DF with all of this data as that will take up too much memory. I wrote pseudo-code for a loop below. I feel like this would help with you running out of memory, however it will still be pretty slow. If you are still running out of memory, do you have a cluster available to you? Also it would be even better to parallelize this. You could write it in python but do you have Spark? If you will be continuing to work with 'big data' I would try to get access to a cluster at your institution or pay for it using AWS or something.
I am going to call your DF with the genes: geneDF and your DF with microRNA mirnaDF. Keep in mind this is python pseudo-code, and is pretty messy. I can clean it up if you think it would be useful to you. I more-so just wrote it to make what I was writing about clearer.
top_adjusted_pairs = pd.DataFrame(columns=['gene', 'miRNA', 'qval'])
for i in range(len(geneDF)):
    # initialize array for the distances between all microRNAs with just gene X
    miRNA_geneX_pvals = []*len(microRNAsDF)
    geneCol = geneDF[i]
    for j in range(len(mirnaDF)):
         mirnaCol = mirnaDF[j]
         # compute distances and write pvalue into array
         pval = dist(geneCol, mirnaCol).pval
         miRNA_geneX_pvals[j] = [miRNA.name, gene.name, pval]
    # now that you have the array of distances between gene X and all miRNAs you can use multiple hypothesis correction. 
    qvals = adjust(miRNA_geneX_pvals['pval'])
    # pick q-val cut off
    top_mirna_geneX = miRNA_geneX_pvals[qvals&lt;0.1]

top_adjusted_pairs.union(top_mirna_geneX)
         
        
```