Vectorizing a complex nested for loop in R (running models on different subsets of a data set, subsetting the data differently for each loop)
I'm not familiar with BGLR but I think I can give some pointers.

First to your direct question - yes you can vectorise.  A simple approach is to use one of the apply functions.  lapply usually is a good fit for such loops.  However this is not a magic bullet and I would strongly suggest to review the code.

I think the main thing that slows you down - besides the time BGLR needs - is how R deals with memory.  The relevant properties are:


most objects need to be placed continuous in memory
memory isn't immediately freed - but R waits till it is necessary and does garbage collect then
the time when R creates a copy of memory is a bit tricky.  It is not when one writes an assignment - at this point the variable just holds a reference to the original data.  But as soon as a change on the data is performed it creates a full copy.


All this effects don't appear in a "clean" R session, but as memory clogs things get significantly slower - agreeing with your observation. 

I see following issues in the code:

Growing lists
fold.Res and rep.Res are growing lists.  As grow them they need to be written anew.  This means first all the previous entries need to be copied (which can take some time) - but more important they can not be written in the same place as the new list takes more space.  This will create fragmented memory - meaning whenever R can't fit the new list where a whole from a previous was created it will first need to garbage collect and then maybe defragment memory.  I think this is the serious slow down you are experiencing.
Allocating lists unfortunately doesn't help, but with the apply functions like lapply this is also taken care of.

creating modified copies
You create phen2 and phen3 as copies - but you only modify and use column 2.  Meaning there is some unnecessary copying going on.  Also this will lead faster to some extra garbage collects as memory fills up.

possible side effects of BGLR
While the first runs are relatively free where to put their data, later may run into troubles finding suitable spots - causing again some superfluous memory cleanups.  This is not so easy to deal with.  I worked with stan which is quite the hog - and the best solution was to run it in separate R processes and save the results and the kill it - it was way faster than having these cleanups going on.  The main scripts would then just collect the files.  I described how I do it in my answer to: Using doParallel to cycle through *.rds files (specifying how to do it outside doParallel).  

to style 
It is a bit confusing that you sometimes refer to columns by name and sometimes by column number.  This makes things harder to read than necessary.  I would suggest always using the name approach.

Those apply functions are awkward to write.  Personally I would consider to rewrite it to use dplyr - in particular piping and do.  However that may be to much work.