Preparing Dataset Minority Class vs Majority Class
My question is: Does it matter if the minority class is set as the positive or negative label in relation to performance of training a model or affecting a loss function such as cross entropy?

No it doesn't.
However in binary classification it's customary to call &quot;positive&quot; the main class of interest, so be careful to be clear about which one is positive/negative when/if you present your results to other people.
Also be careful that precision and recall are usually calculated for whatever is called the positive class, so don't inadvertently use the results of the majority class instead of the one you're interested in.