broadcasting across tensors in `pytorch`
I am using pytorch as an array processing language (not for the traditional deep learning purposes), and I am wondering what the canonical way is to do "batching" parallelism. 

For example, suppose I want to compute svds of two dimensional layers of a 3-d tensor (using torch.svd(), say), and I want to return a tuple of stacked us, stacked s, stacked v. 

Presumably, through the magic of SIMD parallelism, this should be doable in roughly the same time as a single layer svd (on gpu), but how to program it?

PyTorch is a high level software library with lots of python wrappers for highly optimized compiled code. A function or operator either supports batch data or not. 
There is no other way around it than writing your own C/C++/CUDA code and invoke it with python.

Luckily, most functions support batch processing (including torch.svd() as pointed out by jodag) and it can be assumed that the developers (or the compiler) paid attention to data parallelism in the implementation. I recommend you to stack your tensors wherever you can. It usually leads to significant speedups.

Note that the batch dimension is always the first dimension of a tensor. PyTorch supports broadcasting for common operators like +, -, *, / as documented here. Because of possible ambiguities you are sometimes required to reshape your data accordingly to make clear what you want. For example if you want to add a batch of scalars to a batch of vectors you need to do something like:

a = torch.zeros(2, 2)
b = torch.arange(2)
a + b.view(2, 1) # or b.reshape(2, 1)
# tensor([[0., 0.],
          [1., 1.]])

