What is $I$ in the noise described in the paper "Parameter Space Noise for Exploration"?
Yes, since $\tilde{\theta}$ is a vector, to define its distribution one needs a covariance matrix. Here $I$ is the identity matrix, which means that the noise has a zero-mean normal distribution with standard deviation $\sigma$, and different components of this noise are uncorrelated.