What is $I$ in the noise described in the paper "Parameter Space Noise for Exploration"? In the paper Parameter Space Noise for Exploration, the authors describe the noise that they add to the parameter vector as: 

$$
\tilde{\theta} = \theta + \mathcal{N}(0, \sigma^2I)
$$

is $I$ simply the identity matrix, or am I missing something?
Yes, since $\tilde{\theta}$ is a vector, to define its distribution one needs a covariance matrix. Here $I$ is the identity matrix, which means that the noise has a zero-mean normal distribution with standard deviation $\sigma$, and different components of this noise are uncorrelated.