Structure of LSTM gates
Is each gate simply a feed forward neural network who’s output is either squished through sigmoid or tanh depending on which gate it is?


Close. Each gate is an activation function (typically sigmoid for actual gates, but tanh is used for other functions within the cell) over a weighted sum of all inputs to the cell's layer. Although spome reading of the diagrams may imply LSTM cells are processing a few single inputs in isolation, recombined later, in fact the inputs to each cell are the whole previous layer and the whole previous timestep cell states for the whole layer. This is similar to a view of a single neuron in a fully-connected feed-forward network.

Each gate in each cell has its own set of weights - one weight for each input from previous layer plus one weight for each cell state in the same layer, plus a single bias. When you combine these cells into a layer of multiple cells (e.g. when you choose to have an LSTM layer with 64 cells), this results in a separate matrix plus bias for each gate. In descriptions of LSTMs, these different matrices are often named for the type of gate they parametrise, so e.g. there will be a matrix plus bias for the layer's "forget" gate, which might be noted as $\mathbf{W}_f$. 

If you have $N_i$ inputs, and $N_c$ LSTM cells in a single LSTM layer, then $\mathbf{W}_f$ will be a $N_c \times (N_i + N_c)$ matrix of weights, and so will all the other parameter matrices describing other gates and value calculations for the combined cells.

In practice the calculations don't need to be handled per cell, but calculations over a layer of cells can be vectorised, and it is more like having a few parallel fully connected layers that combine in various ways to generate output plus next cell states.

There would be nothing stopping you extending this architecture and making any single gate or value calculation deeper by giving its own hidden layers. I suspect this has been tried by researchers, but cannot find any references. But without this customisation, the gates in standard LSTM are more like logistic regression over concatenated input and cell state.