Detecting license plate using tensorflow
Such a task is getting easier to complete, but there are still difficulties with rotations, skewing, scaling---to name a few issues. Your network has the benefit of simplicity and lightness for the target hardware, but it may suffer under the above conditions.

So 237 images (181+56) may be small for a "generic" approach, depending on how representative and diverse the dataset is. Also, the dataset is unbalanced (the first class has twice as many examples), which causes bias in learning.

There are several ways to expand a base dataset:


Transform images and add them (with the same label, if supervised learning) to the dataset. Many libraries allow to rotate, skew, scale, or even blur images. Be careful as transforming needs be "reasonable", and changing image format over and over creates artifacts that can confuse the machine (e.g. too much JPEG savings on transformed images).
Generate synthetic data. Assuming license plates have a known format, it may be easy to generate images with good fidelity to real plates. This is not always possible, but license plates are formalized, so there should be only a handful patterns (typically plain, light, diplomatic, military vehicles).


Aside the dataset (potential) issue, the graph is fine. However, it may be worth trying different settings. It really depends on how you have ended up with the current graph. Removing pooling layers helps keeping more information, as well as a larger input image. 120x60 looks pretty small, and the comparison to fast-RCNN's RoI pooling layer looks odd, given that RoI comes after the feature maps. So a larger input image could give better results.