Object Detection scratch implementation or high level API?
As you are looking to implement already known networks with a framework such as pytorch/tensorflow, I would definitely suggest reading into this post : https://towardsdatascience.com/object-detection-and-tracking-in-pytorch-b3cf1a696a98

Given that you wanted to implement this from scratch, its important you understand what the following:


Bounding boxes in images
Loss function - Intersection over union
Network architecture - You can use nettron to learn about the network architecture once you have written it. 
Visualize the learning of your model after every 'x' epochs, to gain an intuituve understanding of what your network is trying to learn given the training dataset.