Young children have very short attention spans, which is not surprising given that the optimum study time for an adult is 40 to 45 minutes. So it's quite normal for your students' attention to stray. Also teaching young children is very different to working with adults, you just need to make a few adjustments to how you go about the classes.
You need to think about lots of short activities when you are planning your lessons. At the beginning, this will be very time-consuming as you will need to think through each activity when planning. As time goes by, you will build up a bank of short activities (eg a song, a chant, a nursery rhyme, games such as I spy) that the children are familiar with and that you can introduce when you feel a change is needed. As children get used to your style of teaching, the classroom rules and the types of activity, it will be easier to introduce these short activities into the lesson.
You should also think about pace and variety when you are planning. Some activities should involve whole class focus on the teacher, some should include individual work requiring reflection, some should involve group work and some should include physical movement. The changing focus will also stimulate children, re-focus their attention or calm them down.
You should also insist on having full attention when you are working with or addressing the whole class, for example when telling a story. Children need to understand that when you are talking they should stop what they are doing and listen to you. You could use a signal to indicate when this is necessay, for example clapping your hands.
I'm sure that with time you will get used to teaching children and it will seem less tiring the more you do it.