Seoul , South Korea ( CNN ) -- Four members of a group of 31 North Koreans who accidentally crossed over into South Korean waters on a fishing boat , have decided to defect to the South , despite Pyongyang 's demands that they all be repatriated to the North , the South Korean Red Cross said . The defection of the four North Koreans could spark tensions on the peninsula , where joint U.S.-South Korea military drills have kicked off this week . North Korea had threatened to engulf Seoul in a `` sea of flames , '' a day before the opening of the exercise . The South will return the remaining 27 members of the group through Panmunjom , a truce village on the ground border between the two sides , and send the fishing boat back through the waters in the West Sea on Friday , the Red Cross said . South Korea 's intelligence agency questioned the North Koreans for almost a month since they crossed over in February before deciding to release the group , the aid agency said . There are currently more than 20,000 North Korean defectors in South Korea , according to the South 's Unification Ministry . The number of annual defectors has risen dramatically since the turn of the century and continues to climb . Defectors who are forcefully repatriated to the North after attempting to cross over into third countries such as China , Mongolia or Southeast Asia are commonly sent to prison camps as a punishment . Many defectors who live in South Korea fear for the lives of their family or relatives who remain in the North .
A boat with a crew of five sails into South Korean waters , the Red Cross says . Four of the crew decide to stay in the South . The number of deaths has risen dramatically in the past year .