Is there any way to test how secure is a new cryptosystem? I have investigated Elliptic Curves and after that I have designed a cryptosystem using this technique. How can I test the safety of my scheme compared to another cryptosystems that use factoring such as RSA? There's a way to ensure that my cryptosystem can be used in commercial environment ?
The general answer given to all budding cryptographers is "don't."  Your system is not secure and will be trivially broken if anyone starts using it.

This advice is given out in general because it handles the 99% case.  99% of the time, people who think they have a good algorithm are simply wrong.

Now the cases that remain are the individuals with enough drive to actually learn cryptography.  There's still a very good chance that the first system these people put together has major holes, but there's a big difference between them and the 99%: they're willing to take the time to be objective and learn how to do it properly.

For those, I refer to a famous quote: "Any idiot can design a cryptographic system that they themselves cannot break."  Your system will not be considered secure until others have beaten on it.  Ideally, big names need to beat on it.  However, therein lies a conundrum.  It is far easier to develop a bad cryptographic system than to prove that it's bad.  The big names in the business don't have time to spend on your algorithm until they have a reason to believe it isn't trivially weak.

This is a chicken and the egg problem.  You can't get your algorithm accepted without the big names agreeing it's secure, and the big names wont look at it until your algorithm has seen some acceptance.  What are you to do?

The standard accepted answer in the industry is that, while it means little if someone says "hey, I think I have a good cryptographic algorithm," it means far more if someone with a name for themselves says it.  The community has an accepted way to build a name for yourself: breaking the algorithms of others.  If you want your algorithm to be taken seriously, you first prove that you're the kind of person who knows how to brutally attack their algorithm by showing how well you can break others.

Bruce Schneier, one of those "big names," put together an online self-paced course on this topic.  It's titled A Self-Study Course in Block-Cypher Cryptanalysis.  It consists of several dozen broken algorithms which he has carefully arranged in what he perceives to be an order of difficulty.  In the case of most of them, the task is simple:


Become familiar with a particular (broken) algorithm.
Read a paper about how that algorithm was broken.
Break the algorithm yourself, following the general path laid out by the paper


It's a sort of "following in the footsteps of giants" approach.  It introduces you to a wide array of attacks and encourages you to not only know they exist, but have experience working them.  This way, you are in a good position to try to apply those attacks to your own algorithms.

Once you go through the course, try to find publications of algorithms which have not been broken yet, but do not have the notoriety needed for the big names to go after them.  Break those, publish your results.  This will begin building up your name, and eventually you will be taken seriously when you put forth your own algorithms.

The path to a secure algorithm is long an arduous.  Consider the path undertaken by SHA-3:


Submissions for the NIST hash competition were accepted in 2008.  The total number of submissions is not known because some were rejected before they were even let into the first round.  There were 57 publicly known submissions, 6 of which were rejected outright with no further explanation.
The first round, with 51 submissions were officially announced on December 9, 2008.  There was a conference in February where the algorithms were presented and NIST's criteria for acceptance to round 2 was publicized.  The round ended on July 24, 2009 when NIST published the list of algorithms which made the cut:


10 algorithms were officially withdrawn from the competition after their authors conceded substantial weaknesses.  NIST considers all of these algorithms "broken in round 1"
16 algorithms were not officially conceded, but significant cryptographic weaknesses were found.  NIST considers all of these algorithms "broken in round 1"
11 algorithms were not accepted into round 2.  These were not conceded, and there were no glaring weaknesses.  However, in the name of efficiency of the competition, they were rejected.  Some were rejected because they appeared to be weaker than the others (yet not broken). Some were rejected due to performance issues.  NIST does not consider these algorithms "broken."

14 algorithms were accepted into round 2, down from 53.


Round 2 was a lot more demanding.  Now these algorithms had substantially more attention put on them.  The community was given a year and a half to attack these algorithms, publicizing the final round of advancements on December 10, 2010


9 algorithms were rejected, though NIST points out that none of them were considered "broken" during that period.  They just didn't make the cut.
5 algorithms were accepted into the final around.


The final round lasted almost 2 years, with the winner announced on October 2, 2012.  The entire cryptographic community was paying attention to this, with big names trying to break every one of these 5 algorithms.  In the end, Keccak was chosen as the victor, and is now known as SHA-3

So when you compare the security of your system to a "big name" cryptographic algorithm like RSA, be aware.  This is the kind of scrutiny that the algorithms go under.  If you enjoy that kind of thrill, I highly recommend going through Schneier's self-taught course.  And who knows.  Maybe your algorithm will be the one in a million that is truly secure!  You never know unless you try.

As an aside, you mentioned that you are using Elliptic Curves.  In that particular field, make sure you pay attention to the backdoor issues brought up by Dual EC DRBG.  Those algorithms in particular will be given extra scrutiny for backdoors because of how public the issues of Dual EC were.  You will want to take extra care to make sure the public is aware that your numbers were chosen in an acceptable quasi-random manner (taking digits of pi or e are popular)