After implementing a novel encryption algorithm, how would one go about analyzing its security or get help from others in doing so?
Typically, there are a variety of steps in evaluating a new algorithm.  They start with the quick review


is it already known?
does it vary only in nonrelevant ways from what is known?


which is commonly enough to show vulnerabilities in many amateur attempts at encryption.  The point is that there are a number of well known ways to translate strings of symbols into other strings of symbols, and many of those have been evaluated in common cryptographic studies.  Good books on cryptography cover these.

Then, the more intensive analyses begin.  These include:


functional inversion
analysis of the symbol statistics
differential cryptanalysis


and a variety of other mathematical techniques for extracting information of the keys from the intercept stream.

Also, if the technique involves nontraditional information stores (if it doesn't have keys, if it has strange communication channels and protocols, etc.), then you have to also analyse the various impersonation and interception scenarios on each channel.

For the description given, it would appear that the first cryptographic algorithm would fail the first quick view.  Substitution ciphers are well known.  It doesn't really matter how many times you compose substitution ciphers, their properties are the same.  If you had 10 keys and did substitution 10 times before final send, it would be equivalent to a single key substitution cipher and have no better security.  Unless there is some other tweak to the algorithm, that would quickly be dismissed as already known.