Rolling your own crypto I know this is usually frowned upon, but supposing you were to roll your own encryption cipher (completely on your own or with a small group of friends, with no peer review possible), what could you do to make your custom algorithm as safe as possible (or rather, to keep the danger of using it to a minimum)? Similarly, what would you do for something like a hash function to make the probability of it having some trivial collision or preimage attack as low as possible?

I understand that there would be no way to approach the security of algorithms like AES, and I have no plans to use custom algorithms for anything important, but I thought it would be informative to think of what some best practices would be for rolling your own cryptosystem if you were to do so.
what could you do to make your custom algorithm as safe as possible


Try to re-use existing algorithms and research to the extent possible. 

For instance, suppose you are designing a block cipher (for some reason), and you need some source of non-linearity. Some researchers have already gone to the trouble of finding s-boxes with optimal cryptanalytic stats and minimal implementation costs. There are analogous results for linear layers as well. If you incorporate the results of these works into your design, you can rest assured that they will probably not be the weakest links in your design.

At a higher level, you could also simply re-purpose an existing primitive such as ChaCha and make a hash function out of it (like BLAKE). Or you could re-purpose an existing hash function into a (inefficient) stream cipher, like many others have done/tried to do already.

You (or someone with the same question) might say:


  "But wait, if I do those things then I'm not really coming up with something of my own!"


I think it is safe to say: Symmetric cryptography is pretty much a solved problem by this point.*

You can play around with it if you want, but don't expect to come up with anything (good) that hasn't been done already.

A note about "security"

You need to define your threat model before you can conclude what is/is not good enough in regards to security. 

If your only goal is to keep your messages confidential from your grandparents, then you can probably get away with a pretty low quality design (unless your grandpa happens to be Adi Shamir or someone similar of course).

If your goal is to keep your messages safe from nation-state level actors with billions of dollars for their budget and hundreds (or thousands) of cryptographers and mathematicians at their disposal, as well as good old fashioned thugs, then you should probably think twice about the expectations of your design. 


If this is your threat model, this is where you need to really stick to pre-existing and battle-hardened constructions. An "oops" in your design could cost you or one of your friends/users dearly.


Note

A secure design and a secure implementation are not the same thing. Coming up with a design that is not broken is only half the battle - you also need a way to implement the design in a way that can't be exploited, and this is a non-trivial subject. Just knowing how to program does not mean that you can create a secure implementation of an algorithm.

Finally

If you read all of that and are still interested in doing it, please remember one thing:


Don't advertise your construction for others to use (outside of your friends who you have of course already informed that it is your own home-rolled construction and could break at a moments notice with no warning... right?).


If you set up a website preaching the virtues of your algorithm, the only people who would use your construction are people who don't know any better. This would be taking advantage of peoples ignorance at the expense of their safety for no good reason. If you did something like that in real life as opposed to cyber space, you would probably go to jail.


Advertising your design will probably just put you on a list of snake oil salesman and destroy your own reputation.


*Take my word for what it's worth, I'm just someone that likes cryptography.