Key secrecy vs Algorithm secrecy
The distinction you want to make between the key and the algorithm proper is not based on whether most of the operation is contained in one or the other, but on where the complexity lies. I am not talking about algorithmic complexity here, but complexity in its everyday meaning: difficulty to understand and reason about.

The algorithm proper is complex and hard to reason about. It generally does a whole bunch of arbitrary-seeming bit manipulations, logical and arithmetic operations, and general shuffling of the data. It's very hard for a layperson or even for a cryptographer to know how much privacy all those manipulations actually buy you, and what kind of cryptanalysis it might be vulnerable to. So the best way to be confident about the algorithm's security is to put it out in the open and have it reviewed by experts as widely as possible. MAKE IT PUBLIC.

The key, on the other hand, is a simple concept: it's a bunch of bits which need to be random. There is no need to review the key to assure the correctness of the crypto. Any key is supposed to be as strong as any other key (and if this is not true then it can in principle be discovered by review of the algorithm, not the key). We know that the quality of randomness that is available to generate keys is less than perfect, so in practice some keys may be weak due to lack of randomness, but at least everyone can know without needing to be an expert cryptographer and without needing to make a difficult analysis of the key that good randomness WILL lead to a good key. So use the best randomness you've got available then you need not (MUST not!) share the key with everyone in order to have confidence in your crypto.