What is the algorithm behind multisignature address and redeem script? I know you can create a multisignature address with the command:

bitcoind createmultisig 2 ["pubkey1", "pubkey2", "pubkey3"]


As described here: https://gist.github.com/gavinandresen/3966071

This command returns the address (starting with character '3') and the redeem script.

My question: how to create multi-signature address and redeem script without using bitcoind? What algorithm is used?

Could you point me to some sources?
I stole most of the answer from Andreas' book :-)

The bitcoin developer docs provide this link here, which goes over P2SH to multisig. The idea is to create a so called redeem script, which is a ripemd160(sha256(sigscript)).  

Example from the book:

Redeem Script:    2 PubKey1 PubKey2 PubKey3 PubKey4 PubKey5 5 CHECKMULTISIG
Locking Script:   HASH160 &lt;20-byte hash of redeem script&gt; EQUAL
Unlocking Script: Sig1 Sig2 &lt;redeem script&gt;


see Andreas' book "Masterig Bitcoin, 2nd edition", page 152ff or in the GitHub version here.
The real book at http://chimera.labs.oreilly.com/books/1234000001802/index.html 


  how to create multi-signature address and redeem script without using
  bitcoind? What algorithm is used?


you can use any hashing tools or libraries, Andreas shows with the "bx" library, openssl would do as well.