Can I construct a feasible stream cipher out of HMAC and a secure hash algorithm? I have constructed a stream cipher from a secure hash algorithm and a HMAC. Here is a brief description of the algorithm:

Let: (Actually Objective-C styled pseudo code)


[data SHA512Hash]: SHA-512 hash of data,
[data SHA512HMAC:key]: SHA-512 HMAC of data with key,
[data xor:another]: bitwise XOR of two data and another which have the same length,
[data bytesTo:index]: first index bytes of data,
[data removeFirstBytes:count]: remove first count bytes from data,
[data length]: length of data.
[data concat:other]: Concatenate other after data.
[Data new]: allocate an empty buffer.


Pseudocode: (Or actual Objective-C code?)

Data hkey = [key SHA512]; // Hash to make lengths match.
Data segKey = [IV SHA512];

Data dest = [Data new]; // This will be the output
Data last = (Some constant)

do {
    // Cut off hash-sized chunks of source data
    Data segment = [data bytesTo:[hkey length]];
    [data removeFirstBytes:[hkey length]];

    // Derive a new segment key
    segKey = [segKey SHA512HMAC:hkey]; // or [segKey SHA512HMAC:[hkey concat:last]]? If so, how to decrypt?

    // XOR
    last = [segment xor:[segKey bytesTo:[segment length]]] // Truncates to make length match
    dest = [dest concat:last];
} while ([data length] &gt; 0)


From some other questions I am suggested that this is SHA-512 HMAC running in OFB mode but what about the change in the code comment?

This code seem to decipher itself when given the same key.
Here is your algorithm in formula form, which might be easier to understand for non-objective-C-experts:

Encryption:


$P_1 \mathbin\Vert P_2 \mathbin\Vert \dots \mathbin\Vert P_n := P$ (split plaintext in blocks, same size as hash output, except the last one)
$K := \operatorname{SHA256}(\mathit{key})$
$O_0 := \mathit{IV} (= K)$
$O_i := \operatorname{HMAC}_{\operatorname{SHA256}}(K, O_{i-1})$ for all $i \in \{ 1, \dots, n\}$ (key stream)
$C_i := P_i \oplus O_i$  for all $i \in \{ 1, \dots, n\}$ (the usual stream cipher XOR)
$C := C_1 \mathbin\Vert \dots \mathbin\Vert C_n$


(where the last block $O_n$ is truncated to match the length of $P_n$)

Decryption: same, but with $C_i$ and $P_i$ swapped.


$P_i := C_i \oplus O_i$  for all $i \in \{ 1, \dots, n\}$


As you remarked right, this is output feedback mode (OFB) with HMAC-SHA-512, with the key reused as the IV. (The value you named IV – $K$ here – is actually the effective key, derived by SHA-256 from the original key, the IV is your first segkey, initialized to the same value).

If you use the commented variant with


$O_i := \operatorname{HMAC}_{\operatorname{SHA256}}(K \mathbin\Vert C_{i-1}, O_{i-1})$,


you have some new mode which is like a mixture of cipher feedback mode and output feedback mode, which uses the cipher feedback as part of the key (?). It is not a synchronous stream cipher anymore, as the keystream depends on the previous ciphertext. You can still decrypt it, you just have to use the previous version of segment instead of the previous version of last as input in the decryption function (i.e. you need one more variable in your program).

If you want to stay with established modes of operation, here is cipher feedback mode (CFB):


$O_i := \operatorname{HMAC}_{\operatorname{SHA256}}(K, C_{i-1})$


This is Counter mode (CTR):


$O_i := \operatorname{HMAC}_{\operatorname{SHA256}}(K, i)$


The analogons to ECB, CBC, PCBC don't work with a MAC (or other PRF like a keyed hash), as there is no inverse which would be needed for decryption.

Also, as Ivella remarked, your stream cipher is a lot slower than dedicated stream ciphers, or the same modes applied to a block cipher.