Questions about a SHA3 cipher stream encryption scheme
TL;DR: The idea is good and the implementation should work securely (with a solid security margin), given that some tweaks are implemented.

How secure is it?


  A) If given a large enough password, does this system appear secure?


If you replace "password" with "key", then the answer should be yes.

For analysis purposes, let's define $E_K(M) = \operatorname{MAC}_K(\text{CTR} || \text{Nonce})\oplus M$, with $\operatorname{MAC}_K(M) = \operatorname{Keccak}(K||M)$. The nonce is message based and the counter is call-based, i.e. the counter increases by one every 512 bit. The MAC was already analyzed here.

Let's further define $E^A_K(M) = \text{Nonce}||E_{K1}(M)||MAC_K(E_{K2}(M))$ which is the full authenticated encryption mode. Note that I have used different keys here for authentication and for encryption. This removes potentially dangerous dependencies between the two components. Not furthar that the order of key and ciphertext was swapped for the authentication to obtain more consistency and don't do potentially dangerous things.

The encryption component of the above is basically CTR-Mode, with a split-up counter where one part is "static" and one part is "dynamic", like is done in GCM. As you properly randomize it using the nonce and the counter, this is a proper instance and therefore the standard security bound of $2^{n/2}$ (for non-permutations) holds, i.e. the encryption strength is 256-bit. This is is due to the fact that you expect collisions from the hash after roughly $2^{n/2}$ invocations with different values and thereby can derive information about the plaintext from this. As for the authentication, at least collision attacks are required to break the standard encrypt-then-mac, also giving (at least) 256-bit security.

If you truncate the output of the hash function, security will decrease accordingly. SHA-3-256 would only yield 128-bit strength, as you'd expect collisons after only $2^{128}$ calls instead of $2^{256}$.

Potential Improvements


Make the keys independent.
Make the order of arguments to the hash consistent (i.e. always the key first)
Take associcated data (AD) into account, especially the nonce should always be authenticated.


Real Life applications

If this wasn't for demonstrations purposes you should strongly avoid rolling your own crypto and rather use AES-GCM (which should be faster anyways), any standard AES authenticated encryption scheme, Keyak or the scheme linked by otus in the comments: "Duplexing the sponge: single-pass authenticated encryption and other applications" by Bertoni, Daemen et. al.