Does a concatenation of hashes of differently prefixed variations of any chosen message contain all possible finite bitstrings? Let $A$ denote a sequence of bits.  

Let $H$ denote a cryptographic hash function that has no limit on the length of its input (for example, SHA-3).  

Consider the following infinite sequence of bits: $$B = H(A) \mathbin\Vert H(0 \mathbin\Vert A) \mathbin\Vert H(1 \mathbin\Vert A) \mathbin\Vert H(00 \mathbin\Vert A) \mathbin\Vert H(01 \mathbin\Vert A) \mathbin\Vert H(10 \mathbin\Vert A) \mathbin\Vert \ldots$$ (that is, the prefix goes through all possible bitstrings, sorted by their lengths).  

Can we assume that for any chosen $A$, its corresponding $B$ is unique and contains all possible finite bitstrings (similarly to a binary representation of a fractional part of Pi)? If yes, can we use such technique as a cryptographically secure pseudo-random number generator that has no theoretical period?
Can we assume that for any chosen A, its corresponding B is unique and contains all possible finite bitstrings


No. Due to your requirement $H$ is compressing. This means that collisions must exist. Depending on your hash function construction, a collision might lead exactly the same output sequence. As a constructed example assume a hash function that processes input blocks from right to left and feeds them to SHA3. Any SHA3 collision would then work. Indeed, you cannot argue that your assumption above holds just with standard properties of a hash function. 

However, finding such messages is extremely hard. Already finding two messages that lead one identical block of B is hard by the collision resistance of the hash function. 


  can we use such technique as a cryptographically secure pseudo-random number generator that has no theoretical period? 


No. There are also several arguments against building a PRG this way. For example with common hash functions like SHA3 this construction would be extremely expensive as you cannot reuse previous results in computing B.