To what degree is Siri listening?
Your asked about the trust-worthiness of an algorithm. The answer to your question is a firm and absolute "No" - users can not rely on the algorithm to not record/send conversations or other private audio. Some algorithms are definitely better than others, but no algorithm is perfect - what is invented by the human mind can be defeated by the human mind. 

Let's muse over another word in your question: rely. Reliance is not a binary state of mind; it's much more subtle than that. Consider that we may rely on the mail carrier to deliver our mail each day, but as the consequence of failure is not catastrophic, we do not rely upon it heavily. OTOH, if you are a diabetic living in a remote location, your reliance on the mail carrier's delivery of your insulin has far greater implications. 

The "threat" is also a factor in our reliance. Security-wise, perhaps the thing we have going for us is that nobody really cares what we say or write or do. But if we happen to attract attention to ourselves - perhaps as a suspect, or the target of a criminal enterprise, or have become involved in "matters of national security" as this chap recently did, then our level of reliance on algorithms - and computer security in general - becomes a different matter. And so it is with your reliance on Apple's algorithm: does the mail carrier bring you junk, or does he/she deliver insulin?

Finally, let's consider the other variable in reliance: the quality of the algorithm. Reliance is not a binary state, and no algorithm is perfect. Nevertheless we must still consider how good it is in determining how heavily we should rely upon it. 

Unfortunately it is a practical impossibility for us to know how secure Apple's algorithm is: 


it's not open source - people outside Apple have to disassemble the code to inspect for flaws. They cannot read the source
even if it were open source, it wouldn't be perfect 
even if it were perfect, Apple still makes mistakes
Apple could even abuse their privilege or not catch an employee that attempts this abuse


Instead, and as a substitute for a quantitative analysis of the algorithm itself, perhaps the best we can manage is an attempt at objective assessment of some facts. Consider the following: 


A complete and massive failure of the algorithm would be bad for Apple's business. Consequently, it seems likely that Apple would put a serious effort into the algorithm, and safeguards for the data. 
Apple has weighed up Siri's business value against the risks of failures of their algorithm, and have made the decision to retain Siri. Impose may be a better word than retain in this case as Apple has not provided an option to remove or un-install the Siri software. 


These facts do not shed much light on the actual quality of Apple's algorithm, but they do speak to motives and motivation. We can safely conclude that Apple has reasonably strong financial incentives to design a reliable algorithm. However, it also seems safe to conclude that Apple considers our private data to be within their purview. Why else would they not provide an option for removal of Siri to accommodate their customers' preferences?

None of this is intended as "Apple-bashing". I am an Apple customer - I have spent more money than I'll ever admit on their products. But I feel it's also important to be realistic. Apple's interests do not coincide perfectly with ours. Should we rely on Apple's judgments and algorithms to protect and safeguard our security and privacy? I hope this answer has provided a little food for thought.



Epilogue:

I have just read @Bradley's answer to this question. I have to say that if the allegations leveled by Apple's contractor in Ireland over the "Siri Incident" are true, then Apple deserves far more than the "Apple-bashing" I tried to avoid in my answer. They deserve substantial fines - perhaps even criminal proceedings. It is simply unacceptable that anyone or any organization be allowed to use their wealth and influence to trample the rights of others. More to the point of answering this question, this episode (if true) speaks volumes on how we should consider relying on Apple's software.