Better to give a poor/unfinished talk at a conference or cancel it?
When this happens to me, I just describe it as "work in progress" or a "research attempt" and present what I can with what I've got.  What's wrong with that?  That's how research works.

"...hasn't panned out like I hoped it would when I accepted the talk."  You can say this during your talk, and explain why it didn't pan out, and ask the audience if they can help.  Maybe someone in the audience (your next co-author perhaps?) knows how to proceed, and has the expertise you're lacking.  If nobody can help, maybe that's an indication that there is an obstacle (which is important to know).

You've said you're a PhD student, without expertise in the side-topic.  Feel free to mention it in your talk "I'm a PhD student in [some subject] at [some university] under [someone]."

You said a post-doc invited you to give a talk (which I expect is not in the sense of an "invited speaker"), who presumably knows you're an early PhD student, and presumably knows what early PhD student talks are sometimes like.  My suspicion is that this question is more about inexperience and lack of confidence than an inability to give a reasonable talk due to the work being unfinished.