我昨天去的图书馆: "The library I went to yesterday" or "I went to the library yesterday"?
我昨天去图书馆的 sounds unnatural. I'd say 我昨天去了图书馆 or 我昨天去图书馆了, which means I went to the library yesterday.

我昨天去的图书馆 means the library I went to yesterday. You might have gone to several libraries or just one, because the sentence doesn't imply how many libraries you went. As you said 的 marking that relative clause.

If you say 我昨天去的那个图书馆, that means you are emphasizing that particular library you went.