Back up my Photos library from Recovery Mode in macOS Sierra
.photoslibrary is actually a package (i.e. a folder) which is why Dropbox has difficulty uploading it in one go. Instead, try zipping it first in Terminal using the following:

zip -r /path/to/zippedphotos.zip /path/to/yourlibrary.photoslibrary