What steps can be taken to troubleshoot syncing in iCloud?
The dotted lines mean iCloud is trying to sync the file with its remote servers. This might be due to temporary issues with iCloud services (see this Apple System Status), or that your Mac iCloud configuration has been corrupted and needs to be reset. As you say it has been waiting for over a week, I'm guessing it is the latter.

In such cases, what has worked for me in the past is to save any files waiting for sync  to my desktop, and then reset iCloud with the instruction below. To reset iCloud on your Mac, open Terminal.app (/Applications/Utilities/Terminal.app) and execute the following instructions:

mv ~/Library/Mobile\ Documents/ ~/Desktop/iCloud\ Backup
rm -rf ~/Library/Application\ Support/Ubiquity/peer-* 
killall ubd


Copy and paste each line to Terminal.app then press the Return key. First line, will create a backup of all your iCloud documents in your Desktop folder. Second line, will delete the original documents to force a clean slate on the iCloud. Third line, will force a restart on the iCloud sync process (ubd) and recreate all your iCloud documents. Once that's done, I open the backed up files and move then again to iCloud and they sync fine.

As an aside, in defense of 3rd party developers, this is rarely a problem of the host app using iCloud, as the process responsible for the iCloud sync (ubd) belongs to OS X.

Nonetheless, it's important to note that Byword for iOS does not support RTF files. This means that even though your RTF files can be synced to iCloud, they won't appear on your iOS devices—only on other Macs. The description on the App Stores says:


  ⚠ Important note
  Byword for iOS only works with plain text formats. Supported file extensions are: txt, text, md, mmd, markdown, markdn, mdown, mkdn, markd and fountain.


And the first question on Byword support FAQ also mentions this.