Toilet clogged with paper cat litter
There are no clear cut rules on how an auger should work. They're designed to "screw" into the clog and either break it up so it flows down the drain or catch the clog so you can pull it out with the auger. Keep trying to remove bits of the litter like you said you have done. Chemical stuff probably won't work since the litter is compressed paper and has expanded. They have 25' augers at home stores that would be easier to work with than the 50'. 

Without building information it would be hard to determine if the pipes will get larger as they go down. My experience is that the toilet drains are the largest installed, in residential.

Keep trying with the augers and flushing to clear this yourself. If that doesn't work you might try a 50' auger going down from the vent on your roof close to where the toilet is located. You might be able to rent one from your home store.  If all else fails, call a plumber. Good luck.