applescript CURL with authentication fails
Dedicated Tools

Given the problems encountered with curl and AppleScript, consider using an alternative dedicated tool such as Beautiful Soup. See How To Scrape Web Pages with Beautiful Soup and Python 3 for a comprehensive introduction.

Alternatively, there are numerous tools other that can help, see Web scraping software on Wikipedia. Many of these tools are free, open-source, and can be called from the command line.

I have previously used Web::Scraper for extracting property listings.