Programmatically scrape the latest version of Tomcat for an installation script Looking for some assistance or ideas on how to grab the latest version of a Tomcat release from their website or other website.

I came across these Stack Overflow/Stack Exchange links that were relevant below:


https://stackoverflow.com/questions/22510705/get-the-latest-download-link-programmatically (Nginx, but same idea)
How to get the latest Tomcat version?


..But unfortunately both led to some dead-ends. It did give me some ideas though.

I have some installation scripts where the download URL is predictable based on the version number of the software, so if there's a consistent way of scraping for the latest version of the software I simply add a variable into the script for it to curl / grep for that version on the text and use that variable for download links, untaring, moving, etc.

Hoping to do something with Tomcat in the same vein. A installation script I won't need to keep updating (unless they change their site).

Any sort of thoughts, ideas, are appreciated.

I've also looked at http://tomcat.apache.org/whichversion.html which lists the "Latest Released Version" and this seems like maybe the best, more reliable location to grab the version, but I had trouble getting that with Curl / Grep because of the table structure. I am only just scratching the surface with learning those commands, so maybe someone more well versed could get that working, otherwise I'm definitely open to other thoughts!

curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"


It seems to work but I am not sure if I am entirely happy with it. An additional use-case that I thought of, is it would be nice to grab the latest version of each major version instead of just the absolute latest.

Do you have suggestions on how you would parse the highest number with the link you provided, and/or how to start in one of those directories and parse the highest there? That way for example I could create a latest version install script for Tomcat 7, 8, 9, etc.

Thank you!! (Like I said I am just getting started, so please my apologies with the lack of knowledge but I do appreciate digging in and understanding how the commands work or reached their conclusion)
I've created a couple crude methods of grabbing the latest major versions of releases, as well as absolute latest.

# Grab Latest Tomcat 7
curl -i https://www-us.apache.org/dist/tomcat/tomcat-7/ | grep -Po '(?&lt;=(&lt;a href="v)).*(?=/"&gt;v)'

# Grab Latest Tomcat 8
curl -i https://www-us.apache.org/dist/tomcat/tomcat-8/ | grep -Po '(?&lt;=(&lt;a href="v)).*(?=/"&gt;v)'

# Grab Latest Tomcat 9
curl -i https://www-us.apache.org/dist/tomcat/tomcat-9/ | grep -Po '(?&lt;=(&lt;a href="v)).*(?=/"&gt;v)'

# Grab Latest Tomcat
curl -i https://api.github.com/repos/apache/tomcat/tags | grep '"name"' | head -1 | egrep -o "([0-9]{1,}\.)+[0-9]{1,}"


It's a bit of a hack where I'm looking for text in-between two values on a web page to grab the version number. It's crude and I'm sure there's a more elegant way to grab this, but this what I was able to figure out with my skillset! Maybe it can help someone in the future, but if anyone has better ideas I'm all ears.