Scraping articles on a web site I'm trying to create a scraping API in Express. The API scrapes the different articles featured on the home page.

Here are the issues I'm trying to resolve:


My code's turning into a ton of jquery select statements. How can I resolve this / How can I simplify my code or extract functions from this giant mass?
Are my variable names clear enough?
Are there any Node conventions I'm not following? I've seen a lot of let and I'm using var. If I'm doing anything wrong let me know.


Here is the block of code I'm running:

var express = require('express');
var fs      = require('fs');
var request = require('request');
var cheerio = require('cheerio');
var app     = express();

app.get('/scrape', function(req, res){
  url = 'http://anawesomewebsite.com';

  request(url, function(error, response, html){
    if(!error){
      var $ = cheerio.load(html);

      var title, url, image;
      var articles = [];
      var article = { title : "", url : "", image_src : ""};

      //Get articles featured at top of page
      var i = 0;
      $(".articles li .img img").each(function(){
          var image_src = $(this).attr('src');
          article.image_src = image_src;
          articles[i] = article;
          i = i + 1;
      });

      i = 0;
      $(".articles li a[class='title']").each(function(){
         var title = $(this).text();
         var url = $(this).attr('href');
         article.title = title;
         article.url = "http://vegnews.com" + url;
         articles[i] = article;
         i = i + 1;
      });

      //Get articles featured at bottom of page
      var number_of_articles_beforehand = articles.length;
      i = number_of_articles_beforehand;

      $(".article .image img").each(function(){
        var image_src = $(this).attr('src');
        article.image_src = image_src;
        articles[i] = article;
        i = i + 1;
      });

      i = number_of_articles_beforehand;
      $(".article .image a").each(function(){
         var url = $(this).attr('href');
         article.url = url
         articles[i] = article;
         i = i + 1;
      })

      i = number_of_articles_beforehand;
      $(".article .title").each(function(){
        var title = $(this).text();
        article.title = title;
        articles[i] = article;
        i = i + 1;
      })

    }
    res.send(articles);
  })
})

app.listen('8081')
console.log('Magic happens on port 8081');
exports = module.exports = app;
As for the jQuery/function usage, there is definitely room for improvement but it would be helpful to see an example HTML element that contains the article img, title, and url to understand the page structure a bit.
I think your variable names are clear, but could be simplified. Also, it is common practice to use Camel Case with the first letter lowercase for Node/Javascript instead of underscores between words.


For example, replace number_of_articles_beforehand with numArticlesBefore

Here is an article I found helpful regarding var, let, const 


https://medium.com/javascript-scene/javascript-es6-var-let-or-const-ba58b8dcde75
Javascript allows variable hoisting (click here for more info) with the keyword var whereas variables/identifiers declared as let or const do not hoist.
Personally, I have stopped using var altogether to help improve code readability as mentioned in the article above. Also, it is common practice to use const for the Node package require statements instead of var.
Remove the var title, url, image; line because you are only using those variables within their respective functions.
Lastly, I would encourage you to replace i = i + 1; with i++;. It does the same thing and is easier to read. (FYI, i--; would decrement i by 1).