Getting data from API every x seconds and store it in a database I'm currently getting data from an API every 15 seconds. This data contains a json with 100-2000 new items every time. Now i don't need all the items that are in there but still quite some of them and i want to store these items in a database. I will probably delete items that are added longer then 5-7 days.

Now i just want to check if the way i'm doing this now is "normal". Because i will run this all day and it's going to get a lot of items.

The script looks like this now:

function fetchItems() {
  axios
    .get("http://lorem.com")
    .then((response) =&gt; {

      for (let i = 0; i &lt; response.data.stashes.length; i++) {
        if (some statements) { //filter some of the stashes here with some statements
          stashes.push(filterItems(response.data.stashes[i])); //filter items in stashes and push to new array
        }
      }
      //filtering done
      //add all items to database
      for (let i = 0; i &lt; stashes.length; i++) {
        for (let j = 0; j &lt; stashes[i].items.length; j++) {
          let item = { data i want to store };
          let sql = 'INSERT INTO items SET ?';
          let query = db.query(sql, item, (error, result) =&gt; {
            if (error) console.log(error);
            console.log('item added');
          });
        }
      }
    })
    .catch((error) =&gt; {
      console.log(error);
    });
}


Is this the way this should be done?
Quickly overviewing the code - it would work.
If you are into a more readable approach, I may have just a few (minor) comments:

Declare fetchItems as async, and use await for axios.

Instead of a for loop, use response.data.stashes.filter().
Adding all items in separate queries seems odd, is there a way to push all of them in a single transaction? Build the query outside in a helper function.

So your flow may look something like this:

const fetchItems = () =&gt; {/*... axios ...*/};
const filterItems = (item) =&gt; {/*.. return true or false */};
const buildQuery = (items) =&gt; {/*.. return SQL string ..*/};

async function execute() {
  try {
    const items = (await fetchItems()).filter(filterItems);
    const query = buildQuery(items);
    await db.query(sql);
  } catch (err) {
  /* handle it */
  }
}