Job execution on records supplied by database I have a requirement to check all records in a table which is big (Around 1.5-2 million records). What I am doing is very simple, just get all the records and loop through one by one, and do things. I have a quartz job which will run every 2 minutes to check things.

ApplicationDetailCheckingJob.java

public class ApplicationDetailCheckingJob implements Job {

@Autowired
private DatabaseService databaseService;

@Value("${EnableJob}")
private boolean enableJob;

@Override
public void execute(JobExecutionContext jobExecutionContext) {
    if (enableJob) {
        try {
            LogService.info(this.getClass().getName(), "Started Checking...");
            //Get All Records
            List&lt;Applications&gt; allApplicationDetails = databaseService.getAllAppDetails();
                //Perform Check For Each Application
                allApplicationDetails.stream().forEachOrdered((app) -&gt; {
                    //Check For Columns Values
                    //If SendTweet = 1, Send tweet to the Application's User
                    //If SendEmail = 1, Send Email to User
                    //If Send SMS = 1, Send SMS to User
                    //Rest is also similar procedures
                });
            LogService.info(this.getClass().getName(), "Finished checking.");
        } catch (Exception e) {
            LogService.error(this.getClass().getName(), e);
        }
    } else {
        LogService.info(this.getClass().getName(), "ApplicationDetailChecking Job is Disabled");
    }
 }
}


DatabaseService.java

public List&lt;Applications&gt; getAllAppDetails() {

    String sqlQuery = "SELECT * FROM Applications";
    List&lt;Applications&gt; appDetails = defaultJdbcTemplate.query(sqlQuery, BeanPropertyRowMapper.newInstance(Applications.class));

    LogService.info(this.getClass().getName(), "Number of records found = " + appDetails.size());

    return appDetails;
}


Can I improve these methods? Is there a better way?

If I use it in the live DB, I doubt the job can be finished before it's started again after 2 mins.
For starters code-wise, I'm guessing you can try allApplicationDetails.parallelStream().

Try to use more method references if possible as there are potential performance advantages to doing so.

Your try-catch is unusually wide, if an Exception is thrown from the first stream element, no further processing is done. Are you sure that is OK?

How is jobExecutionContext being used?

Other more worthy considerations...


  What I am doing is very simple, just get all the records and loop through one by one, and do things.


Can you batch some of these operations together? For example, do your 'downstream' libraries accept multiple inputs at once, or do they only send individually?

Do you really need to retrieve all 22 columns of 1.5 to 2 million records? Are there perhaps temporal columns on the table so that you can only fetch the most recently updated rows? Or a subset of the columns to minimize network traffic?

The bigger question is... can your actions be farmed off to an ExecutionService provider so that they can be performed asynchronously? 

Simple in-memory processing of 1.5 to 2 million records in 2 minutes is relatively trivial with recent processing power. Having to incur network latencies (from emailing, twitter-ing tweeting, SMS-ing) is obviously going to introduce the 'timeout' issues that you are witnessing.

Most of these questions aren't really going to be covered by a code review (seeing how I've only dedicated four sentences to that), so perhaps you may want to drop by Programmers.SE for some of the more hard-hitting questions I've asked.