Title: Save page source from Amazon in Java
Tags: java;eclipse
Question: I am trying to save a page source from Amazon so I can see the price of an item. When I try to save it to a file, it only saves about 60 lines, most of them being white space. I can see the source from my browser and it is thousands of lines long. It goes for any page that I am trying to search. Heres the link I tried: http://www.amazon.com/gp/product/B015WCV70W/ref=s9_simh_gw_g147_i2_r?ie=UTF8&amp;fpl=fresh&amp;pf_rd_m=ATVPDKIKX0DER&amp;pf_rd_s=desktop-2&amp;pf_rd_r=0XHXJAF2NQ35BP5Y435K&amp;pf_rd_t=36701&amp;pf_rd_p=dc68ddd1-99ac-45e5-8c23-e9e0811a2b2c&amp;pf_rd_i=desktop
Is there an easier way to do this?
Here's my code:
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.util.Scanner;
public class DownloadPage {
public static final Scanner in = new Scanner(System.in);
public static void main(String[] args) throws IOException {
System.out.print("Enter URL: ");
savePage(in.nextLine());
static void savePage(String entURL) throws IOException{
URL url = new URL(entURL);
URLConnection con = url.openConnection();
InputStream is = con.getInputStream();
BufferedWriter bw = new BufferedWriter(new FileWriter("text.txt"));
BufferedReader br = new BufferedReader(new InputStreamReader(is));
String line = null;
int count = 0;
while (br.ready()) {
bw.write(br.readLine());
bw.newLine();
count++;
}
line = null;
bw.close();
System.out.println("wrote successfully " + count);
Sorry if I didn't format it right, its my first post.

Here is the accepted answer: This is because you use br.ready(), so every network pause cause the end of cycle
This block gives me 20632 lines of html
int count = 0;
while (true) {
String line = br.readLine();
if (line == null) {
break;
bw.newLine();
count++;

Here is another answer: The url is just a load point for a javascript app, which renders the HTML into your browser.
If you want to capture the rendered page, try Selenium/WebDriver which emulates a browser (and will run a javascript app).

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 2, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2016:00:05/21/16', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<java><eclipse>', 'Title': 'Save page source from Amazon in Java', 'ViewCount': 54, 'id': 37366215, 'user_AboutMe': '<p>I am a high school senior that is super interested in how things work. Whether it is electronics, coding, or business, I try my best to learn as much as possible about subjects that interest me. I started my own Green Screen business when I was in tenth grade and am now booked weekly!</p>\n', 'user_AccountId': 7763049.0, 'user_DisplayName': 'Edison Grauman', 'user_Reputation': 15.0, 'user_Views': 13.0, 'user_id': 5875162.0}}