Title: How can I get html source in Android?
Tags: android;html;eclipse;parsing
Question: I need to extract web page source to string in Android. I tried to do it using HttpClient, HttpGet and HttpResponse, but this method doesn't work. I have to do try/catch to every initialization of this methods and anyway, the application force closes.
public static String getHtmlSource(String webPage){
HttpClient client = new DefaultHttpClient();
URI url = null;
try {
url = new URI("http://www.kinopoisk.ru/film/581493");
} catch (URISyntaxException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
HttpGet request = new HttpGet(url);
HttpResponse response = null;
try {
response = client.execute(request);
} catch (ClientProtocolException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String html = "";
InputStream in = null;
try {
in = response.getEntity().getContent();
} catch (IllegalStateException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
try {
while((line = reader.readLine()) != null)
{
str.append(line);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
try {
in.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
html = str.toString();
return html;
}
LogCat
09-18 19:28:07.989: W/dalvikvm(5708): threadid=1: thread exiting with uncaught exception (group=0x41f64300)
09-18 19:28:08.029: E/AndroidRuntime(5708): FATAL EXCEPTION: main
09-18 19:28:08.029: E/AndroidRuntime(5708): java.lang.RuntimeException: Unable to start activity ComponentInfo{com.gavrilov.egor.movies/com.gavrilov.egor.movies.MoviesList}: android.os.NetworkOnMainThreadException
09-18 19:28:08.029: E/AndroidRuntime(5708):     at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2059)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2084)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at android.app.ActivityThread.access$600(ActivityThread.java:130)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1195)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at android.os.Handler.dispatchMessage(Handler.java:99)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at android.os.Looper.loop(Looper.java:137)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at android.app.ActivityThread.main(ActivityThread.java:4745)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at java.lang.reflect.Method.invokeNative(Native Method)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at java.lang.reflect.Method.invoke(Method.java:511)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:786)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:553)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at dalvik.system.NativeStart.main(Native Method)
09-18 19:28:08.029: E/AndroidRuntime(5708): Caused by: android.os.NetworkOnMainThreadException
09-18 19:28:08.029: E/AndroidRuntime(5708):     at android.os.StrictMode$AndroidBlockGuardPolicy.onNetwork(StrictMode.java:1117)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at java.net.InetAddress.lookupHostByName(InetAddress.java:385)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at java.net.InetAddress.getAllByNameImpl(InetAddress.java:236)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at java.net.InetAddress.getAllByName(InetAddress.java:214)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:137)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:164)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:119)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:360)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at com.gavrilov.egor.movies.ParseHtmlPage.getHtmlSource(ParseHtmlPage.java:32)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at com.gavrilov.egor.movies.MoviesList.onCreate(MoviesList.java:30)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at android.app.Activity.performCreate(Activity.java:5008)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1079)
09-18 19:28:08.029: E/AndroidRuntime(5708):     at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2023)
Comment: I'll be thankful to know what am I doing wrong.

Comment: It will be a lot easier to try and help you if you post the actual *code* that is giving you errors.

Comment: It works and you are probably doing it wrong.

Comment: You expect someone telling you what you are doing wrong without showing us what you are doing (in terms of code)?

Here is the accepted answer: Can it be you don't have INTERNET permission in the manifest?
Edit: Ah, you are doing that on the main thread, no?
Comment for this answer: Remove those try/catch and show your LogCat output. It will probably point out the exception line pretty clear. And maybe you're doing that on the main thread?
Comment for this answer: Of course it doesn't but what does LogCat print? And - you shouldn't do it on the main thread (in case you do)
Comment for this answer: Without try/catches this doesn't work too
Comment for this answer: Yes, I do it in the main thread. I'll try to solve this problem.

Here is another answer: See this issue to solve exception: How to fix android.os.NetworkOnMainThreadException?.
For HTML parsing you can use: http://jsoup.org/

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 2, 'CommentCount': 4, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2012:00:09/18/12', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<android><html><eclipse><parsing>', 'Title': 'How can I get html source in Android?', 'ViewCount': 368, 'id': 12480423, 'user_AboutMe': '', 'user_AccountId': 1854960.0, 'user_DisplayName': 'Egor Gavrilov', 'user_Reputation': 25.0, 'user_Views': 7.0, 'user_id': 1680754.0}}