Re: HTTPUrlConnection does not download the whole page
The87Boy wrote:
public String getPage(String link) {
String pageEscaped = "";
try {
URL url = new URL(link);
// Open the Connection
HttpURLConnection conn = (HttpURLConnection)
url.openConnection();
// Set the information
conn.setRequestProperty("user_agent", "Mozilla/5.0
(Windows; U; Windows NT 6.0; da-DK; rv:1.9.1.4) Gecko/20091016 Firefox/
3.5.4 (.NET CLR 3.5.30729)");
conn.setRequestProperty("max_redirects", "0");
There is a set-Method to disable redirects, no need to set that
property directly.
conn.setRequestProperty("timeout", "300");
There are two methods allowing you to set the timeout for
connect and read, no need to set that property. Also it might
have no effect on the behavior of the connection-class, because
it most likely will not parse the data you set to the header.
conn.setRequestMethod("GET");
This is the default-method and only changes (also autoamtically)
if you set doInput to true.
conn.setDoOutput(true);
// Connect
conn.connect();
You don't need call that, it happens already when calling
getInputStream.
// Get the Status-Code and add it to the HashMap
int statusCode = conn.getResponseCode();
What is the value of statusCode?
String page = this.getPage(conn.getInputStream());
[...]
} catch (IOException e) {System.err.println(e.getCause
());System.err.println(e.getMessage());}
A simple e.printStackTrace() should give out all the informations
you print here and more that are most likely valuable to find the
reason for problems.
public String getPage(InputStream is) throws IOException {
BufferedReader br = new BufferedReader(new InputStreamReader
(is));
This uses the encoding of the system, not the encoding being
used by the server when sending the data, so you most likely
will corrupt your data.
String line = "";
StringBuilder sb = new StringBuilder();
while ((line = br.readLine()) != null) {
sb.append(line+'\n');
System.out.println(line);
Any lines being given out while reading in data?
Regards, Lothar
--
Lothar Kimmeringer E-Mail: spamfang@kimmeringer.de
PGP-encrypted mails preferred (Key-ID: 0x8BC3CD81)
Always remember: The answer is forty-two, there can only be wrong
questions!