Re: using sockets to open connection to a search engine

From:

=?ISO-8859-1?Q?Arne_Vajh=F8j?= <arne@vajhoej.dk>

Newsgroups:

comp.lang.java.programmer

Date:

Mon, 15 Jan 2007 17:58:14 -0500

Message-ID:

<45ac0703$0$49196$14726298@news.sunsite.dk>

Damo wrote:

Hi,
I'm trying to open a connection to altavista.com through java to
retrieve the search results for a query. This is the code I'm using, it
works for google and yahoo but not altavista or MSN.

s = new Socket("altavista.com",80);
p = new PrintStream(s.getOutputStream());
p.print("GET /web/results?q=java HTTP/1.0\r\n");
p.print("User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.1) Gecko/20061010 Firefox/2.\r\n");
p.print("Connection: close\r\n\r\n");
in = s.getInputStream();

If you type this : www.altavista.com/web/results?q=java into
the address bar, it will return the result page.

Put something in between your browser and AltaVista and
see what the browser sends.

You already have User-Agent, but maybe it wants Referrer or
Accept or Accept-Language or Accept-Encoding.

Or maybe it wants HTTP/1.1 (which requires Host).

There is a limited number of things to add until
you are fully browser compatible.

Arne