[jasonlambert]

A reminder that using automated software to produce a local copy of this website for offline browsing (or any other reason) is strictly forbidden.

Running any automated software against this site specifically, or my servers in general is viewed as an attempted denial service attack, and will be reported to your ISP.

Thanks for you're co-operation.

[eatinglemur]

That's nice, and what, pray tell, is the ISP obligated to do? What crime has been committed?

[jasonlambert]

Originally Posted by eatinglemur

That's nice, and what, pray tell, is the ISP obligated to do? What crime has been committed?

It is an automated DOS against my server.
One individual yesterday sent 20,000 HTTP requests for PHP pages in 1 hour, seriously degrading the performance of this site. Such floods are no different to using a SYN or ICMP flood or any other form of malicious denial of service attack that ties up the resources of my server preventing legitimate traffic getting to the site. Many ISP's state in their terms of service that such activity is not allowed.

[eatinglemur]

Originally Posted by jasonlambert

One individual yesterday sent 20,000 HTTP requests for PHP pages in 1 hour.

How rude!

[cheetahman]

Originally Posted by eatinglemur

How rude!

Very True

[Santa's little helper]

Originally Posted by jasonlambert

A reminder that using automated software to produce a local copy of this website for offline browsing (or any other reason) is strictly forbidden.

Out of curiosity isn't that what Google does ?

[kkubasik]

Not exactly, google uses a 'crawler' that follows links and behaves much more like a human (ie viewing 1 page at at time) automated downloaders tend to open several simultanious connections to a server at the same time and download as much content as possible. In addition search engines respect the file robots.txt which may contain a list of files to be indexed, and files not to be indexed. (linuxforums.org has one, here should you want to see one) its a fine line, but in the end, the only *serious* difference that I know is google does this in moderation, and behaves more like a fast reader would as opposed to a DOS style script which has no reguards for moderation.

[lakerdonald]

It also has to do with the frequency of the indexing. Somebody using one the aforementioned programs most likely will index a site more times than something like a google crawler would (and with the billions of pages indexed by google, I think the explanation is self-explanatory) causing bandwidth to be adversely affected more, and more often.

[ozar]

Originally Posted by jasonlambert

It is an automated DOS against my server. One individual yesterday sent 20,000 HTTP requests for PHP pages in 1 hour

Hmm... maybe that's what is causing the hiccup I talked about in this thread:

http://www.linuxforums.org/forum/topic-45213.html

[jasonlambert]

It's certainly possible.