FutureQuest, Inc. FutureQuest, Inc. FutureQuest, Inc.

FutureQuest, Inc.
Go Back   FutureQuest Community > FutureQuest Site Owners (All may read - Only Site Owners May Respond) > Questions & Suggestions
User Name
Password  Lost PW

Reply
 
Thread Tools Search this Thread Display Modes
Old 03-23-2006, 06:31 PM   Postid: 146625
Andilinks
Site Owner
 
Andilinks's Avatar

Forum Notability:
338 pts: An Honor To Be Around
[Post Feedback]
 
Join Date: Apr 2002
Location: San Antonio, Texas
Posts: 7,204
Xenu blocked globally?

Does this mean I can remove my .htaccess blocks of this referrer? I am finding a unique error message, "forbidden request" from all FQ hosted sites (and a few interesting ones that I didn't know were FQ clients).

I'm pleased with this, though it is not done by so many hosts yet that I need to find a new link checker. Tom Cruise must be pleased as well. (Xenu is the devil in the Scientology "religion.")

Andi
Code:
SetEnvIf User-Agent ^Xenu bad_bot=1
deny from env=bad_bot
__________________
@andilinks Twitter � Andrea Silver FB
Andilinks is offline   Reply With Quote
Old 03-23-2006, 07:28 PM   Postid: 146626
 Terra
CTO FutureQuest, Inc.
 
Terra's Avatar
 
Join Date: Jun 1998
Location: Z'ha'dum
Posts: 7,719
Re: Xenu blocked globally?

"Xenu" is not on our bad bot list...

Most likely, they are attempting HTTP methods that are not whitelisted?
GET, HEAD, POST, PUT

--
Terra
sysAdmin
FutureQuest
Terra is offline   Reply With Quote
Old 03-23-2006, 08:24 PM   Postid: 146629
Andilinks
Site Owner
 
Andilinks's Avatar

Forum Notability:
338 pts: An Honor To Be Around
[Post Feedback]
 
Join Date: Apr 2002
Location: San Antonio, Texas
Posts: 7,204
Re: Xenu blocked globally?

The log entry is the same (403) with or without the .htaccess blocking. I added the .htaccess code because Xenu was a bandwidth hog, something changed; I am not using the most current version of Xenu which is 1.2g released 03/05. I'm not complaining, I just find it odd that it is now blocked. Has "GET" just recently been disallowed?


Code:
68.35.xxx.xxx - - [23/Mar/2006:20:07:54 -0500] "GET / HTTP/1.1" 403 286 "-" "Xenu Link Sleuth 1.2f"
Code:
68.35.xxx.xxx - - [23/Mar/2006:20:12:34 -0500] "GET / HTTP/1.1" 403 286 "-" "Xenu Link Sleuth 1.2f"
__________________
@andilinks Twitter � Andrea Silver FB
Andilinks is offline   Reply With Quote
Old 03-24-2006, 02:48 AM   Postid: 146640
 Terra
CTO FutureQuest, Inc.
 
Terra's Avatar
 
Join Date: Jun 1998
Location: Z'ha'dum
Posts: 7,719
Re: Xenu blocked globally?

Aha...

Quote:
Link Sleuth

is being blocked...

When reporting rogue UserAgent's, please paste in the entire string as we do perform substring matches...

Another mystery solved...

--
Terra
sysAdmin
FutureQuest
Terra is offline   Reply With Quote
Old 03-24-2006, 11:44 AM   Postid: 146647
Andilinks
Site Owner
 
Andilinks's Avatar

Forum Notability:
338 pts: An Honor To Be Around
[Post Feedback]
 
Join Date: Apr 2002
Location: San Antonio, Texas
Posts: 7,204
Re: Xenu blocked globally?

I might add though that there is a downside to blocking Xenu, and that would be a gradual loss of backlinks and PageRank, which in the current climate also means a loss of traffic and revenue. Even hobby websites do depend to some extent on Google and other SE traffic.

I was blocking Xenu with the knowledge that this was the case but my site was subject to extra Xenu-abuse simply because I have a page about Xenu.

Xenu is a good application that is unfortunately easily abused. I have been corresponding sporadically with Xenu's developer Tilman Hausherr and he is resistant to changing the referrer name though I haven't proposed that he produce a version that is less easily abused.

This wholesale blocking has been an ongoing problem but it may be reaching a turning point for Xenu. There is a Xenu User Group on Yahoo and I think I will be bringing the subject up there, since both the abuse and the blocking hurts the internet generally.

In either case, the abuse or the loss of backlinks is a problem for everyone. I double check the Xenu rejects with iRider but this is not the case for all Xenu users because without iRider the checking is very tedious.

I am open to trying another link-checker but so long as Xenu is widely used blocking it hurts FQ clients, particularly those that are online to make money.

Andi
__________________
@andilinks Twitter � Andrea Silver FB
Andilinks is offline   Reply With Quote
Old 03-24-2006, 05:16 PM   Postid: 146660
georgeek
Registered User

Forum Notability:
332 pts: An Honor To Be Around
[Post Feedback]
 
Join Date: Jan 2004
Posts: 251
Re: Xenu blocked globally?

Quote:
Originally Posted by Andilinks
I might add though that there is a downside to blocking Xenu, and that would be a gradual loss of backlinks and PageRank
How come?

- George
georgeek is offline   Reply With Quote
Old 03-24-2006, 05:42 PM   Postid: 146666
Andilinks
Site Owner
 
Andilinks's Avatar

Forum Notability:
338 pts: An Honor To Be Around
[Post Feedback]
 
Join Date: Apr 2002
Location: San Antonio, Texas
Posts: 7,204
Re: Xenu blocked globally?

Quote:
How come?
To those who use Xenu to check links, FQ hosted sites look like broken/dead links which need to be removed. There is no way to gauge this effect in total but I think it is safe to assume that it exists, Xenu is quite popular.
__________________
@andilinks Twitter � Andrea Silver FB
Andilinks is offline   Reply With Quote
Old 03-24-2006, 06:21 PM   Postid: 146667
georgeek
Registered User

Forum Notability:
332 pts: An Honor To Be Around
[Post Feedback]
 
Join Date: Jan 2004
Posts: 251
Re: Xenu blocked globally?

Quote:
Originally Posted by Andilinks
To those who use Xenu to check links, FQ hosted sites look like broken/dead links which need to be removed. There is no way to gauge this effect in total but I think it is safe to assume that it exists, Xenu is quite popular.
Someone would have to be very naive to use Xenu to check for valid inbound links in the first place. Doubly so if they interpreted "error code: 403 (forbidden request)" as indicating anything other than Xenu was unable to spider the site.

- George
georgeek is offline   Reply With Quote
Old 03-24-2006, 06:38 PM   Postid: 146668
Jeff
Site Owner
 
Jeff's Avatar

Forum Notability:
872 pts: Dignified Competence!
[Post Feedback]
 
Join Date: Jun 2000
Location: Great Lakes
Posts: 4,223
Re: Xenu blocked globally?

Quote:
Someone would have to be very naive to use Xenu to check for valid inbound links in the first place.
The way I read the above, the fear is that many small to medium sized sites using Xenu to check their outbound links would delete those that Xenu marks as dead or forbidden thinking those sites are down. This seems to be Xenu's main purpose. I doubt someone with a link to your site on their homepage would go by Xenu only, but I imagine if you were down to Xenu you might lose many low-pagerank links to your site from various links pages of small sites where the webmasters use Xenu to make sure they are not including dead links on their sites.
Jeff is offline   Reply With Quote
Old 03-24-2006, 06:51 PM   Postid: 146669
Andilinks
Site Owner
 
Andilinks's Avatar

Forum Notability:
338 pts: An Honor To Be Around
[Post Feedback]
 
Join Date: Apr 2002
Location: San Antonio, Texas
Posts: 7,204
Re: Xenu blocked globally?

Quote:
Someone would have to be very naive...
There is naiveté out there, even if your premise were true. Inbound links aren't the issue, outbound links are. I haven't made this mistake but it is naive to assume that others won't.

And fact of the matter is that sometimes I will use Xenu's 403 as a default indication if the link isn't that important to me in the first place, and so some have already lost links from my site even though I'm aware of the problem (which not everyone is).
__________________
@andilinks Twitter � Andrea Silver FB
Andilinks is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 visitors)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -4. The time now is 12:47 PM.


Running on vBulletin®
Copyright © 2000 - 2014, Jelsoft Enterprises Ltd.
Hosted & Administrated by FutureQuest, Inc.
Images & content copyright © 1998-2014 FutureQuest, Inc.
FutureQuest, Inc.