phantom duplicate urls found by Google

Andilinks · 09-12-2008, 11:02 AM

This question is important only because Google traffic is important, and is only important to people who derive income or traffic from Google referrals.

Is there any way to get a legitimate url followed by a forward slash "/" to return a 404 instead of a version of the same page without css?

In a rational world I would pose this question to Google, but they have turned rationality on its head by antrhopomorphizing its dumb search engine. The modifier "dumb" is not a cute put-down, it is a serious assessment of its current behavior. Yes, I could somehow pose this question to Matt Cutts in a blog comment and probably will try soon if no one here has an answer to the question above.

Google has notified me that it has found duplicate pages on my site and then lists the same url twice, once with a fwd slash appended and once without. I have not yet been penalized for this nonsense but that will be coming if I don't figure out what to do. As an experiment I did change the name of one page and used a 301 redirect but the redirect also handles the slash appended url so gbot will still see it as two pages.

*Juan G* · 09-12-2008, 01:20 PM

Quote:

Originally Posted by Andilinks

Is there any way to get a legitimate url followed by a forward slash "/" to return a 404 instead of a version of the same page without css?

Instead of example.css:

<link rel="stylesheet" type="text/css" href="example.css" />

something to try would be /example.css:

<link rel="stylesheet" type="text/css" href="/example.css" />

Quote:

Originally Posted by Andilinks

Google has notified me that it has found duplicate pages on my site and then lists the same url twice, once with a fwd slash appended and once without. I have not yet been penalized for this nonsense but that will be coming if I don't figure out what to do. As an experiment I did change the name of one page and used a 301 redirect but the redirect also handles the slash appended url so gbot will still see it as two pages.

The Apache server adds the trailing slash to directories (see the thread Mod Rewrite to remove end slash). Probably you are referring to files with and without trailing slash, which probably need a redirect (excluding directories) to only one of the two versions. That should work.

Andilinks · 09-12-2008, 01:54 PM

Thanks for the reply Juan, I think this may be the answer. There is just one thing, about the code in that expired thread:

Code:

RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)(\.php|\.html)/$  /$1$2 [R=301,L]

Running experiments with mod_rewrite code on a live website has always ended badly for me so I'd like to confirm that the following changes are in order before I send it to the server:

Code:

RewriteCond %{REQUEST_myfile.shtm} !-d
RewriteRule ^(.+)(\.php|\.shtm)/$  /$1$2 [R=301,L]

*Juan G* · 09-12-2008, 07:43 PM

Well, the following are just possible, untested ideas.

If you only want to remove a possible trailing slash (excepting for directories), I think you can use:

Code:

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)/$ /$1 [R=301,L]

Or, if you also wish to remove "www." from all URLs (another possibility for duplicates):

Code:

RewriteEngine on

RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)(/?)$ http://example.com/$1 [R=301,L]

RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^(.*)(/?)$ http://example.com/$1/ [R=301,L]

If you had IRMs, probably adding this line (or other solution) would be necessary before each of those two RewriteRule:

Code:

RewriteCond %{HTTP_HOST} ^(www\.)?example\.com$ [NC]

Of course, write your domain instead of example.

Since these are just untested suggestions, maybe others will indicate mistakes or better ways. Also, different situations (for example IROs, etc.) require different solutions. You may see Apache 1.3 URL Rewriting Guide (FQuest is currently using that Apache version, I think).

Andilinks · 09-12-2008, 08:05 PM

Quote:

If you only want to remove a possible trailing slash (excepting for directories), I think you can use:

Code:
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)/$ /$1 [R=301,L]

OK, this is what I want to do, my question is, do I replace "{REQUEST_FILENAME}" with {REQUEST_myfile.shtm} or leave it as is? I know it must be obvious, it's just not obvious to me. :)

Since it's untested I'll wait until 3 am to try it.

Thanks.

*Juan G* · 09-12-2008, 08:19 PM

About REQUEST_FILENAME, leave it as is:

Quote:

REQUEST_FILENAME
The full local filesystem path to the file or script matching the request.

(Apache module mod_rewrite)

Good luck! :)

Andilinks · 09-12-2008, 08:20 PM

OK, I'll try it tonight and report back here in the morning. Thanks again!

Andilinks · 09-13-2008, 09:39 AM

It works perfectly, thanks. Now we'll just have to wait to see how long it takes Google to notice. Again, I'll report that event here when it happens.

The notice of duplicate title tags on pages identical except for the trailing slash has been on my Webmaster Tools Content analysis page for over a week—at first I just ignored them thinking it was just a ridiculous error. But as I've learned the hard way, ridiculous errors by Google can be very costly for me in dollar amounts that are trivial for Google but devastating for me.