PDA

View Full Version : To explain Google Email Warning: Googlebot found an extremely high number of URLs on your site



mario06
16-01-11, 10:17
Google Email Warning: Googlebot found an extremely high number of URLs on your site

I received this email (warning) from Googles Webmaster Tools

Please Note that I do not have vBSEO Installed.

I want a fix for this problem before google lowers my page rank or bans me altogether, my adsense revenue has already decreased by half and the Google Bots have literally attacked my site.

The is the second case of such a problem with vBET described here: http://www.vbenterprisetranslator.com/forum/vbet4-troubleshooting/1076-warning-google.html

I have sent the email to all vBET admins in PM as requested, I cannot publicly post the email as you should know because it goes against Googles TOS with my Local Privacy Laws as it would be a sure ban for my Google Adsense account so don't ask.

I need an immediate fix for this problem without further delay, it is 3 days that I've been talking about this in the link above.

Regards

yahoooh
16-01-11, 17:32
i think the reason that you enable all 52 languages
i feel that if vbe make the languages requst as ajax by click langages and thin bring the enabled links for other languages may help on this

tavenger5
16-01-11, 17:39
i get the same warning and I only have 12 languages enabled

vBET
16-01-11, 20:03
i think the reason that you enable all 52 languages
i feel that if vbe make the languages requst as ajax by click langages and thin bring the enabled links for other languages may help on this

No this is not the reason. Our and lot of other forums have included all languages and it is fine. So you can use as many languages as you can. The issue is somewhere else and at this moment we know 2 different reasons of that.

EDITED:
Now we know more reasons :) So please open your own thread and give us your mail from Google :)

vBET
16-01-11, 20:05
i get the same warning and I only have 12 languages enabled

Hi - please create new thread about your isue with your email. You got different lniks and we kow 2 reasons of this issue. To help we must know your case. So please open new thread about it and we will help you :)

vBET
16-01-11, 20:06
mario06 - I'm just during checking your links. I will give you answer in short time :)

Please note that as we described before - our staff is available to start working on issue when new ticked arrives in form of new thread about the issue. Otherwise we are not able to manage it and we inform our customer that he needs to open his own new thread about his issue. Just like you just did and now we are able to help you :)

vBET
16-01-11, 20:38
Hi :)

One of reasons (there will be others - still during investigation) of your issue was that you allowed to index such pages as member pages. Those are not relevant and most member pages are in 99% same - only differs by member username. So when it was additionally translated you made 52 times more such pages which are almost same (same pages content after translation will give same translated content). We officially advise to block irrelevant pages by robot.txt - here you will find how to do this: http://www.vbenterprisetranslator.com/forum/general-discussions/243-vbet-performance.html#post1178

Please note that this will make those pages disappear form Google index and this is good for your SEO since Google will see and index only relevant pages with relevant content.

Also as I see you made changes since Google generated this email, because as I see you blocked member pages for guests. This also solves this part of issue. Still if you want guest see member pages, but Google not index it - just use robots.txt for that :)

I see that there are also links for archive and threads - I'm going to analyze this right now :)

vBET
16-01-11, 21:22
Hi again :)

I think I know what can be the other cause - as I'm looking on flags links there is no link without language= at all... Please tell what is your default forum language - for this language link should be without language= because now every page you have duplicated fo Google. Once for normal URL and second time with language=YOUR_DEFAULT_LANGUAGE.

Also please tell did you changed our template named vbenterprisetranslator_flag? We need to know is it vBET bug or generated by customer change.

The reason I think could be also (cannot check it now but we already have TODO task which will allow to check it in the future) that you just had cleaned cache and there was issue with Google connection - in that case pages generate without content (because translation are not obtained) and literally content is same for all translations. But such things if happen are temporary and if you are no blocked by Google those disappears itself - those are issues between your and Google server and vBET can do nothing with this. I do not tell that it was the reason. I think it could be. As I'm looking now on links given by Google I see no described issue at all. So it could be it.

Still - please answer questions about default language and template.

I see there one more thing – your translated archive pages for subforums are not working and give empty results. Main and post pages are OK. For archive subforum pages there is error. This also makes lot of links going to same content – empty content and this also can be reason why Google wrote you this. Please tell which version of vB are you using? Supported 3.8 or some lower – as I remember such thing happens on 3.6 because of vB bug. If it is not supported version and vB bug – I will find thread about it and I strongly recommend to write to vB team about solving this bug. Unfortunately I expect that they will not do this since it is not supported version anymore. But – first please tell which version are you using, so I will no waist more words and your time for speculations ;)

vBET
16-01-11, 21:39
Hi again again :)

This time I checked your show thread links and I see 2 things here
1. No one is guilty - as I saw few of those links (not all see number 2) all those threads was very short having only one sentence in it. So all those were very similar by default - 99% of content was layout. It has nothing to do with vBET - it is just the content of your own threads. vBET only translates it so it really is more similar pages for those, but if you disable vBET not translated pages will still be similar to each other if you have only one sentence in only post in thread. So we have nothing to do here.
2. Maybe you are killing your forum yourself by limiting number of posts to see by guests. Now all pages for me have same content with communicate that I should register. Remember that Google is looking your page as guest - so if your limiting mod is stupid it will kill your forum, because Google will see everywhere same content telling him to register ;) Please make sure that limiting mod is smart enough or disable it.

mario06
17-01-11, 09:19
Hi again again :)

This time I checked your show thread links and I see 2 things here
1. No one is guilty - as I saw few of those links (not all see number 2) all those threads was very short having only one sentence in it. So all those were very similar by default - 99% of content was layout. It has nothing to do with vBET - it is just the content of your own threads. vBET only translates it so it really is more similar pages for those, but if you disable vBET not translated pages will still be similar to each other if you have only one sentence in only post in thread. So we have nothing to do here.
2. Maybe you are killing your forum yourself by limiting number of posts to see by guests. Now all pages for me have same content with communicate that I should register. Remember that Google is looking your page as guest - so if your limiting mod is stupid it will kill your forum, because Google will see everywhere same content telling him to register ;) Please make sure that limiting mod is smart enough or disable it.

1. It's an Italian Forum, the content in the posts is average and this has nothing to do with what you said.

2. I'm not killing the forum because the please register mod after 3 visits (one of the most popular installed mods) has nothing to do with the bots as they go through, even children know that furthermore I've been using vBET ever since the free version came out (about a year?!?) and nothing has been changed or added since then.

The problem is that my forum is considered to be a medium forum with 368,000 threads, when Google sends it bots on my site they effectively see 368,000 threads BUT on www.google.52languages they were seeing 1.540,000 indexed threads which caused the google bot to attack my site trying to find where they are. My cache was set for five days with all languages being cached BUT now (testing) I've set it to 2 days with 4 languages being cached, at this point I can confirm that the problem IS vBET because now I'm down to 766,000 indexed threads and the Google bots have stopped attacking my site.

Example: Your site has 5,578 effective threads BUT Google sees 334,000 indexed threads, do you understand where the problem is?

Without offense you must find out what is happening with google and find a workaround for the problem BECAUSE Google has found out the 52x languages is fake|duplicate indexed links meaning that the translated links don't physically exist and I can't argue that since it's true, it's cached on our servers, it's like cheating.

On small forums they don't pay much attention but on medium|big forums it doesn't pass by unobserved.

You should start investing the language parameter to URL ('address?language=en') this is what is triggering Googles bots to verify vBET powered sites. You have vBSEO (I Don't) which means that you are using the different language parameter which at the moment is passing by Googles attention.

Alot of other vBET users are having the same problem so it's not a coincidence!!

PS. Google Help Center Statement: In general, this message is shown when we discover a relatively high number of URLs for your site.

Regards

vBET
17-01-11, 17:45
1. Please turn on Italian flag on your forum (I didn't find it) - right now users who changed language are not able to turn back to original language. As I wrote content of threads which I was able to see before I saw that I have to register was only one sentence in all cases. This it not what personally I assume as average length of content. Once again this makes pages looks very similar since the difference between short threads is only one sentence - the rest is layout which is same. I do not know how you are so sure that it has nothing to do with this case. If you know something what we do not please just share it so we will be able to help you faster. Do you know why this email was sent?

2. I do not knot what is school program for children in your country, but I hope that it will be very useful for them to know that please register mod after 3 visits has nothing to do with the bots as they go through.
Please focus on the issue. And give required answer - I didn't know which mod are you using and I didn't even ask because it is not important for me - important was is it boot safe. If you are 100% sure that it is safe then great. Just in case you had any doubts you can see how Google sees your forum in Webmaster Tools.

Please note that Google boots are not attacking any sites - Google do not harm sites. We advice to use cache for longer allowed period, because it is better for performance. Also if you feel that it for sure the only reason is number of translation on your page - feel free to disable some of those and enable it again one by one from time to time. Still please consider that your license is active for 6 months and you got this email now. It is hard to believe that for 6 months Google didn't saw those links - in fact many of those you have already indexed. So I'm not sure that this is the cause of email which you got. Even if it is please consider all information which was given to you after analyze of those links.

I do not know what you mean by effective threads.

Also please note that we are working on this issue - you got results of our analyze. And please hold with telling what Google considers - because it is you who considers that Google find it not correct. WE have lot of forums which have absolutely no issue with Google and have all languages turned on. And it is not true that content is not physically exists, please do not tell what we can argue because something... The content exists for everyone who visits it and it physically exists in database in original and in cache in translated form. This is not worlds lottery - this is translation it couldn't happen if content didn't exists. And nothing is cheating here - please read Google TOS - this is Google who claims that translation can be cached for max 15 days. It is EVERYTHING OK with Google in case of automatic page translations - it is Google who made this translation :)

Also no client can tell that any content which is streamed to him is from static file or not. In fact in this days almost nothing is static now. In vBulletin no view hoes from static HTML structure - everything base on data from database (languages, templates, styles). I have no idea what is your knowledge about client server architecture with thin client, but please note that content exists if link responses. Telling that Google feels it cheating because it doesn't exists has nothing to do with reality.

Also please note that You can in every moment use vBET SEO links and have /en/ instead of language= - this configuration is just up to you and vBET supports it without vBSEO.

About note which you gave - as I wrote if you feel you discovered the real reason of your issue then you have solution. Still please consider that this is some generic answer without even checking what happens. I wouldn’t count on it:
1 - you have similar amount of links for half year.
2 - Google mail itself tells:
"Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot."

So once again please consider our advices and analyze result. If you feel it is only issue of number of translations which you turned on - then you know the solution.

Do you still need our help/advise in this issue?

mario06
17-01-11, 18:46
The only thing that was turned off was the Italian language (I turned it on) since it is the forums native language I didn't think it mattered anyway I was good since a year ago so whats up now?

Here's an example (your site) of what I mean when I say effective indexed posts.

Before vBET (effective forum indexed stats):
http://img203.imageshack.us/img203/275/5582.jpg

After vBET (not effective forum indexed stats):
http://img266.imageshack.us/img266/8489/3330001.jpg

What I'm trying to say is that the 52x language translated posts are now being considered duplicate posts by Google. The email is self explanatory

In my own words when I say a bot attack I mean when you have 300 Google bots heavily crawling your website at the same time eating and tearing away on your bandwidth, ram and cpu and causing forum stalls. And they wrote it in their letter below in a more professional manner but the meaning is the same.

Official Google Email:
Googlebot encountered problems while crawling your site http://mysite.com/.
Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.

Did you read what the Google support team wrote me? -> In general, this message is shown when we discover a relatively high number of URLs for your site.

You wrote everything and nothing because I don't understand your English, just give me a fix in simple steps like;
1. Do this
2. Do that
3. And this
etc.

Regards

mario06
19-01-11, 06:03
Still waiting for a remedy.

Regards

vBET
20-01-11, 19:36
The only thing that was turned off was the Italian language (I turned it on) since it is the forums native language I didn't think it mattered anyway I was good since a year ago so whats up now?

It is just advice from our team that without this flag your users are not able to turn back to original language when they click on some flag. So it is wise to keep flag for your default language :)


Here's an example (your site) of what I mean when I say effective indexed posts.

Before vBET (effective forum indexed stats):
http://img203.imageshack.us/img203/275/5582.jpg

After vBET (not effective forum indexed stats):
http://img266.imageshack.us/img266/8489/3330001.jpg

What I'm trying to say is that the 52x language translated posts are now being considered duplicate posts by Google. The email is self explanatory


This is NOT true. You have issues with your forum. You have broken archive - and most links you got from Google in email are about archive. You totally ignored our support - you didn't answer questions about version and keep telling what you think and what is not true. If you want to solve your issue please fallow our support hints. We asked you about your vB version because you have broken archive. Is it supported 3.8 or some not supported version? We know that archive is broken in 3.6 because of vBulletin bug. This gives you thousands of not working links - and you got from Google email about wrong links. Instead of listen to our support and cooperate you keep telling some ridiculous idea and you think it is important to put it in red.
What is your purpose here? You want our support to solve your issue or you are just trying to scare others? Because if you want to solve this issue please cooperate with our support.

Also about image which you gave - this is your misinterpretation. Google doesn’t care about number of posts shown by forum. And it cannot care about it. It is because Google indexes all the pages which are not blocked! On forum you have member pages, groups, bogs perhaps, albums and so on... and so on... Also forum can have lot of other plugins which creates additional pages (like arcade).
So please understand: you are wrong. Google do not consider translated content as duplication. You have issue with your own forum and we discovered areas of possible cause. So please - fallow our instructions/questions. Otherwise this can be marked as solved since you seem be not interested with our support just like to repeat what you wrongly think - we already read this - there is no any need to repeat it again. This is not true. Please focus on solving your issue if you want it to be solved. Please answer the question and keep attention on your archive.


Official Google Email:
Googlebot encountered problems while crawling your site http://mysite.com/.
Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site's URL structure. Googlebot may unnecessarily be crawling a large number of distinct URLs that point to identical or similar content, or crawling parts of your site that are not intended to be crawled by Googlebot. As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.

EXACTLY! YOU HAVE WRONG STRUCTURE! YOU HAVE WRONG CONTENT! and we wrote it to you at the very beginning (http://www.vbenterprisetranslator.com/forum/troubleshooting/1394-google-email-warning-googlebot-found-extremely-high-number-urls-your-site.html#post6846) - you have crashed archive where are thousands of not working links! So please start using our support which you asked for instead of trying to scare our users. What are you expecting from us when you ask for support and ignore it?


You wrote everything and nothing because I don't understand your English, just give me a fix in simple steps like;
1. Do this
2. Do that
3. And this
etc.

Regards

Here you got exact information that you should block member pages and how to do this:
http://www.vbenterprisetranslator.com/forum/troubleshooting/1394-google-email-warning-googlebot-found-extremely-high-number-urls-your-site.html#post6845

Here you got information that your archive is crashed, question about your version of vBulletin and what to do if it is not supported version:
http://www.vbenterprisetranslator.com/forum/troubleshooting/1394-google-email-warning-googlebot-found-extremely-high-number-urls-your-site.html#post6846
This is very exact instruction - please tell which version of vBulletin are you using.

And here you got information that you jut shouldn't do anything with posts which are very short (just 1 sentence) because this is issue of the content not of vBET. When content will get longer (users will get replies) those pages content will be not similar anymore:
http://www.vbenterprisetranslator.com/forum/troubleshooting/1394-google-email-warning-googlebot-found-extremely-high-number-urls-your-site.html#post6847

So:
1. Keep your member pages blocked in way like you made it recently (only for registered users) or by robots.txt like it is described here: http://www.vbenterprisetranslator.com/forum/general-discussions/243-vbet-performance.html#post1178
2. Check in Webmaster Tools does your "3 visits for Guest" mod is Google save – check how Google see your forum by giving URL to some thread. Check several times (cannot tell does Google will use same IP for each check). This way you will be really sure that the mod is save for robots.
3. You have broken archive – please tell which version of vBulletin are you using?
4. For short threads you can just give some longer answers there (different of course) or just keep it as it is hoping that people will answer those threads.

PS.
Please take my appologies for late answer. I was at delegation and Kamil was told to leave this issue to me. So once again sorry :)

pepito69
20-01-11, 22:50
Hello,
I have to say that Michal is right, i had the same warning from google and it's easy to solve if you look at waht URLs are causing the problem. In my example, i had a lot of duplicated content because vbet inset: &language=en, in a French URL. If you go to parameters in webmaster tools you can solve this excluding this parameter. And also look at your robots file. I also have disallow in robots archive files, don't need it. There is a lot of work to do with robots.txt if you use VBET.
And i don't suggest to use more than 20 languages, you don't need it :)

mario06
21-01-11, 07:56
Double Post, Sorry!!

mario06
21-01-11, 08:38
It is just advice from our team that without this flag your users are not able to turn back to original language when they click on some flag. So it is wise to keep flag for your default language :)



This is NOT true. You have issues with your forum. You have broken archive - and most links you got from Google in email are about archive. You totally ignored our support - you didn't answer questions about version and keep telling what you think and what is not true. If you want to solve your issue please fallow our support hints. We asked you about your vB version because you have broken archive. Is it supported 3.8 or some not supported version? We know that archive is broken in 3.6 because of vBulletin bug. This gives you thousands of not working links - and you got from Google email about wrong links. Instead of listen to our support and cooperate you keep telling some ridiculous idea and you think it is important to put it in red.
What is your purpose here? You want our support to solve your issue or you are just trying to scare others? Because if you want to solve this issue please cooperate with our support.

Also about image which you gave - this is your misinterpretation. Google doesn’t care about number of posts shown by forum. And it cannot care about it. It is because Google indexes all the pages which are not blocked! On forum you have member pages, groups, bogs perhaps, albums and so on... and so on... Also forum can have lot of other plugins which creates additional pages (like arcade).
So please understand: you are wrong. Google do not consider translated content as duplication. You have issue with your own forum and we discovered areas of possible cause. So please - fallow our instructions/questions. Otherwise this can be marked as solved since you seem be not interested with our support just like to repeat what you wrongly think - we already read this - there is no any need to repeat it again. This is not true. Please focus on solving your issue if you want it to be solved. Please answer the question and keep attention on your archive.



EXACTLY! YOU HAVE WRONG STRUCTURE! YOU HAVE WRONG CONTENT! and we wrote it to you at the very beginning (http://www.vbenterprisetranslator.com/forum/troubleshooting/1394-google-email-warning-googlebot-found-extremely-high-number-urls-your-site.html#post6846) - you have crashed archive where are thousands of not working links! So please start using our support which you asked for instead of trying to scare our users. What are you expecting from us when you ask for support and ignore it?



Here you got exact information that you should block member pages and how to do this:
http://www.vbenterprisetranslator.com/forum/troubleshooting/1394-google-email-warning-googlebot-found-extremely-high-number-urls-your-site.html#post6845

Here you got information that your archive is crashed, question about your version of vBulletin and what to do if it is not supported version:
http://www.vbenterprisetranslator.com/forum/troubleshooting/1394-google-email-warning-googlebot-found-extremely-high-number-urls-your-site.html#post6846
This is very exact instruction - please tell which version of vBulletin are you using.

And here you got information that you jut shouldn't do anything with posts which are very short (just 1 sentence) because this is issue of the content not of vBET. When content will get longer (users will get replies) those pages content will be not similar anymore:
http://www.vbenterprisetranslator.com/forum/troubleshooting/1394-google-email-warning-googlebot-found-extremely-high-number-urls-your-site.html#post6847

So:
1. Keep your member pages blocked in way like you made it recently (only for registered users) or by robots.txt like it is described here: http://www.vbenterprisetranslator.com/forum/general-discussions/243-vbet-performance.html#post1178
2. Check in Webmaster Tools does your "3 visits for Guest" mod is Google save – check how Google see your forum by giving URL to some thread. Check several times (cannot tell does Google will use same IP for each check). This way you will be really sure that the mod is save for robots.
3. You have broken archive – please tell which version of vBulletin are you using?
4. For short threads you can just give some longer answers there (different of course) or just keep it as it is hoping that people will answer those threads.

PS.
Please take my appologies for late answer. I was at delegation and Kamil was told to leave this issue to me. So once again sorry :)

1. My robots.txt has been the same, I didn't change anything and it seems fine:
User-agent: *
Disallow: /admincp/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /members/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /modcp/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
2. No it doesn't save
3. My archive works fine, all links tested and working, where did you see it broken?
4. The only small posts are where the new members present themselves and just say "Hello, my name is Tom" I can't do anything about, 90% of the posts are guides meaning medium sized.


pepito69 said something interesting which involves the parameter url like I said above
In my example, i had a lot of duplicated content because vbet inset: &language=en, in a French URL. If you go to parameters in webmaster tools you can solve this excluding this parameter.
but he only got it for one language I'm getting it for all languages so is this the reason and what must I do exactly?

PS. I don't have vbSEO installed BUT use vBSEO sitemap which already excludes the archive, am I suppose to exclude it the robots.txt to? I never seen anyone here or any other place exclude it in their robots.txt

I want simple detailed instructions,
EXAMPLE:

1. Do this
2. Do that
3. And that

pepito69,
I never seen such a big robots.txt as you have, is all that really necessary?
In realty there are only 3 bots that are considered Bad Bots* and it wouldn't it be easier to put them in your .htaccess file, no?
PS. Your flags are not showing

*Baiduspider
*Mozilla/5(.*)Twiceler
*Yeti

pepito69
21-01-11, 18:30
Thanks Mario,

Yes, my robots file maybe is to big, isn't optimized but it works for me and don't hurt me.
I will include this bots in my htaccess, thanks.


but he only got it for one language I'm getting it for all language

No my friend, i also have it in all languages, that was just an example. But, since i used the parameters in webmaster tools i have increased my google crawl again. I still don't know how to solve other problems with VBET, more important for me, like the obe i have with the code language and the broken words that use some accents (ó, ñ, etc).
Well, i exclude archive in robots.txt, and i don't have any archive webpage in google index.
I thought many times to use vbseo sitemap but my site is very well crawled buy google. I always apply this rule: if something works, don't touch it.

Sorry i can not tell you what you must do, i only share my experience.

Thanks . My flags don't show in /index.php but are visible in the forums site, i will solve it now.

vBET
21-01-11, 20:55
1. My robots.txt has been the same, I didn't change anything and it seems fine:
User-agent: *
Disallow: /admincp/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /members/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /modcp/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
2. No it doesn't save
3. My archive works fine, all links tested and working, where did you see it broken?
4. The only small posts are where the new members present themselves and just say "Hello, my name is Tom" I can't do anything about, 90% of the posts are guides meaning medium sized.


pepito69 said something interesting which involves the parameter url like I said above
but he only got it for one language I'm getting it for all languages so is this the reason and what must I do exactly?

PS. I don't have vbSEO installed BUT use vBSEO sitemap which already excludes the archive, am I suppose to exclude it the robots.txt to? I never seen anyone here or any other place exclude it in their robots.txt

I want simple detailed instructions,
EXAMPLE:

1. Do this
2. Do that
3. And that

pepito69,
I never seen such a big robots.txt as you have, is all that really necessary?
In realty there are only 3 bots that are considered Bad Bots* and it wouldn't it be easier to put them in your .htaccess file, no?
PS. Your flags are not showing

*Baiduspider
*Mozilla/5(.*)Twiceler
*Yeti

1. It wasn't same - otherwise Google wouldn't show you link to member page because Google doesn't visits pages blocked by robots.php and you got in warning email this link:

http://YOUR_PAGE/member.php?u=63920&language=ro
Now it is OK. It wasn't some time ago and Google was there. You can be calm now about this part :)
2. If it is not safe for robots and shows to robots blocking communicate you should seriously consider removing this mod. Otherwise Google will start indexing pages which he see - so pages with blocking communicate. Also in this case it is main cause of your issue - all pages looks for Google same - just blocking communicate.
As I understand you checked in webmaster tools how Google see your forum and you saw that Google see communicate that he should register - yes?
3. As I wrote in the beginning your links to sub-forums in archive aren't working at all. Only to main forum and to threads. This makes thousands of wrong links on your archive pages. For example:

http://YOUR_FORUM/archive/index.php/t-25194.html?language=pl
This is link to tread archive and it is working but in it content there are links on the top of page to sub forums where the thread is included. Example:

http://YOUR_FORUM/archive/index.php/f-182.html
And all those links are not working - only to main forum. All archive sub-forums links are not working. This makes you thousands of not working links on thousands of archive threads pages.
YOU STILL DIDN'T TOLD US WHICH VBULLETIN VERSION ARE YOU USING...
At this moment I can bet that it is not supported vBulletin 3.6 - please do not worry we will not stop supporting you. We need to know it to help you better, because in 3.6 it is known vBulletin bug. If it is other version it is possible that it is vBET bug. It is critical to discover what is the cause to help you.
4. I'm not writing about 90% of your posts. I was writing about posts listed in Google mail. Others are out of scope of this discussion. Google listed you only threads with one sentence as I wrote you (at least those which I was able do see before your mod told me to register). Do you understand now? It is not the issue with whole your forum. As I already wrote - I would not worry about those - people will write answers there and those will not be similar anymore - just like 90% of your posts which looks completely different.

Please note that pepito69 can have different configuration that you. In your case ignoring language parameter is not advised, since you are not using SEO links and this is only way to recognize that URLs are different. If you tell Google to ignore this attribute for Google all URLs witch differs only value of language parameter will be considered as same content - this will create chaos in indexed pages which will randomly change languages in Google index, or Google will just index your main language pages. I think so - newer checked and do not advise for your case. Please do not do that as long as you do not use SEO links.

Simple instructions at this moment:
1. If your "3 times mod" is not Google safe – disable it and contact mod author about it.
2. TELL US WHICH VERSION OF VBULLETIN ARE YOU USING.

This is 3rd time we are asking about this... How you are expecting us to help you if you do not cooperate. WE do not have magic wand which will create solutions. We need to know what is going on on your server.

About big .htaccess - I do not know what you are writing about exactly, but I bet that for you it is not necessary such big - I assume that there are rules to block vBSEO links for member pages and so on. You are not using vBSEO and you are not using SEO links - your small robots.txt is fine.

mario06
22-01-11, 07:24
TELL US WHICH VERSION OF VBULLETIN ARE YOU USING.
I'm using version 3.6.12, you should know this since I paid an additional fee to get it working (make it compatible) because of the archive problem so if it's the archive thats causing this then it's (no offense) YOUR FAULT since you did it. : http://www.vbenterprisetranslator.com/forum/vbet-integration-other-plugins/810-vbet-vbulletin-3-6-a.html


If your "3 times mod" is not Google safe – disable it and contact mod author about it.
Author said it doesn't harm google.


1. It wasn't same - otherwise Google wouldn't show you link to member page because Google doesn't visits pages blocked by robots.php and you got in warning email this link:
1. The only thing that i added was /members/ .. everything else including memberslist.php was already there, MY FAULT, so let's close this argument.


2. If it is not safe for robots and shows to robots blocking communicate you should seriously consider removing this mod. Otherwise Google will start indexing pages which he see - so pages with blocking communicate. Also in this case it is main cause of your issue - all pages looks for Google same - just blocking communicate.
2. There is absolutely no need to be rude, I do understand how to use the robots.txt, but in your optimization guides (my Opinion) your lean more towards vBSEO setups, just keep in mind that at least 80% of vBulletin users don't use vBSEO so you should keep 2 separate threads when you talk about tips, tweaks and optimization, example; "vBSEO Optimization" and "Non vBSEO Optimization" to make it more user friendly.


3. All the archive links that you posted as broken seem to be working fine for me, here's a snapshot of your broken archive link example:


http://YOUR_FORUM/archive/index.php/t-25194.html?language=pl
http://img138.imageshack.us/img138/364/43312523.jpg


4. htaccess comment wasn't for you but for the member pepito69


Listen Michael I don't want to argue every single letter that's written with you, I just want an easy 1. 2. 3. fix and nothing more.

Best Regards


[OVERTIME]
From what the google representative told me, the url structure below is being indexed 4 times but in realty for them it's considered to be 1 and should be indexed as such and that's probably why I got the email. The underneath post is just an example;


http://www.mysite.com/showthread.php?t=64661 Original
http://www.mysite.com/showthread.php?t=64661&language=pl Translated
http://www.mysite.com/showthread.php?t=64661&language=en Translated
http://www.mysite.com/showthread.php?t=64661&language=es Translated
[OVERTIME]

vBET
22-01-11, 19:17
TELL US WHICH VERSION OF VBULLETIN ARE YOU USING.
I'm using version 3.6.12, you should know this since I paid an additional fee to get it working (make it compatible) because of the archive problem so if it's the archive thats causing this then it's (no offense) YOUR FAULT since you did it. : http://www.vbenterprisetranslator.com/forum/vbet-integration-other-plugins/810-vbet-vbulletin-3-6-a.html

We have lot of users. We know what we have included in actual thread. Thanks for giving your vBulletin version :)

About the issue. You was informed that integration can be not possible since your version is not supported and you decided to go for it anyway. Bug in archive which you have is on vBulletin side. You was also informed in PM about it.
So you was aware about not working archive sub-forums on your not supported old vBulletin version, from the very beginning. You was also told that because this is vBulletin bug you should contact vBulletin team to solve it.
Also as I see in the very thread you are pointing on you refused our future support. But of course you have full right to blame anyone for anything :)

What to do now with archive? You have now several options:
1. Contact vBulletin team to solve their bug. As you know (from old stories) vBUlletin 3.6 have bug and when you add any additional parameter in URL to archive sub-forum page then it crashes. It is out of vBET scope since old vBulletin behave this way even when vBET is not installed - so in short words: it is vBulletin bug. It is also out of our scope because it is not supported version. So you can ask vBuletin are they are still responsible for their code and will they correct it. They can refuse if they also consider vB3.6 as unsupported.
2. You can block whole archive in robots.txt - not good idea we do not advise it.
3. You can disable translation of archive pages. There will be less pages in Google index, but there will be no more broken links.
4. You can edit layout of your thread archive page so links to archive sub-forums will not be included. You can make it included for not translated page and disabled for translated one. On your main forum archive page there broken links to sub-forum will be still on, but this is only several links not thousands on thousands archive thread pages. You could even made 301 redirection for main forum archive page so it will be only in original language.

Please tell us which version do you prefer and we will guide you more in chosen one.


Author said it doesn't harm google.
I'm lost here. First you wrote it is safe. Then you wrote it is not safe. Now it is safe again. What I'm interested is what I wrote to you about Webmaster Tools. We are not interested what mod author is telling you - this is between you and him. We are interested how Google see your forum. If Google see your forum as blocked communicate then the mod harms Google, because Google see everywhere same content (so each post is considered as duplicated content). In this case we advice to not use the mod anymore until it will be safe for Google - so until you will be able to see in Webmaster Tools that Google see all the pages normally and it is not blocked. This is how it is and we have nothing more to say in this area. It is your choice will you fallow advice or not. So check it first and if it is not safe then made your own decision who you want to listen.


2. There is absolutely no need to be rude, I do understand how to use the robots.txt, but in your optimization guides (my Opinion) your lean more towards vBSEO setups, just keep in mind that at least 80% of vBulletin users don't use vBSEO so you should keep 2 separate threads when you talk about tips, tweaks and optimization, example; "vBSEO Optimization" and "Non vBSEO Optimization" to make it more user friendly.

I apologize if you consider our answer as rude. I just added in TODO list to make separate robot.txt file for non SEO links. Thanks for your note.


3. All the archive links that you posted as broken seem to be working fine for me, here's a snapshot of your broken archive link example:


http://YOUR_FORUM/archive/index.php/t-25194.html?language=pl
http://img138.imageshack.us/img138/364/43312523.jpg

Please read carefully and once again what I wrote to you in last response. I never wrote that the link:

http://YOUR_FORUM/archive/index.php/t-25194.html?language=pl
is broken. I wrote that IT IS WORKING - broken are links included ON THIS PAGE. Please read again what was described before. I know that this page is working. That is why I wrote you:

For example:

http://YOUR_FORUM/archive/index.php/t-25194.html?language=pl
This is link to tread archive and it is working but in it content there are links on the top of page to sub forums where the thread is included. Example:

http://YOUR_FORUM/archive/index.php/f-182.html
And all those links are not working - only to main forum. All archive sub-forums links are not working.
I wrote you that this link is working - you do not have to give me screen shots and prove to me that it is working ;) Just please read carefully what I'm writing to you. And now look on your image and pay attention on top links (links to archive sub-forums) - those are broken :) (except link to archive main forum)


Listen Michael I don't want to argue every single letter that's written with you, I just want an easy 1. 2. 3. fix and nothing more.

Great :) Hope you understand the area of issue now and know what are the choices. Please tell us which one you prefer and we will guide you in this direction.


From what the google representative told me, the url structure below is being indexed 4 times but in realty for them it's considered to be 1 and should be indexed as such and that's probably why I got the email. The underneath post is just an example;


http://www.mysite.com/showthread.php?t=64661 Original
http://www.mysite.com/showthread.php?t=64661&language=pl Translated
http://www.mysite.com/showthread.php?t=64661&language=en Translated
http://www.mysite.com/showthread.php?t=64661&language=es Translated

All those pages have completely different content and any boot couldn't tell that it is same when the text is completely different. Those are different pages which serves different content targeted for different receiver, and the answer have no sense for us at all. Please PM me whole message you got, with contact to source of this message. We would like to explain it. Most probably person who check it saw not real content, but only blocking message (like I did right now) - so ins such case the feedback have no value at all. Once again - please PM details we will verify it.

mario06
23-01-11, 09:22
http://img31.imageshack.us/img31/2613/95809131.jpg
Here you clearly state that this link wasn't working (wrong means the same) marked in red and now you say that you never said it?!?


I'm lost here. First you wrote it is safe. Then you wrote it is not safe. Now it is safe again. What I'm interested is what I wrote to you about Webmaster Tools. We are not interested what mod author is telling you - this is between you and him. We are interested how Google see your forum. If Google see your forum as blocked communicate then the mod harms Google, because Google see everywhere same content (so each post is considered as duplicated content). In this case we advice to not use the mod anymore until it will be safe for Google - so until you will be able to see in Webmaster Tools that Google see all the pages normally and it is not blocked. This is how it is and we have nothing more to say in this area. It is your choice will you fallow advice or not. So check it first and if it is not safe then made your own decision who you want to listen.

Since I'm English and know the language very well, I've always said that that limited guest mod doesn't hurt, harm, bother, supreme, suppress the google bots.



About the issue. You was informed that integration can be not possible since your version is not supported and you decided to go for it anyway. Bug in archive which you have is on vBulletin side. You was also informed in PM about it.
So you was aware about not working archive sub-forums on your not supported old vBulletin version, from the very beginning. You was also told that because this is vBulletin bug you should contact vBulletin team to solve it.
Also as I see in the very thread you are pointing on you refused our future support. But of course you have full right to blame anyone for anything

You told me that for an additional fee you would have made my version compatible then after the work you said that their was a problem with the archive which I didn't find because all the links worked for me. I contacted vbulletin (when you told me that) and they said that everything was ok.


I contacted The vBulletin Support again today and I got a similar response as a year a so ago; in simple words my archive WORKS AS IT SHOULD! Read vBulletins comments below, I hid the name for pivacy reasons but I can Assure you that he is the oldest member there so now what?
http://img153.imageshack.us/img153/9333/vbulletinm.jpg
http://img203.imageshack.us/img203/5306/77575048.jpg

PS. If your still unsatisfied with their response and want me to ask a more technical question post it here and I will copy/paste it to the vBulletin Support Team directly ok?

Since vBulletin said there's no bug and everything is working fine the problem is on your end so I'm waiting for the fix.

Best Regards

vBET
24-01-11, 19:05
http://img31.imageshack.us/img31/2613/95809131.jpg
Here you clearly state that this link wasn't working (wrong means the same) marked in red and now you say that you never said it?!?

You are really wasting the time. And once again you didn't read carefully. Please - read whole point. If I write you: "I'm little confused" - you will gonna prove me that I wrote you that I'm little??? What it is about anyway?...

This is what I wrote:

3. As I wrote in the beginning your links to sub-forums in archive aren't working at all. Only to main forum and to threads. This makes thousands of wrong links on your archive pages. For example:

http://YOUR_FORUM/archive/index.php/t-25194.html?language=pl
This is link to tread archive and it is working but in it content there are links on the top of page to sub forums where the thread is included. Example:

http://YOUR_FORUM/archive/index.php/f-182.html
And all those links are not working - only to main forum. All archive sub-forums links are not working. This makes you thousands of not working links on thousands of archive threads pages.
I gave you example of location where you will find broken links and next example of broken link itself (broken after translation). Sorry - but what is the problem here? Do you still do not know what it is about or you are trying to prove something here?
Also please note that you can use quotes - including images is not necessary :)


You told me that for an additional fee you would have made my version compatible then after the work you said that their was a problem with the archive which I didn't find because all the links worked for me. I contacted vbulletin (when you told me that) and they said that everything was ok.

Not true - what I wrote you you have in your PMs and I also. EOT. Read again if need to remind this. We already started quoting our PMs in other thread (you linked it here) so no need to start it again. Any other content of PM included publicly here will be considered as SPAM, since it is already in other topic. Any duplication made by same user is considered as SPAM on this forum and on any other forum I know. So please – do not start to SPAM.


I contacted The vBulletin Support again today and I got a similar response as a year a so ago; in simple words my archive WORKS AS IT SHOULD! Read vBulletins comments below, I hid the name for pivacy reasons but I can Assure you that he is the oldest member there so now what?
http://img153.imageshack.us/img153/9333/vbulletinm.jpg
http://img203.imageshack.us/img203/5306/77575048.jpg

Archive sub-forum pages on vBulletin 3.6 crashes if you ad any additional parameter to URL. You can check it even on clean instance - without vBET nor any other mod. I checked it on your forum with vBET disabled. So it is vBulletin bug, since vBulletin have plugin architecture and supports to plugin additional functionality. Archive have its own extension points what means that archive also was predicted to be extended. If adding any parameter to URL breaks whole page it means that there is a bug in vBulletin code and it is vBulletin team work to solve it.



PS. If your still unsatisfied with their response and want me to ask a more technical question post it here and I will copy/paste it to the vBulletin Support Team directly ok?

I do not understand what they mean by writing that it is not intended to subforums to be shown in archive – it is and it is generated by vBulletin... Maybe you do not understand each other. Just give them the URL:

http://YOUR_FORUM/archive/index.php/f-182.html
And tell that it will broke if you add any parameter to URL - this is the bug. You can also give them previous paragraph.


Since vBulletin said there's no bug and everything is working fine the problem is on your end so I'm waiting for the fix.

No it means that they do not know what you mean or they just flush you off. It will not make their bug disappear from their code and jump to our code, just because they wrote that there is no bug. It doesn't work this way. And I wrote you how you can check yourself that it is vBulletin bug. I wrote it to you then, I wrote it now. So just check it and please stop writing that it is on our side because it will not make it appears on our side after repeating it again and again. Did you even checked it? Did you disabled vBET and checked yourself that the link will crash with any parameter added to URL? Did you?

mario06
25-01-11, 08:30
The vBulletin Member is a senior member and I don't believe at all that he made a mistake checking the archive (he accessed the forum as an admin) nor that he brushed me off. "Maybe" we misunderstood each other so I've asked for further assistance from them so lets see what they say again.

Asking for support is considered to be SPAM for you? If that's the case I Apologize for asking for support! You must seriously look at this matter from my point of view, vBulletin says that there's no bug, you say that it's a vBulletin bug so put yourself in my shoes and ask whom must I believe?

PS. I never quoted anything from a PM sent to or from you, I only quoted your quotes.

PS2. Since I'm not a coder telling me to check things makes no sense so I must either trust you or vBulletin at this point.

PS3. Since we live in a civilized world we democratically debated certain issues, things said and different points of view and nothing more. There are no hard feelings on my part. I just want to pacifically resolve the problem :)

Best Regards

vBET
25-01-11, 10:04
The vBulletin Member is a senior member and I don't believe at all that he made a mistake checking the archive (he accessed the forum as an admin) nor that he brushed me off or that we misunderstood each other since English is our mother language. The snapshots speak clearly. You don't want or are unable to fix the issue and I'll accept that.

Asking for support is considered to be SPAM?!? If that's the case I Apologize for asking for support!! You said that there is a bug, Vbulletin said that there isn't!! Who do I have to believe?

PS. I never quoted anything from a PM sent to or from you, I only quoted your quotes.

PS2. If the thread went off topic it isn't my fault, i asked you numerous times to give me a simple fix in a 1. 2. 3. manner BUT I only received this
1. It's his fault
2. It's your fault
3. It's their fault.

Since we live in a civilized world we democratically debated on certain issues, things said and different points of view and nothing more. For me the issue is closed so you can close this thread. No hard feelings on my part. :)

Best Regards

It is not what you believe about - it is about what really happens and you can simply check it yourself - we gave you instructions how to do it. So please trow your believes away, because those leads you in wrong direction and just check it. Did you check it?

Please note that we do not advice to leave the issue just as it is. We strongly recommend to solve it. We gave you other possibilities if vBulletin team do not want to correct their bug. You can use one of those as temporary solution and still contact vBulletin team to solve their bug. If you wish to solve it please tell us which possibility you prefer and we will guide you more in this direction. At this moment, according to your wish - the topic considered solved.

mario06
25-01-11, 10:57
I gave them the information so I'm waiting for a response, I didn't close this issue yet. :)

Regards

vBET
25-01-11, 17:59
I gave them the information so I'm waiting for a response, I didn't close this issue yet. :)

Regards

So I also set here not solved yet. We realy would like to help you make it working :)

tavenger5
26-01-11, 17:50
Why are you using the archive again? That's a massive amount of unnecessary duplicate content.

mario06
29-01-11, 08:43
Why are you using the archive again? That's a massive amount of unnecessary duplicate content.

The archive is recommended to be dissabled IF using vBseo by the vBSEO team, I am not using vBSEO or Zoints so there is no benefit in disabling it for me.
PS. Google separates and pick the better of the 2 from the normal forum threads and the archive forum threads so they don't penalize you for that so that's not the reason. Here's a Google Guide Duplicate Content Guide from Youtube, pay close attention:

http://www.youtube.com/watch?v=6hSoXutuj0g

mario06
29-01-11, 08:50
So I also set here not solved yet. We realy would like to help you make it working :)


Here are the answers from vBulletin Support Team, I have sent them the entire list of urls from the Google email (like I sent you) and your broken link examples.


xxxxxxxxxxx
Jan 25th '11 11:14am


http://mysite.com/archive/index.php/t-61966.html?language=zh-TW

As you can see, going to any of those urls there is no "crash" So I am not sure what they are talking about.

If there is a bug, please provide me with the bug report. But there is no crashing going on.

All the best,
xxxxxxxxxxx
Support Team, vBulletin

Purchase vBulletin, the most powerful forum software available. (http://www.vbulletin.com/)
mailto:support@vbulletin.com



xxxxxxxxxxxx
Jan 26th '11 01:12am

The person you were talking to said there was a bug, and that he was working with us about it. I do not see an issue with your archive urls, they're not crashing the forum. I would like to hear about this bug report, since the archive code has not changed in ages.


All the best,
xxxxxxxxxxxxx
Support Team, vBulletin

Purchase vBulletin, the most powerful forum software available. (http://www.vbulletin.com/)
mailto:support@vbulletin.com

So here you go Michael, they have stated in numerous answers that the archive has no problem, now what's next?

Regards

AfrikaansAlbanianArabicBelarusianBulgarianCatalanChineseCroatianCzechDanishDutchEnglishEstonianFilipinoFinnishFrenchGalicianGermanGreekHaitian CreoleHebrewHindiHungarianIcelandicIndonesianIrishItalianJapaneseKoreanLatvianLithuanianMacedonianMalayMalteseNorwegianPersianPolishPortugueseRomanianRussianSerbianSlovakSlovenianSpanishSwahiliSwedishTaiwaneseThaiTurkishUkrainianVietnameseWelshYiddish
Translated to other languages supported by vB Enterprise Translator 4.10.1