PDA

View Full Version : Solved indexed still going down for this week



badawidollah
10-04-10, 02:33
Sitemap stats
Total URLs: 1,026,021
Indexed URLs: 25,172

slowly going down
before this it is about 59,xxx (VBSEO ONLY)
now its going down step by step

my index at google down too
Results 1 - 10 of about 92,800 from nusantaraku.org. (0.11 seconds)

before about 236,000

badawidollah
10-04-10, 05:02
Sitemap stats
Total URLs: 1,026,021
Indexed URLs: 24,608

badawidollah
10-04-10, 16:13
Sitemap stats
Total URLs: 1,026,021
Indexed URLs: 23,215
Submit a Sitemap

down again

badawidollah
10-04-10, 16:57
Sitemap stats
Total URLs: 1,026,021
Indexed URLs: 22,552

still going down

badawidollah
10-04-10, 17:36
Sitemap stats
Total URLs: 940,000
Indexed URLs: 22,434

badawidollah
10-04-10, 17:41
what happen with my forum?

vBET
10-04-10, 20:29
In our opinion there is no way that Google react that fast on your sitemap changes. I understand that you committed sitemap today several times and notices that indexed pages are going low, but this is just coincidence. If they are going down they would do the same if you didn't made any submission at all. Google simply doesn't react that fast with re-indexing.

What could be a reason here - did you made any changes in your robots.txt recently? This is first think which goes to my mind - did you block some irrelevant content as we suggest for optimization? If yes then that is a reason - blocked pages are just removed from index just like you wanted it.

Other reasons - please find our what was changed on your forum recently. And by recently I do not mean today, but in last 2 months.

badawidollah
11-04-10, 01:43
Total URLs: 1,031,393
Indexed URLs: 21,838
Submit a Sitemap

i didnt do block irrelevant content.
this is the sitemap on


Include Show Post Pages
Include Archive Pages
Include Show Thread Pages
Include Forum Display Pages
Include Tag Pages


i`ll off tag pages

badawidollah
11-04-10, 01:44
i change some subforum too.
maybe thats the problem too

badawidollah
11-04-10, 01:54
User-agent: *
Disallow: /admincp/
Disallow: /cgi-bin/
Disallow: /clientscript/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /sitemap/
Disallow: /tags/

User-agent: *
Disallow: /printthread.php
Disallow: /subscription.php
Disallow: /payments.php
Disallow: /profile.php
Disallow: /faq.php
Disallow: /calendar.php
Disallow: /private.php
Disallow: /sendmessage.php
Disallow: /sendmessage.php?do=
Disallow: /showgroups.php
Disallow: /reputation.php
Disallow: /report.php
Disallow: /threadrate.php
Disallow: /postings.php
Disallow: /newthread.php
Disallow: /search.php
Disallow: /newreply.php
Disallow: /register.php
Disallow: /login.php
Disallow: /faq.php
Disallow: /image.php
Disallow: /cron.php
Disallow: /joinrequests.php
Disallow: /usercp.php
Disallow: /member.php

i put in robots.txt a minute ago

badawidollah
11-04-10, 08:59
Sitemap stats
Total URLs: 1,030,240
Indexed URLs: 21,709

vBET
11-04-10, 14:55
i change some subforum too.
maybe thats the problem too

This can be a reason because old links do not exist anymore. Please note that what you see in webmaster tools is NOT how many pages you have indexed. It only tells you how many URLs from your sitemap is in index. As you know you have indexed much more pages (some are not in sitemap). So if URL changes in sitemap (like after subforum change) then it is not indexed yet under this URL (it can still be under old one) and webmaster tool will show you less indexed URLs. Because it does NOT shows all indexed URLs, but how many URLs from sitemap is indexed. You can see all by making advanced query in Google.

If you have 301 redirect from old links to new ones then those shouldn't be removed from index - just re-indexed. Of course until it will be re-indexed webmaster tool will show you less URLs.

Please note that such Google changes can be also not related at all with your actions - it can be caused by Google server farms replications, algorithm changes and Google only knows what else. Of course it is always good to keep hand on pulse and check does it is not related to our actions so you could react on such cases.

vBET
11-04-10, 15:07
i put in robots.txt a minute ago

Please note that this will remove blocked content from index. Because it is not relevant content it will be good for your SEO (trash out of index = more valuable content in index).

vBET
11-04-10, 15:08
Sitemap stats
Total URLs: 1,030,240
Indexed URLs: 21,709

Does Google shows you errors in sitemap?

badawidollah
11-04-10, 15:38
Does Google shows you errors in sitemap?

yes
they said redirect pages

i got this too

Results 1 - 10 of about 87,100 from Please Choose Your Languages Bel (http://www.nusantaraku.org). (0.34 seconds)

before i got 236,000
and still going down too

badawidollah
11-04-10, 15:40
i hope this is will ok in 2-3 month.
i changed about 50% subforum name.

vBET
11-04-10, 22:42
yes
they said redirect pages

i got this too

Results 1 - 10 of about 87,100 from Please Choose Your Languages Bel (http://www.nusantaraku.org). (0.34 seconds)

before i got 236,000
and still going down too

You can be calm about redirects in sitemap. Google advices to put there only final URLs, but still know how to handle redirects and still indexes final URL (those after redirect). At this moment we do not put translated URLs into sitemap only add language code, because of performance reasons. Translation of each URL in each language in one moment during sitemap generation would be too expensive. Anyway we will think about some solution in this area.

vBET
11-04-10, 22:43
i hope this is will ok in 2-3 month.
i changed about 50% subforum name.

So this definitively can be the reason. Still it shouldn't if you really have 301 redirect from old to new one, but what really happens on Google side is mystery.

Still remember that you added some new content to robots.txt - so those blocked URLs will and SHOULD disappear from index.

badawidollah
12-04-10, 04:07
URLs not followed
When we tested a sample of URLs from your Sitemap, we found that some URLs redirect to other locations. We recommend that your Sitemap contain URLs that point to the final destination (the redirect target) instead of redirecting to another URL.
HTTP Error: 301
URL: Go to URLhttp://www.nusantaraku.org/forum/en/85735-post1.html
Problem detected on: Apr 11, 2010

vBET
12-04-10, 12:49
I see 2 issues there:
1. Your link looks strange - are you sure that you made integration with Sitemap Generator appropriately?
2. I see that you have there double redirect (301 redirect twice - one after another). Please integrate vBSEO appropriately - see here for details about vBSEO integration changes: http://www.vbenterprisetranslator.com/forum/vbet-announcements/537-vbet-4-2-1-released-great-performance-improvments-beta.html

badawidollah
13-04-10, 13:16
RewriteEngine On

RewriteBase /forum/

RewriteCond %{HTTP_HOST} !^www\.nusantaraku\.org
RewriteRule (.*) http://www.nusantaraku.org/forum/$1 [L,R=301]

RewriteRule ^((urllist|sitemap_).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

RewriteRule ^/?(af|sq|ar|be|bg|ca|zh-CN|hr|cs|da|nl|en|et|tl|fi|fr|gl|de|el|iw|hi|hu|is|id|ga|it|ja|ko|lv|lt|mk|ms|mt|no|fa|pl|pt|ro|ru|sr|sk|sl|es|sw|sv|zh-TW|th|tr|uk|vi|cy|yi)/$ vbenterprisetranslator_seo.php?vbet_lang=$1&redirected=/forum/index.php [L,QSA]
RewriteRule ^/?(af|sq|ar|be|bg|ca|zh-CN|hr|cs|da|nl|en|et|tl|fi|fr|gl|de|el|iw|hi|hu|is|id|ga|it|ja|ko|lv|lt|mk|ms|mt|no|fa|pl|pt|ro|ru|sr|sk|sl|es|sw|sv|zh-TW|th|tr|uk|vi|cy|yi)/(.*)?$ vbenterprisetranslator_seo.php?vbet_lang=$1&redirected=/forum/$2 [L,QSA]

RewriteCond %{REQUEST_URI} !(admincp/|modcp/|vbseo_sitemap/|cron)
RewriteRule ^((archive/)?(.*\.php(/.*)?))$ vbenterprisetranslator_seo.php [L,QSA]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !^(admincp|modcp|clientscript|cpstyles|images)/
RewriteRule $ vbenterprisetranslator_seo.php [L,QSA]

RewriteCond %{REQUEST_URI} !(admincp/|modcp/|cron|vbseo_sitemap)
RewriteRule ^((archive/)?(.*\.php(/.*)?))$ vbseo.php [L,QSA]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !/(admincp|modcp|clientscript|cpstyles|images)/
RewriteRule ^(.+)$ vbseo.php [L,QSA]


is it correct?

and for url in sitemap
which one must choose,
/en/ or ?language=en

i am using default one now, but adding this below <?php
because cant run without this code


require_once(DIR. '/includes/vbenterprisetranslator_functions_vbseo.php');
define('DIR', dirname(__FILE__).'/../');


Sitemap stats
Total URLs: 331,647
Indexed URLs: 9,047

thanks

vBET
14-04-10, 01:01
is it correct?

and for url in sitemap
which one must choose,

i am using default one now, but adding this below <?php
because cant run without this code





thanks

Your rules looks OK. Please check readme.html in vBET package and see what changes in vBSEO files you have to do. I see that link which you gave still have double redirect (you can see it i.e. in Opera).

About your change in Sitemap Generator file - use one for /en/
Also require_once -where are you adding it - in which file? I can see that it is wrong, because DIR is defined after it is used - you have it in wrong file. Please see here for Sitemap Generator integration:
http://www.vbenterprisetranslator.com/forum/vbet4-integration-other-plugins/415-sitemap-generator-integration.html

Please fallow precisely integration instructions for vBSEO and Sitemap Generator. If you are not sure which steps you done and which not - just start from original files and do the changes again but precisely.
If you feel like it is trouble for you we can offer you this integration as Upgrade Service (http://www.vbenterprisetranslator.com/upgrade-service.php) ($30).

Also if all links which you have in your sitemap are like this one which you gave, this can be the reason of your issues - wrong Sitemap integration. Please note that the link which you gave is broken - there is no title included. Does original links (without language code) are also broken? Please PM me your full sitemap content.

badawidollah
14-04-10, 18:14
vbseo_sitemap_functions.php modified.


<?php
require_once(DIR. '/includes/vbenterprisetranslator_functions_vbseo.php');



vbseo_log_entry("[showthread] forum_id: $forumid, thread_id: $threadrow[threadid], total posts: $totalposts, pages: $totalpages, views: $threadrow[views] $prior");
//additional code
$alanguages = array ('ar','zh-CN','en','ms','ru','th');
if(!$archived) {
$strlenhost = strlen($vbseo_vars['bburl']."/");
$currentthreadurl = vbseo_url_thread($threadrow, 1, $archived);
$currentthreadurl_before = substr($currentthreadurl,0,$strlenhost);
$currentthreadurl_after = "/".substr($currentthreadurl,$strlenhost);
$lastpost =& $threadrow['lastpost'];
$vbseo_stat['t'] += sizeof($alanguages);
foreach($alanguages as $language) {
vbseo_add_url($currentthreadurl_before.$language.$currentthreadurl_after, $prior, $lastpost, $freq);
}
}
//end additional code

for($p=1; $p<=$totalpages; $p++)

my default is "id"
thanks

** i am using vbseo sitemap generator 2.6 beta 1
i am using sample /en/address
my users down 15%

vBET
14-04-10, 22:57
This looks OK right now. Also please PM me your actual sitemap result files - as I wrote link which you gave before looks broken (no title included) I would like to see does it still happens and if yes then does for original links also. In such case I will check it on your side and fix it.

NOTE: you still have double redirect - please integrate vBSEO appropriately. You unnecessarily loose performance.

badawidollah
16-04-10, 13:41
URLs not followed
When we tested a sample of URLs from your Sitemap, we found that some URLs redirect to other locations. We recommend that your Sitemap contain URLs that point to the final destination (the redirect target) instead of redirecting to another URL.
HTTP Error: 301
URL: Go to URLhttp://www.nusantaraku.org/forum/ru/obat-obatan-dan-farmasi/116222-pintar-membaca-obat.html
Problem detected on: Apr 16, 2010

** now i remove vbet from sitemap.
my google index still going down

Results 1 - 10 of about 79,200 from nusantaraku.org. (0.10 seconds)

vBET
16-04-10, 15:52
I see that you do not have double redirections anymore - it will save you performance. I also see that your links for translated pages in sitemap are not broken anymore.

As I wrote at the beginning we do not believe that your sitemap content has anything to do with your pages disappearing from index. 1st - Google just doesn't react so fast so it is not reaction on your sitemap; 2nd - sitemap has nothing to do with pages removed from index, sitemap simply shows what you would like to have indexed and does nothing more.
So removing translated URLs from sitemap will not solve the issue. It will make Google warnings about 301 redirect disappear. Please note that Google will know and index your translated pages anyway. Google will find links to it directly on your forum, but indexing it can take more time that if they are in sitemap.

Please keep in mind that you added new entries to robots.txt and blocked some pages. So those blocked pages will and should be removed from Google index. In Google webmaster tools you can even check exactly which URLs are blocked by robots.txt (please note that those data are not up do date - those are updated periodically).

Also please think what other changes you did on your forum. You wrote that you changed names of over 50% of forums. It shouldn't be reason if you have 301 redirect - still it is only shouldn't. Please think did you also changed some privileges on forum? If you blocked access to some forums for guests, then definitively this can be reason. Please remember that Google see your forum as guest. If you blocked some content then on pages which was previously indexed Google see same communicate about not enough privileges. And in such case remove those form index as duplication of content (all blocked pages are the same). Please remember that I'm writing here about blocking by vB privileges not by robots.txt which can also be the reason.

One more time - think what other changes was made on your forum in last 2 months.

Also - are you able to determine which of your pages disappear from index? Maybe disappears only those which you blocked by robots.txt and we have no issue at all. If you are able to write some robot which will collect your actual links displayed by Google and compare differences between one harvesting and another, then you could see what is missing (I do not know does Google gives some ready solution).

And there is one more thing - please keep in mind that in searching results Google shows only approximated number of results. And Google is changing his algorithms and cache all the time. So please keep in mind that numbers shown in searching results are not real and those can change (giving better or worst approximation) even if you have still same pages indexed. Of course in your case pages are really disappearing, because you blocked some in robots.txt

badawidollah
18-05-10, 17:58
index going up and back to normal.
no more redirection

AfrikaansAlbanianArabicBelarusianBulgarianCatalanChineseCroatianCzechDanishDutchEnglishEstonianFilipinoFinnishFrenchGalicianGermanGreekHaitian CreoleHebrewHindiHungarianIcelandicIndonesianIrishItalianJapaneseKoreanLatvianLithuanianMacedonianMalayMalteseNorwegianPersianPolishPortugueseRomanianRussianSerbianSlovakSlovenianSpanishSwahiliSwedishTaiwaneseThaiTurkishUkrainianVietnameseWelshYiddish
Translations made by vBET Translator 4.10.1