PDA

View Full Version : Solved More load problems



Snake
16-03-10, 11:54
Ok so i have done loads of testing.

Over a period of 24hrs my load rises steadily into the 30.00's

A server restart fixes it for another 24hrs.

If i disable the mod i do not get this problem.

Do not say that with mod disabled there is less traffic as that is not true, google still sends the same traffic with mod disabled according to my stats.

Please explain, load is driving me mad.

tavenger5
16-03-10, 19:32
Sounds like bots are hitting the translated pages when the mod is on. You should look into optimizing apache or getting a server with more cpu power. Are you running vboptimise or any type of caching mechanism, like memcached?

Snake
16-03-10, 21:35
The fact is my forum gets 30,000 uniques a day, if i disable the mod load drops instantly, and bots and users are still show pages so are still demanding server power, it is simply that showing translated pages use 10x the resources than standard vb pages from the normal database. It's poorly written code, and needs to be fixed! The other mods never did this, just vbet, wish i never changed over, but going back is too late now. :mad:

moman
16-03-10, 22:21
It really sounds like you're running on an underpowered server. I'm on a 8-core nehalem machine (so we're looking at 8 more virtual cores due to the HT, for a total of 16). I've also extensively optimized this machine using my own techniques as well as pointers from the folks at vbulletin.com.

vBET raises my load from about 2.5 to 3.0-3.5, depending on number of users, and this obviously grows based on the cache size. However, I don't think that's too bad at all, as my simultaneous users range from a fairly high 800 to an even higher 1,200.

vBET
17-03-10, 00:38
Hi :) Of course translation MUST take some resources - there is no magic. Translate page means take output result and change it. Because of that translation will always take longer than normal page.

I understand that you are not too happy having more server load, but please note that vBET is taking resources ONLY FOR TRANSLATION. For normal pages it adds only flags. So all this additional load comes from additional traffic to translated pages. As you wrote your traffic didn't go down instantly after disabling vBET (if you have it a while then it will decrease after disabling - trust me) and server load is lower - it is obvious - robots are still crawling URLs to translated pages, users are still finding in Google links to your translated pages. So you still have same traffic, but now under translated links is simply duplicated content - normal page which is not translated. If you want to stay with disabled vBET we strongly recommend to add rule in your .htaccess file which will redirect all translated pages to normal one, otherwise you can loose your SEO because of duplicated content.

Please note that we have already planned supporting of other cache systems and our translation algorithms are instantly optimized. I.e. we just discovered how drastically decreases PHP performance when working on large strings and we modified our algorithm. It is already released in vBET 4.2.0 with additional configuration options. And we will move all improvements also to vBET 3.x which is still supported :)

I understand that in your opinion our code is weak. I do not know on what you are basing your expectations. We have fastest translation mod for vB - there is nothing what works better. Translation will take some resources and our mod takes it less that any other. You can see how fast vBET can work on many forums. If you have issues on your server, then please consider configuration changes or adding server resources. You will not put 20 liters of water into 10 liters bucket.

Our devise is: "We have lot to change". And that is why we are experimenting, changing algorithms, profiling and spend lot of time looking for solutions which will require less resources. Still we know no mod which could be any competition to vBET and there are some other translation mods. We made lot of algorithm changes which we had to trow away because they didn't help, during this process we also discovered many improvements. You can have your impression basing on your server issues, but please consider do you have any better solution? What could you give a hint that maybe vBET is not wrong solution since is working on thousands of forums, maybe you are just trying to put 20 liters of water into 10 liters bucket. Still - we have lot to change and great TODO list in optimization section (about 70% to experiment will it help or not) :) And you are 100% right - we can do it better, we will and we ARE doing it all the time :) Just wait till we move all improvements which we made during vBET4.x implementation :)

If I can give you some hints - please check how you can optimize vBET: http://www.vbenterprisetranslator.com/forum/general-discussions/243-vbet-performance.html
Especially consider disabling some languages and blocking irrelevant pages by robots.txt

What is time of response for translated pages? What is your CPU usage? What is your memory usage? If it is acceptable then you have nothing to worry about. People often are scared by increasing server load and even don't know what does it means. 10 times more server load does NOT means 10 time more resources used. It just means that more threads are waiting in queue, what is perfectly normal, since right now your threads have to wait for Google response if some translation is not cached yet. So thread is waiting for Google response and it takes NO CPU AT ALL during this time. Because of that your server load WILL be bigger even if vBET could take no resources at all (what of course is not possible).

About crashing your server - it is obviously your server issue. It happens periodically. I had similar issue on my server. It was caused by some Apache bug so one Apache thread was growing and growing with memory usage until whole memory was consumed ans server cached. Only one thread behaves like that - other Apache threads was normal. I played with Apache configuration and issue is solved. I think that Apache just had some memory leakage - as I remember I put lower value of requests which can be hold by one thread. There were also other changes. I suggest to check your memory usage and monitor it for some time. Also it can be helpful to check average amount of memory used by one Apache thread, made some calculations and set appropriate value of max threads for Apache.

If you have any more questions please just ask :)

vBET
18-03-10, 09:21
Hey - I just was on your forum it translates ULTRA FAST... So what you are claiming about and why so angry attitude about vBET, when you have super fast translation service?...

Please consider what server load means. Understanding its meaning can be very helpful to understand what happens on server and how it can be related to threads which are waiting in queue not because you have no resources, but are waiting for response from other server (Google in this case).

In my opinion you have super fast translations and you have nothing to worry about :)

Snake
18-03-10, 12:49
I have been building sites 10 years, i am fully aware 10x load does not mean 10x resources, stop treating me like a idiot and spoon feeding me rubbish. The cold facts are with this mod vs the other free mod your load is mega. And at peak times my site is now slow and unresponsive. Yes, pages translate fast off-peak, but at a cost of a slow server later in the day. I have a quad core, raid 15k SAS server, that is optimized just fine, it run 0.50 all day long before this with large traffic. It is the VBET code that adds load and makes the server slow in peak times, it's a FACT, it is not increases in traffic, i have had the same traffic and bots before and the server coped fine, it is the translation product. Period. Hurry up and fix it, i really dont want to pay another £300 a month for a server upgrade just to run a MOD lol.

tavenger5
18-03-10, 16:03
How many languages do you have enabled? How many posts do you have? Do you have vbseo and sitemap generator installed? How many bots are hitting the site a day?

Snake
19-03-10, 00:11
Hi!

32 languages enabled.

100,000 posts.

vbseo and sitemap installed.

sitemap plugin says 1000,000 pages crawled a day.


The fact is, if i disable the mod and restart the server, bot and users are still hitting my server the same as before, the same pages are in still in googles index and thus i get the same traffic enabled or not.

The only difference is that with the mod disabled the bots and google searchers get the english page, so, traffic is identical, the only difference is they are not seeing a translated page, thus avoiding the mods database issues.

It's clear as day to me this mod has been released without proper testing, and clearly the owner is not interested in addressing the issues.

Fail.

Snake
19-03-10, 00:19
Michael, you are wrong, this load is not caused by an increase in traffic, it is caused by reading and writing translations to mysql.

If vbulletin team can read and write to mysql with low load why cant you?

Poor code my friend.

vBET
19-03-10, 02:43
I have been building sites 10 years, i am fully aware 10x load does not mean 10x resources, stop treating me like a idiot and spoon feeding me rubbish. The cold facts are with this mod vs the other free mod your load is mega. And at peak times my site is now slow and unresponsive. Yes, pages translate fast off-peak, but at a cost of a slow server later in the day. I have a quad core, raid 15k SAS server, that is optimized just fine, it run 0.50 all day long before this with large traffic. It is the VBET code that adds load and makes the server slow in peak times, it's a FACT, it is not increases in traffic, i have had the same traffic and bots before and the server coped fine, it is the translation product. Period. Hurry up and fix it, i really dont want to pay another £300 a month for a server upgrade just to run a MOD lol.

I understand that you was already familiar with informations about server load which I gave you. Please note that I have no knowledge about how advanced is each of thousands our users and each time I will give ass complete information as it can be helpful. It doesn't means that I'm treating you as idiot - it means that I care about giving you informations which can be useful for you and evaluation of your server condition. Please point me the other free mod which you are talking about I will gladly make some comparison :) Also any time you are free to choose best solution for you.

I checked your forum again and again it seems to respond very fast. Please give me best time frame to being able to observer what you are writing about your slow responses at peak time.

If you want to check how your traffic changed because of vBET - please generate some report which will show you all the traffic to translated pages - this is what you earned thanks to vBET.

You are absolutely right that translations needs additional resources - there is no other way and you will never find product which will translate your site without any cost. As I already mentioned most time requires waiting for Google translation when it is not cached, and during this time your threads have to wait for response what have biggest impact on server load. You can set bigger cache time to live - then often required translations will be already in cache. But for not cached translations any product will have to wait for translation. There is no other way.

Please note that thinks have to be broken to be able to fix it.

As I already wrote you, we constantly improve vBET performance. And I already wrote you, that we have ready performance improvements which are now on beta stage in vBET4.x. Today we will release new vBET4.x version with additional performance improvements. And when bugs for those (if any) will be corrected we will move those improvements to vBET3.x It is not necessary to push.

Also no one forces you to paid another £300 a month for a server - you are making your own decisions and you have lot of options here. Including decreasing number of supported languages, or even switching to other product which you mentioned is much better. We completely understand that solutions which you are using must fit to your needs and possibilities. We are glad to give our clients better and better product. And we are aware that in situation where request must wait for response form another server your server load will be bigger no matter what solutions we will use. We will be glad if you stay with our product and configure it to fit yo your possibilities. And we will gladly give you a hand in this area :)

Please note that we just gave new solution to integrate with Sitemap Generator. If you are integrated - please see new integration instructions here: http://www.vbenterprisetranslator.com/forum/vbet-integration-other-plugins/98-sitemap-generator-integration.html
It dramatically increases speed of sitemap generation (on our forum over 12 times).

vBET
19-03-10, 02:55
It's clear as day to me this mod has been released without proper testing, and clearly the owner is not interested in addressing the issues.

Fail.

If you have doubts about proper testing I propose to check history of vBET - it was tested by hundreds of real forums before it advanced to paid version :)

About addressing the issue. I'm sorry. I wrongly assumed that giving you in first response this informations, I was clear that we are addressing the issue:

Please note that we have already planned supporting of other cache systems and our translation algorithms are instantly optimized. I.e. we just discovered how drastically decreases PHP performance when working on large strings and we modified our algorithm. It is already released in vBET 4.2.0 with additional configuration options. And we will move all improvements also to vBET 3.x which is still supported :)

...

Our devise is: "We have lot to change". And that is why we are experimenting, changing algorithms, profiling and spend lot of time looking for solutions which will require less resources. Still we know no mod which could be any competition to vBET and there are some other translation mods. We made lot of algorithm changes which we had to trow away because they didn't help, during this process we also discovered many improvements. You can have your impression basing on your server issues, but please consider do you have any better solution? What could you give a hint that maybe vBET is not wrong solution since is working on thousands of forums, maybe you are just trying to put 20 liters of water into 10 liters bucket. Still - we have lot to change and great TODO list in optimization section (about 70% to experiment will it help or not) :) And you are 100% right - we can do it better, we will and we ARE doing it all the time :) Just wait till we move all improvements which we made during vBET4.x implementation :)

If I can give you some hints - please check how you can optimize vBET: http://www.vbenterprisetranslator.com/forum/general-discussions/243-vbet-performance.html
Especially consider disabling some languages and blocking irrelevant pages by robots.txt

I feel fully responsible for this misunderstanding. Once again very sorry. Please tell us, which way we have to tell you that we are improving vBET all the time, and how we can assure you again that vBET3.x will have another performance improvements, to make you clear that we are addressing the issue? We will always gladly improve our way of communication with client :)

vBET
19-03-10, 03:02
Michael, you are wrong, this load is not caused by an increase in traffic, it is caused by reading and writing translations to mysql.

If vbulletin team can read and write to mysql with low load why cant you?

Poor code my friend.

You was already noted that we have planned supporting of cache systems (file and existing engines). Please consider what are your intentions in this discussion and does it is going in right direction to improve tour server state - if it is still the point.

Your question assumes that load is caused by communication with mysql. Can you please give us source of this diagnose? We will happily study it :)

sarangan
22-04-10, 09:31
I had the same problem, sever overload. Finally I have disabled VBET from my forum and everything is normal now. :)

vBET
22-04-10, 14:52
I had the same problem, sever overload. Finally I have disabled VBET from my forum and everything is normal now. :)

Which version you used? Please update to last version - we made great performance improvements. Many users wrote their thanks for that, seeing big difference - especially in server load area :)

EDITED:
I just checked your forum and vBET is working there - please do not write false statements about vBET condition.

AfrikaansAlbanianArabicBelarusianBulgarianCatalanChineseCroatianCzechDanishDutchEnglishEstonianFilipinoFinnishFrenchGalicianGermanGreekHaitian CreoleHebrewHindiHungarianIcelandicIndonesianIrishItalianJapaneseKoreanLatvianLithuanianMacedonianMalayMalteseNorwegianPersianPolishPortugueseRomanianRussianSerbianSlovakSlovenianSpanishSwahiliSwedishTaiwaneseThaiTurkishUkrainianVietnameseWelshYiddish
Translated to other languages thanks to vB Enterprise Translator 4.9.99