PDA

View Full Version : Already done Circular checking and switching of APIs to keep flow of translations



Simon Lloyd
04-10-11, 01:10
I have my character limit for google set to 100,000 perday so with my settings "Always Use Google", "Use Google API V2", "Use Google Detection" when i reach that limit and no longer get results from paid Google would it be possible for free APIs then start producing results?

So for instance i use my Google preset limit and Google no longer returns a result for me (probably returning an error code like those in your Google test code) when the result is not returned it would be good if vBET automatically recognised the fault code and then sent request to another API like Microsoft (or any others that vBET later supports) this way we are assured of getting some result - for me this would be very very valuable given that there are limits even with paid versions, it would allow you to extend your translations limits.

e.g
Google set 100,000 Charcters per day > used up> vBET move to next API in list > Microsoft 400k per hour or 4M when limit reached vBET check next API and previous to see if limit is lifted or has some allowance > either move to next API or back to Google paid when limit reached again > check next API....etc and so the circular checking after recieving an error code would carry on, so pretty much constant ability to have translations.

vBET
04-10-11, 09:28
I understand your description and your point. Now we have to find out how it suppose to work technically.

One issue I see here is how we will recognize that we already have limits available after those where reached before.

We can simply each time ask preferred provider and then go to the next one. This will cost performance - because for each request to page which requires translation, we will made unsuccessful call to preferred provider, then to next one (so it can be several unsuccessful calls when vBET will support more APIs).

Other solution would be to store information that preferred provider is not available and go directly to next one. This would be much faster, because checking local variable is much faster than waiting for response from external server. This time we have other issue - we do not know when preferred provider is available. We can of course made some scheduled task which would ask for simple (short) translation for example once per hour/day to check it. So in this strategy we have to decide how often by default such task suppose to work. Of course we would check it only when some provider is marked as not available.
Also if we mark providers as unavailable - what to do when we know that all providers are not available - add some information for end user or just translate what is in cache and the rest as original, without any additional info about temporary lack of translation providers.

No matter which way it will be done, Google will be treated as one API (v1 or v2 depending on configuration) - there is no sense to split it, because Google v1 will be closed very soon.

Another thing is to allow to configure providers queue for each language pair separately. At this moment vBET already allows to configure translation provider for each language pair. I think that we can change it from one value to comma separated values (CSV). This way we will know for each language pair which providers support this translation and what are order preferences (just order on CSV list).

PLEASE NOTE: this will have some performance impact anyway. Instead of creating one object for translation we will have to create array of such objects and additional wrapping object (to make it transparent for other parts of code and less bugs prone). Of course we will not create objects for providers we know are not available at this moment.
Solution for this would be to reconfigure for better performance and remove providers queue - just like it is right now - one provider per language pair.
This shouldn't be expensive for performance, but still some additional logic and memory consumption.

Please tell which solution is preferred.

vBET
04-10-11, 18:23
And one more potential solution. If we will mark whole API as not available and check it by scheduled task is it available now, then we do not have to make queue of providers. We can do it this way - always is created only one translator object (better memory usage) and in one request we ask for translation only one provider (better CPU). If it will be not available, then it will be marked as not available and results will be empty (worst reliability). But only first one, because next time we will use another provider from the queue. And in case if no provider is available, then dummy translator will be used - return same values (but do not cache it) so some parts will be not translated but page will not have empty parts like now when provider is not available.

vBET
04-10-11, 22:31
Just quick announcement - we are already implementing this feature.

We want to release it fast (as BETA) because of common issues, caused by limits set by translation providers. We are also looking for other APIs which can be supported by vBET :)

Simon Lloyd
04-10-11, 22:36
My thoughts were to send the check translation first to see if preferred provider is available, so you gave us code to check if google or MS is responding, so at time of call for translation test googleapi (name of my test file with your test code) if translation is true use preffered, if translation is flase or code is not 200 then try next provider in list and perform their api test before using.

You could have a listbox where the user can list each provider one per line in order of preference (this allows when you add support for other APIs the user can just add them to the list), so my list could look like this:
Microsoft
MyTranslator
Google
YourTranslator
AnOtherTranslator

Assuming the daft names i entered were real providers, on call for translation MS test code would run, if response 200 use MS if not run MyTranslator test code, check response for 200 if yes use it if not run Google test code**********etc

This way you never have to store any information on providers (otherwise you could have text boxes where users could enter their set limits for each provider but i think this information wuld be useless as they could change it and it would mean more checking back and checking forward before making a translation) you would never have to worry if limits were available again so no need for a cron job to run to check these, the load on server for that one small translation check (the code you provided in FAQ) would be nothing.

Hopefully i explained that ok so you get my idea, i think it could all be done just by that small check and without storing anything.

Simon Lloyd
04-10-11, 22:37
Just quick announcement - we are already implementing this feature.

We want to release it fast (as BETA) because of common issues, caused by limits set by translation providers. We are also looking for other APIs which can be supported by vBET :)I sent you one or two (in a post you deleted because of the links) that you could approach, if you want a beta volunteer i'm your man :)

vBET
04-10-11, 22:57
I sent you one or two (in a post you deleted because of the links) that you could approach, if you want a beta volunteer i'm your man :)

Your message was softly deleted, because its content was advertisement written by someone else, but we have access to this message and we are on it :)

We even already send question email to one of those translations providers about payment details. Some of those are paid (even when it is described as free it is not on API level - same thing you have with Google you can translate free by browser, but not by API), but prices can be competitive, so it is still good (more competition better prices).
Some we have investigate are those really external translations API or just local dictionaries written by own users (this is also one thing on our TODO list - allow to modify and put own translations) - Radek has this part.

So we are working to improve vBET and made it as cheap in usage as possible :)

vBET
05-10-11, 13:52
We are in last stages of tesing new functionality. You can already see changed description: http://www.vbenterprisetranslator.com/forum/vbet4-troubleshooting/413-faq-2.html#post8914 (see last NOTE)

Simon Lloyd
05-10-11, 18:03
Thanks Michael, i made a quick post in taht Faq which no doubt you will have to remove because its not the correct place for it :) if you would like to test on a a live board that calls many translations PM me and i'll give you access to admincp and forum root, i will also put google translation limit that i have set up and down at your command so you can test :)

vBET
06-10-11, 00:50
OK so. Providers queue is implemented and it will be included in releases 3.5.1 and 4.4.3. vBET 3.5.1 will be released today. vBET4.4.3 is still in test stage. Booth releases will be BETA so everyone can test it in larger forums that test one. Please note that we are already testing 3.5.1 on one of our real forums. Still because of important changes it is in BETA stage first.

Simon Lloyd
06-10-11, 06:59
Does it need to be a scheduled task and one particular provider turned off for an hour at a time?, i made a suggestion here Circular checking and switching of APIs to keep flow of translations where perhaps we could always start at the top of our providers list and make test call (like the one you provided to test Google response and Microsoft response) if test call response is 200 or text is translated then use that provider, if the response is not 200 or test text is not translated (using the same text for each test and the REGEX to check the translated text) then move to next provider, every translation call can then start at the top of the list and work down

Not having an empty result would be good because once we have an empty return thats how it remains, i've already had many people complaining that this is the case in my forum.

vBET
06-10-11, 11:33
It doesn't have to be this way, it is right now. Thanks for your note. Still - no. It has no sense. Please note that asking for external translation is more time consuming thing in whole vBET (and it is not up to us). There is no sense to mad thousands of request when we already reach limits. This would increaser response time, CPU consumption and memory consumption also (more object created).

We discovered that Google information about probably abuse TOS disappears after some time. We don't know but maybe if we will keep asking when we are already blocked, Google can block for more time. Maybe not, but still actual strategy is much better for performance. At the end you have circular checking. If one is not available it is marked as unavailable and another is used. If another is not available then same thing happens. We just do not check is it available again each request what has no sense (it can be millions of queries before it will be available) just once per hour. And if it will be available it will me marked so we will go back to preferred one - and you have a circle here. Also testing each time will made your limits reached faster or costs higher if you use paid translations (it still counts as translation).
Also we are prepared to have other providers too. When we will support more than 2 such strategy will be a killer for your server. Imagine 5 testing calls to different providers and then real translation for each translation request. No. Thanks for your idea :) We really appreciate users ideas, this time we will stay with actual solution.

Please note that you can change how often vBET should check providers availability. Now it is one per hour, but you can reconfigure that in Admin CP -> Scheduled Tasks -> Scheduled Task Manager and set it for example for every 10 minutes 0 just like task RSS Poster Robot do it now.

vBET
06-10-11, 13:34
Little change made - we will check provider availability not once per hour but every 10 minutes. If you already upgraded to vBET 3.5.1 before this message please just download vBET package again and upload product file again.

The change was made because we found on our real forum that often provider is unavailable for short time. We will investigate it more to look for another improvements.

Simon Lloyd
06-10-11, 15:51
Great work guys :), i have upgraded to this but will download the latest fix and use that, i will create a new thread for feedback on this.

AfrikaansAlbanianArabicBelarusianBulgarianCatalanChineseCroatianCzechDanishDutchEnglishEstonianFilipinoFinnishFrenchGalicianGermanGreekHaitian CreoleHebrewHindiHungarianIcelandicIndonesianIrishItalianJapaneseKoreanLatvianLithuanianMacedonianMalayMalteseNorwegianPersianPolishPortugueseRomanianRussianSerbianSlovakSlovenianSpanishSwahiliSwedishTaiwaneseThaiTurkishUkrainianVietnameseWelshYiddish
Translated to other languages supported by vB Enterprise Translator 4.9.2