PDA

View Full Version : Solved Accents not displaying correctly...



freddie3218
07-03-11, 16:09
Hi.

I'm using vbulletin 4.08.

I use vbet for mainly two languages... English and French. I have the forum default language set to English. However, whenever a French user posts, the translation displays correctly in English. However, in French the accents do not display. Instead, it displays like this:


Dans quelle langue vous allez & #233;crire vos messages. Forum traduira automatiquement vos messages & #224; d'autres utilisateurs langues. En outre, il se traduira par des postes dans d'autres langues & #224; votre langue par d& #233;faut.

It should display like this:


Dans quelle langue vous allez écrire vos messages. Forum traduira automatiquement vos messages à d'autres utilisateurs langues.

My forum is set up to display both languages as ISO-8859-1. Therefore, I am not using UTF-8

Also, I followed the installation instructions very closely.

The only modification I made to vBet is that I changed the default flags using CSS-Sprite. However, I looked at the code, and nothing I did should effect the way that things are being displayed.

Do I have to convert my db to UTF-8 to solve this problem?

Any advice will be appreciated. Thanks. :)

freddie3218
08-03-11, 17:56
Well.... I disabled every other plugin, and I still have the same problem. Looks like a bug in vbet

kamilkurczak
08-03-11, 19:35
hello, do you use UTF-8 forum and database or not?
If not - I should have a solution for you. I wil prepare it and write here.

If you can - please paste here a url to the thread with this issue.
Thanks

freddie3218
09-03-11, 00:15
hello, do you use UTF-8 forum and database or not?
If not - I should have a solution for you. I wil prepare it and write here.

If you can - please paste here a url to the thread with this issue.
ThanksNo, I do not use a UTF-8 forum or database. I use ISO-8859-1.

You can see the problem live by viewing this thread:

$ 1 This is a test (http://kokoku.ca/forums/showthread.php?t=418)

Thanks. :)

freddie3218
10-03-11, 01:05
Any news on what is causing it?

kamilkurczak
10-03-11, 10:04
hello,

I am testing it on my test forum (in not UTF) and I can't to reproduce it but you can try this solution.
I have something for you:
1. open, make a copy and edit this file:
forum root/includes/vbenterprisetranslator_functions_utils.php
2. in this function find:

function vbet_encodeToSafeUTF8(&$string) {
//TODO how to check does utf-8 is involved? how often this function is used?
// if (! ereg('[\200-\237]', $string) and ! ereg('[\241-\377]', $string))
// if (!preg_match('/[\200-\237]/', $string) and !preg_match('/[\241-\377]/', $string))
// if (!preg_match('/[\200-\237\241-\377]/e', $string))
// return $string;

// decode three byte unicode characters
$string = preg_replace('/([\340-\357])([\200-\277])([\200-\277])/e',
'\'&#\'.((ord(\'\\1\')-224)*4096 + (ord(\'\\2\')-128)*64 + (ord(\'\\3\')-128)).\';\'',
$string);

//decode two byte unicode characters
$string = preg_replace('/([\300-\337])([\200-\277])/e',
'\'&#\'.((ord(\'\\1\')-192)*64+(ord(\'\\2\')-128)).\';\'',
$string);

return $string;
}
3. and comment all lines without return like this one:

function vbet_encodeToSafeUTF8(&$string) {
//TODO how to check does utf-8 is involved? how often this function is used?
// if (! ereg('[\200-\237]', $string) and ! ereg('[\241-\377]', $string))
// if (!preg_match('/[\200-\237]/', $string) and !preg_match('/[\241-\377]/', $string))
// if (!preg_match('/[\200-\237\241-\377]/e', $string))
// return $string;

// decode three byte unicode characters
// $string = preg_replace('/([\340-\357])([\200-\277])([\200-\277])/e',
// '\'&#\'.((ord(\'\\1\')-224)*4096 + (ord(\'\\2\')-128)*64 + (ord(\'\\3\')-128)).\';\'',
// $string);

//decode two byte unicode characters
//$string = preg_replace('/([\300-\337])([\200-\277])/e',
// '\'&#\'.((ord(\'\\1\')-192)*64+(ord(\'\\2\')-128)).\';\'',
// $string);

return $string;
}
4. save and test!

this should work only for new posts. all existing posts you should edit by hand.
I not shure that you have a ISO forum but you can test it. If not good - please overwrite these changes bu old file

freddie3218
10-03-11, 15:30
Hi,

That was a good try. However, it does not work.

The output now shows: "Ceci est un test. S'il vous plaît ignorer ce message."

The correct output should be: "Ceci est un test. S'il vous plaît ignorer ce message."

Therefore, I reverted back.

Here is one thing I noticed:

When posting and editing, it does not display correctly. However, if I hit the WYSIWYG editor button to go into "basic mode", then it WILL display correctly in the WYSIWYG editor. However, when I try to save it, it reverts to showing incorrectly again. It seems that the WYSIWYG editor is not parsing things correctly before saving when I'm using VBET in a foreign language, or when I'm using the [lang= ] tags. However, if I post a foreign phrase without using the [lang=fr] tags, then it will display correctly when saving. So it seems that VBET is not allowing the WYSIWYG editor to parse correctly before saving... if that makes any sense.

kamilkurczak
10-03-11, 23:41
you have this issue when you would like to post on translated page or not? please describe what i should do to have the same results on my test forum.

freddie3218
11-03-11, 07:19
you have this issue when you would like to post on translated page or not? please describe what i should do to have the same results on my test forum.I don't know if you can reproduce this on your forum or not. However, this is what happens:

Vbulletin version 4.08
I have two languages on my site: English and French. I disabled all other languages, and all other flags.
All English and French is translated throughout the site properly, with the proper accents. The only time the accents do not display properly is if someone is using the French language and posts in French... having the "French language" chosen in their user profile. In this case, VBET will assign the proper BBcode tags automatically when they post. IE: "[lang=fr]" However, once they submit their post, the accents on their post will NOT appear properly. If they press the "edit post" button, they will see their post in the advanced WYSIWYG editor displayed improperly (I have all accounts set up to use the WYSIWYG editor, even in quick reply, by default so that all users can attach pictures from their hard-drive). However, if they toggle the "switch editor mode" button back and forth, they will see their post appear properly. Yet, when they try to save the changes to their post, it will once again display improperly when they save their post.

This problem only happens when the user is using the "FRENCH LANGUAGE" in their user profile settings and posting in the French language... thus making vbet add the [lang=fr] to a post automatically.

My forum default language is English.

If you need to take a look around on my forum... pm me, and I will send you the necessary information to check out my admin panel setup.

I hope this information is helpful.

Thanks.

kamilkurczak
11-03-11, 10:06
yes. I can confirm it on my test forum!
This is a vBET bug so I need to move this thread into bug reporting section.

I noticed that there is a encoding issue with letters with ASCII code under &#256

I can reproduce it on my test forum so I can fix it without access to your forum:)

I will try to find the best solution for you.

freddie3218
12-03-11, 06:02
yes. I can confirm it on my test forum!
This is a vBET bug so I need to move this thread into bug reporting section.

I noticed that there is a encoding issue with letters with ASCII code under Ā

I can reproduce it on my test forum so I can fix it without access to your forum:)

I will try to find the best solution for you.
Well the good news is that I fixed the problem myself. The bad news is that I cannot confirm that this solution will work as a replacement function... as it only works on my forums.

Regardless, here is the new function:



function vbet_encodeToSafeUTF8(&$string) {

$string = utf8_decode($string);
return $string;
}


This function converts UTF-8 to ISO-8859-1

kamilkurczak
12-03-11, 13:31
thanks for this good news. I will test it and confirm

kamilkurczak
13-03-11, 13:15
hello,
Your solution is not good in generally. Why? This is working with letters with ASCII code under 256 (converted and displayed good) but letters with ASCII code over 256 I have "?" signs.

So We need to combine our two solutions in one ;)

example:

function vbet_encodeToSafeUTF8(&$string) {

$string = preg_replace('/([\340-\357])([\200-\277])([\200-\277])/e',
'\'&#\'.((ord(\'\\1\')-224)*4096 + (ord(\'\\2\')-128)*64 + (ord(\'\\3\')-128)).\';\'',
$string);

//decode two byte unicode characters
$string = preg_replace('/([\304-\337])([\200-\277])/e',
'\'&#\'.((ord(\'\\1\')-192)*64+(ord(\'\\2\')-128)).\';\'',
$string);

$string = utf8_decode($string);

return $string;
}

this line:
[\304-\337] define than letters with ASCII code under 256 will be not decoded by my function but only by
utf8_decode

I am testing it still.I am not entirely sure that everything works correctly now.
You can test it too!

freddie3218
14-03-11, 05:10
For my purposes, the solution works fine... as I'm only translating into two langauges... English and French. Both are ISO-8859-1 (latin1) languages. Therefore the "utf_decode" function works fine.

However, with a global scope in view, the utf_decode function will probably not work for other languages outside of "latin1".

I am not much of a vbulletin programmer. However, I will offer one suggestion:

Perhaps using an "If" statement may suit your purposes better if your solution does not work. In the example below, the "translated_language_var" would be the destination language to translate into.

IE:


$latin1 = array(English,French,Spanish,Italian....etc);

if (in_array(translated_language_var,$latin1)){

$string = utf_decode($string);

}else{

original solution;
}

return $string;

Perhaps using something like this would be a sure method to keep everything safe...??? I'm sure you could easily use this pseudo code and make it work for vbulletin. :)

kamilkurczak
27-03-11, 22:30
solved - will be included in the next relase

AfrikaansAlbanianArabicBelarusianBulgarianCatalanChineseCroatianCzechDanishDutchEnglishEstonianFilipinoFinnishFrenchGalicianGermanGreekHaitian CreoleHebrewHindiHungarianIcelandicIndonesianIrishItalianJapaneseKoreanLatvianLithuanianMacedonianMalayMalteseNorwegianPersianPolishPortugueseRomanianRussianSerbianSlovakSlovenianSpanishSwahiliSwedishTaiwaneseThaiTurkishUkrainianVietnameseWelshYiddish
Translations supported by vBET 4.10.1