#58 — UnicodeDecodeError 'ascii' codec can't decode byte 0xc3 in position 23: ordinal not in range(128)

by (anonymous) — last modified Jul 09, 2009 03:35 PM
State Resolved
Version:
Area Functionality
Issue type Bug
Severity Medium
Submitted by (anonymous)
Submitted on Aug 27, 2006
Responsible
Target release:


I have a bilingual site in English and Spanish. At each hit to the web site I get this warning:

2006-08-27T19:14:04 WARNING PlacelessTranslationService Decoding problem in: Switch language to ${language} 'ascii' codec can't decode byte 0xc3 in position 4: ordinal not in range(128)

and in all the pages, the title of the link to switch to Spanish is

"Switch language to ${language} (Content translation not available)"

See http://pupeno.com/publications/software-release-cycle for an example. Now, the problem gets even worse when you click the link to see the Spanish version. Instead of just getting the content in English or a message stating that the content is only in English you get (translations in parens):

Error en el sitio (Error in the site)
Este sitio encontró un error intentando completar su petición (This site encountered an error trying to complete your petition). Los errores fueron (the errors were):

Tipo de Error (Error Type)
 UnicodeDecodeError

Valor del Error (Error Value)
 'ascii' codec can't decode byte 0xc3 in position 23: ordinal not in range(128)

Petición hecha en (Query on date)
 2006/08/27 19:16:58.687 Universal

and whet I check the logs I can find this backtrace:

2006-08-27T19:16:58 ERROR Zope.SiteErrorLog http://pupeno.com/[…]/not_available_lang
Traceback (innermost last):
  Module ZPublisher.Publish, line 115, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 41, in call_object
  Module Shared.DC.Scripts.Bindings, line 311, in __call__
  Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec
  Module Products.CMFCore.FSPageTemplate, line 195, in _exec
  Module Products.CMFCore.FSPageTemplate, line 134, in pt_render
  Module Products.PageTemplates.PageTemplate, line 104, in pt_render
   - <FSPageTemplate at /pupeno.com/not_available_lang used for /pupeno.com/publications/software-release-cycle/software-release-cycle>
  Module TAL.TALInterpreter, line 238, in __call__
  Module TAL.TALInterpreter, line 281, in interpret
  Module TAL.TALInterpreter, line 749, in do_useMacro
  Module TAL.TALInterpreter, line 281, in interpret
  Module TAL.TALInterpreter, line 457, in do_optTag_tal
  Module TAL.TALInterpreter, line 442, in do_optTag
  Module TAL.TALInterpreter, line 437, in no_tag
  Module TAL.TALInterpreter, line 281, in interpret
  Module TAL.TALInterpreter, line 780, in do_defineSlot
  Module TAL.TALInterpreter, line 281, in interpret
  Module TAL.TALInterpreter, line 457, in do_optTag_tal
  Module TAL.TALInterpreter, line 442, in do_optTag
  Module TAL.TALInterpreter, line 437, in no_tag
  Module TAL.TALInterpreter, line 281, in interpret
  Module TAL.TALInterpreter, line 728, in do_defineMacro
  Module TAL.TALInterpreter, line 281, in interpret
  Module TAL.TALInterpreter, line 772, in do_defineSlot
  Module TAL.TALInterpreter, line 281, in interpret
  Module TAL.TALInterpreter, line 457, in do_optTag_tal
  Module TAL.TALInterpreter, line 442, in do_optTag
  Module TAL.TALInterpreter, line 437, in no_tag
  Module TAL.TALInterpreter, line 281, in interpret
  Module TAL.TALInterpreter, line 749, in do_useMacro
  Module TAL.TALInterpreter, line 281, in interpret
  Module TAL.TALInterpreter, line 715, in do_condition
  Module TAL.TALInterpreter, line 281, in interpret
  Module TAL.TALInterpreter, line 457, in do_optTag_tal
  Module TAL.TALInterpreter, line 442, in do_optTag
  Module TAL.TALInterpreter, line 437, in no_tag
  Module TAL.TALInterpreter, line 281, in interpret
  Module TAL.TALInterpreter, line 749, in do_useMacro
  Module TAL.TALInterpreter, line 281, in interpret
  Module TAL.TALInterpreter, line 507, in do_setLocal_tal
  Module Products.PageTemplates.TALES, line 221, in evaluate
   - URL: file:LinguaPlone/skins/LinguaPlone/global_languageselector.pt
   - Line 35, Column 6
   - Expression: standard:'context/languageSelectorData'
   - Names:
      {'container': <PloneSite at /pupeno.com>,
       'context': <ATDocument at /pupeno.com/publications/software-release-cycle/software-release-cycle>,
       'default': <Products.PageTemplates.TALES.Default instance at 0x4cf929ec>,
       'here': <ATDocument at /pupeno.com/publications/software-release-cycle/software-release-cycle>,
       'loop': <Products.PageTemplates.TALES.SafeMapping object at 0x50c3d46c>,
       'modules': <Products.PageTemplates.ZRPythonExpr._SecureModuleImporter instance at 0x4cfbc60c>,
       'nothing': None,
       'options': {'args': ()},
       'repeat': <Products.PageTemplates.TALES.SafeMapping object at 0x50c3d46c>,
       'request': <HTTPRequest, URL=http://pupeno.com/[…]/not_available_lang>,
       'root': <Application at >,
       'template': <FSPageTemplate at /pupeno.com/not_available_lang used for /pupeno.com/publications/software-release-cycle/software-release-cycle>,
       'traverse_subpath': ['view'],
       'user': <SpecialUser 'Anonymous User'>}
  Module Products.PageTemplates.Expressions, line 185, in __call__
  Module Products.PageTemplates.Expressions, line 180, in _eval
  Module Products.PageTemplates.Expressions, line 77, in render
  Module Products.PageTemplates.ZRPythonExpr, line 76, in call_with_ns
  Module Products.CMFCore.FSPythonScript, line 103, in __render_with_namespace__
  Module Shared.DC.Scripts.Bindings, line 325, in __render_with_namespace__
  Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec
  Module Products.CMFCore.FSPythonScript, line 164, in _exec
  Module None, line 27, in languageSelectorData
   - <FSPythonScript at /pupeno.com/languageSelectorData used for /pupeno.com/publications/software-release-cycle/software-release-cycle>
   - Line 27
  Module AccessControl.ZopeGuards, line 482, in protected_inplacevar
  Module AccessControl.ZopeGuards, line 401, in __iadd__
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 23: ordinal not in range(128)

I am running:

 Plone 2.5,
 Zope (Zope 2.9.4-final, python 2.4.3, linux2),
 Python 2.4.3 (#1, Aug 6 2006, 23:42:05) [GCC 3.3.5 (Debian 1:3.3.5-13)],
 PIL 1.1.5

LinguaPlone 0.9.0 - $Revision: 10093 $

If there's a workaround, like a simple patch to a template or source code, I'd like to know, since this bug can be pretty bad: every user with its browser set to Spanish that arrives at my site is very likely to encounter that error since most of the content is not translated.

Thank you.
Steps to reproduce:
1 - Install LinguaPlone 0.9.0 on Plone 2.5 on Zope 2.9.4 or something along the lines.
2 - Enable support for Spanish and English. It is important to choose Spanish because it is known to contain an UTF-8 character that is not valid ASCII.
3 - Write content in English.
4 - Try to see it in spanish.
Added by Jarn on Sep 07, 2006 04:54 PM
Issue state: unconfirmedopen
Target release: 1.0None
I think this might be solved with newer LP and/or PLT (perhaps from SVN).
Added by (anonymous) on Sep 18, 2006 03:57 PM
This is just a 'me too' response on slightly different software. Error on switching from site-base language (English) to any other.

Plone 2.5
Zope 2.9.3-, python 2.4.1, linux2)
Python 2.4.1 (#2, May 5 2005, 11:32:06) [GCC 3.3.5 (Debian 1:3.3.5-12)]

Updated PloneLanguageTool to 1.4 - no difference.

- Mark Barratt
Added by (anonymous) on Sep 18, 2006 04:30 PM
Revision 10176 in the SVN to languageSelectorData.py in LinguaPlone/Skins/LinguaPlone fixes this for me.

- Mark Barratt
Added by Xavier Masson on Oct 10, 2006 02:02 PM
I have the exact same problem only on a french site with dual langage French English.

Is there people that have not problem with linguaplone on site with such langage as french or spanish with theses characters ?

if some people do not have the probleme and some have it could also comme from the site configuration.

Beetwen the setup. by encoding, the os encoding, the site locale and linguaplone I have a hard time figuring who determines what and how all this interact.

Somebody as pointers on good docs on this ?
Added by Xavier Masson on Oct 10, 2006 02:03 PM
I have the exact same problem only on a french site with dual langage French English.

Is there people that have not problem with linguaplone on site with such langage as french or spanish with theses characters ?

if some people do not have the probleme and some have it could also comme from the site configuration.

Beetwen the setup. by encoding, the os encoding, the site locale and linguaplone I have a hard time figuring who determines what and how all this interact.

Somebody as pointers on good docs on this ?

for information using the latest linguaplone languageSelectorData.py (11112) did not do anything for my problem ...
I will try to get the whole product and do another test but I suspect it will not change anything
Added by Xavier Masson on Oct 10, 2006 04:08 PM
Problem solved with the last trunk from the SVN reposytory

and this may help me become less stupid http://www.amk.ca/python/howto/unicode
Added by Ruediger Koch on Jan 24, 2007 12:38 PM
I'm relative new to plone. I made a setup for 3 languages (german,english,french).
Translation works for german to english and english to german. Even french to english/german works. But when
I start translation to french the unicode error appears at once.
I have the last windows binaries and as mentioned the last subversion for LinguaPlone/PloneLanguageTool.
Nothing changes. Any idea?

Thanks
Added by Goffinet Francois on Feb 01, 2007 03:03 PM
I know the same problem with french translation. After an update of all concerned products, I folowed this discussion : http://comments.gmane.org/[…]/63918 and I create a $PYTHON//site-packages/sitecustomize.py filethat modify the python's default codec in utf8 :

import sys
sys.setdefaultencoding('utf8')

I restarted my instance and I do not have any UnicodeDecodeError.
Added by Ruediger Koch on Feb 02, 2007 10:53 AM
it works, - super - thanks
Added by Hanno Schlichting on Mar 23, 2007 07:24 AM
Issue state: openresolved
Target release: None1.0
All of the mentioned problems are fixed in the latest (unreleased) versions, so I think we can close this report.
Added by Thorsten Strusch on May 16, 2007 07:16 AM
Thanks Goffinet Francois!
The file sitecustomize.py also solved my problem on a german site ( Zope 2.9.6, Python 2.4.4, Plone 2.5.2 - without linguaplone, build with the UnifiedInstaller under debian/etch).
Added by Robert Herzog on Nov 06, 2007 08:29 PM
Hi,
the "'ascii' codec can't decode..." problem came back with a brand new 3.0.2 Plone installation and Linguaplone2.0 on Ubuntu Linux 7.10, when using a quadrilanguage experimental site (English, Dutch, French and German).
I cured it (apparently until now) by inserting a sitecustomize.py file (in the site-packages directory of the Python instance inside the Plone directory) with the two lines as mentioned in a preceding comment dated Feb. 2007.

import sys
sys.setdefaultencoding('utf8')

Robert Herzog
Belgium
Added by Robert Herzog on Nov 06, 2007 08:31 PM
Hi,
the "'ascii' codec can't decode..." problem came back with a brand new 3.0.2 Plone installation and Linguaplone2.0 on Ubuntu Linux 7.10, when using a quadrilanguage experimental site (English, Dutch, French and German).
I cured it (apparently until now) by inserting a sitecustomize.py file (in the site-packages directory of the Python instance inside the Plone directory) with the two lines as mentioned in a preceding comment dated Feb. 2007.

import sys
sys.setdefaultencoding('utf8')

Robert Herzog
Belgium
Added by Robert Herzog on Nov 06, 2007 08:32 PM
Hi,
the "'ascii' codec can't decode..." problem came back with a brand new 3.0.2 Plone installation and Linguaplone2.0 on Ubuntu Linux 7.10, when using a quadrilanguage experimental site (English, Dutch, French and German).
I cured it (apparently until now) by inserting a sitecustomize.py file (in the site-packages directory of the Python instance inside the Plone directory) with the two lines as mentioned in a preceding comment dated Feb. 2007.

import sys
sys.setdefaultencoding('utf8')

Robert Herzog
Belgium

No responses can be added.