#206 — Wrong encoding in email when submitting a new issue

State	Resolved
Version:	1.2
Area	Process
Issue type	Bug
Severity	Medium
Submitted by	Martijn Jacobs
Submitted on	Dec 13, 2008
Responsible	Maurits van Rees
Target release:	1.2

Return to tracker

Last modified on Feb 15, 2009 by Maurits van Rees

Sometimes the encoding in the email send when a new issue is added is not correct. The content is encoded in latin-1 instead of the character set used by plone. This is caused by the "Site Setup" -> "Mail Settings" -> "Site 'From' name" property.

Default this is a regular python string, however, when you edit this and press "save", it becomes an unicodestring. (I should report this in the regular plone bugtracker as well).

When this is a unicode string, the dtml template for rendering the email message (poi_email_new_issue) is converting the content to latin-1 (which I cannot explain).

Steps to reproduce:: Add a new issuetracker and add an issue.
Inspect the email send by the issue, you notice the encoding is correct, even with the most strange characters.
Go to "Site Setup" -> "Mail Settings" and edit the "Site 'From' name" property.
Create a new issue in the issuetracker with some non-ascii characters.
Inspect the email send by the issue, you notice the encoding is not correct. When you manually convert the content to unicode and convert it to latin-1 you see it is correct.

Added byMartijn JacobsonDec 13, 2008 05:30 PM

Responsible manager: maurits → (UNASSIGNED)

I commited a fix in the trunk svn for this (revision 77517). Maybe you can take a look at it if you find this solution acceptable?

regards,

Martijn

Added byMaurits van ReesonDec 15, 2008 07:11 PM

Issue state: unconfirmed → open

Responsible manager: (UNASSIGNED) → maurits

Hi Martijn,

What I think is happening here is that in portal_properties/site_properties you have default_charset='utf-8' and in the Properties tab of the Plone Site you have email_charset='iso-8859-1' (or 'latin-1'). The email charset is what is used right before sending out the email. I would say: keep them both to the same character set, preferably utf-8. Maybe there are good reasons to have them separate, but I am not surprised if this causes problems.

That said, email sending from Poi *should* work even with some strange character set settings. But it's hard enough to make it work with all utf-8 so I am not looking for even more ways of making it difficult. ;-)

I slightly changed your fix: I now use getSiteEncoding (so default_charset) instead of email_charset to encode the fromName. The idea is that the dtml page gets input from the rest of the site as well, which will be a string encoded by the default_charset (or perhaps unicode). So we want the fromName to have the same encoding. Encoding the end result with the email_charset is done later.

Oh, and your fix contained some tabs instead of spaces; please only use spaces.

Someone had an idea to change the current dtml email page to a template, like is currently done in events.py, which sounds like a good idea. Wait a minute: that is Neo|4D in irc, and I see his name is Martijn. Is that you? :-)

There are likely more spots that fail when the default_charset and the email_charset differ. I'll keep it in mind and open this issue.

Added byMartijn JacobsonDec 15, 2008 08:50 PM

Hi Maurits.

Unfortunately the charsets ('default_charset' and 'email_charset') are the same : both utf-8. It really has something to do with the 'email_from_name' property : When set in the "Mail Settings" panel in plone it becomes an unicode string and the dtml is then jumping into 'unicode mode' or something. Changing back the 'email_from_name' property in the ZMI converts it back to a regular python string, and the problem goes away.

Anywho, after rewriting it so the dtml isn't used anymore (like in events.py) the problem will go away anyway. I can rewrite that part if you like, however I would like to have a new egg released of Poi soon so maybe that can happen in the next version?

Sorry for the tabs, I normally don't use tabs so it was a onetime mistake.

BTW, neo|4d on irc is me indeed :)

Added byMaurits van ReesonFeb 15, 2009 02:29 PM

Issue state: Confirmed → Resolved

Target release: None → 1.2

I still did not manage to reproduce this, but it should be fixed with r80866, where I indeed use templates in python instead of dtml.

No responses can be added.