Then I found this excellent explanation of using UTF8 with PHP, which is well worth a read.

Encoding gives me a headache but from this explanation this is how I see it.

I had some character that the parser does not know how to interput because it was outside the byte range of the UTF8 format. Some of the PHP functions, like iconv, still let some non-UTF8 characters through which breaks the parser. The preg_replace just rips out any non-UTF8 character based on it’s byte sequence and replaces it with a question mark.

From that article above, I use the following code to remove any non-UTF8 characters.

//reject overly long 2 byte sequences, as well as characters above U+10000 and replace with ?
$some_string = preg_replace('/[\x00-\x08\x10\x0B\x0C\x0E-\x19\x7F]'.
 '|[\x00-\x7F][\x80-\xBF]+'.
 '|([\xC0\xC1]|[\xF0-\xFF])[\x80-\xBF]*'.
 '|[\xC2-\xDF]((?![\x80-\xBF])|[\x80-\xBF]{2,})'.
 '|[\xE0-\xEF](([\x80-\xBF](?![\x80-\xBF]))|(?![\x80-\xBF]{2})|[\x80-\xBF]{3,})/S',
 '?', $some_string );

//reject overly long 3 byte sequences and UTF-16 surrogates and replace with ?
$some_string = preg_replace('/\xE0[\x80-\x9F][\x80-\xBF]'.
 '|\xED[\xA0-\xBF][\x80-\xBF]/S','?', $some_string );

For people unfamiliar with how widgets work, I’ll briefly explain. We give you an embed code (javascript) that you will place in your site HTML (where you want to display the widget). When this page loads, the javascript is enabled and calls a URL from our site to retrieve HTML. The code then places this HTML into a div container in your sites HTML using the DOM and innerHTML. Hey presto, the widget is rendered into your site.

The encoding problem occurred due to a site using another character encoding type other than UTF8. The HTML we send is encoded in UTF8 and the first time the poll widget loaded, it renders and looks fine. But it appeared that after a vote, so another request for HTML, the HTML seemed to adopt the sites encoding, causing the text in the poll to look a mess.

This bugged the hell out of us and like all the best solutions it was a simple one. You simply need to add a charset attribute to the script tag that references or encloses your javascript code.

So something like…

<script type="text/javascript" language="javascript" charset="utf-8" src="http://static.polldaddy.com/p/2064343.js"></script>

Here’s a decent description of the problem.