Keith Devens .com

Unicode testing

I figured it'd be fun to paste in some foreign-language text and see how my site handles it now that I do Unicode

I got the following texts from this helpful Unicode test page. I really wanted to find the Shema somewhere where it wasn't shown as an image, but I wasn't able to.

По оживлённым берегам
Громады стройные теснятся
Дворцов и башен; корабли
Толпой со всех концов земли
К богатым пристаням стремятся;

Ἰοὺ ἰού· τὰ πάντʼ ἂν ἐξήκοι σαφῆ.
Ὦ φῶς, τελευταῖόν σε προσϐλέψαιμι νῦν,
ὅστις πέφασμαι φύς τʼ ἀφʼ ὧν οὐ χρῆν, ξὺν οἷς τʼ
οὐ χρῆν ὁμιλῶν, οὕς τέ μʼ οὐκ ἔδει κτανών.

पशुपतिरपि तान्यहानि कृच्छ्राद्
अगमयदद्रिसुतासमागमोत्कः ।
कमपरमवशं न विप्रकुर्युर्
विभुमपि तं यदमी स्पृशन्ति भावाः ॥

This is supposedly Chinese, but my browser doesn't have the fonts installed so I get a bunch of question marks:

子曰：「學而時習之，不亦說乎？有朋自遠方來，不亦樂乎？
人不知而不慍，不亦君子乎？」

有子曰：「其為人也孝弟，而好犯上者，鮮矣；
不好犯上，而好作亂者，未之有也。君子務本，本立而道生。
孝弟也者，其為仁之本與！」

ஸ்றீனிவாஸ ராமானுஜன் ஐயங்கார்

بِسْمِ ٱللّٰهِ ٱلرَّحْمـَبنِ ٱلرَّحِيمِ

ٱلْحَمْدُ لِلّٰهِ رَبِّ ٱلْعَالَمِينَ

ٱلرَّحْمـَبنِ ٱلرَّحِيمِ

مَـالِكِ يَوْمِ ٱلدِّينِ

إِيَّاكَ نَعْبُدُ وَإِيَّاكَ نَسْتَعِينُ

ٱهْدِنَــــا ٱلصِّرَاطَ ٱلمُسْتَقِيمَ

صِرَاطَ ٱلَّذِينَ أَنعَمْتَ عَلَيهِمْ غَيرِ ٱلمَغضُوبِ عَلَيهِمْ وَلاَ ٱلضَّالِّين

Here's testing some punctiation:

The convention in English is “to use double quotation marks to indicate quotation, and ‘single quotation marks’ for nested quotations.”

En français la convention est « d'utiliser les guillemets français doubles pour les citations, et “ les guillemets anglais doubles ” ou bien ‹ les guillemets français simples › pour les citations imbriquées. »

Auf Deutsch ist die Vereinbarung »umgekehrte zweifache Anführungszeichen für die Zitate zu benutzen, sogar ›einfache Anführungszeichen‹ für die verschachtelte Zitate«; diese Anführungszeichen „dürfen auch solche ‚englische‘ Anführungszeichen sein.“

The en-dash is used between numbers such as in: 1685–1750 (J. S. Bach). It is longer than the hyphen (as in “en-dash”, or, more properly, “en‐dash”) but shorter than the em-dash, which is used — like this — as a sort of parenthesis. Neither should be confused with the horizontal bar which is used to introduce quotation in some cases.
― Like this?
― Right.

The ellipsis is… well, it just is.

It'll be interesting to see if my StructuredText parser dies on all of this.

Other than that, a concern I have is that someone would be able to post invalid Unicode data to my site and have me take it. I wonder if any other environments (than PHP) ensure that text coming into it is in the proper encoding.

Ok, here goes. Will I corrupt MySQL? Will I make PHP crash? Tune in next time, same bat time, same bat channel.

---

Ok then. Everything seems to have worked (but of course I still can't verify the Chinese), except that the Arabic is not right-justified. I wonder what I have to do to do that. The page I got this from had the following to begin the Arabic blockquote: <blockquote xml:lang="ar" lang="ar" dir="rtl">. Language and direction are two things I guess I should shoehorn into my StructuredText parser. Interestingly, that page's encoding was ASCII and they used all entities to include the other languages, so that page didn't actually use Unicode itself at all.