An IDNA problem in right-to-left scripts
draft-alvestrand-idna-bidi-02

Status of this Memo

By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire on July 12, 2008.

Abstract

The use of right-to-left scripts in internationalized domain names has presented several challenges. This memo discusses some problems with these scripts, including one resulting from a constraint on the use of combining characters at the end of an RTL domain label, causing some words to be declared invalid as IDN labels, and proposes a means for ameliorating this problem.

Table of Contents

1.  Introduction and problem description
2.  Detailed examples
    2.1.  Dhivehi
    2.2.  Yiddish
    2.3.  Strings with numbers
3.  An expanded justification for the bidi rule
4.  Modification to RFC 3454
    4.1.  Alternative approach
5.  Other issues in need of resolution
6.  Backwards compatibility considerations
7.  IANA Considerations
8.  Security Considerations
9.  Acknowledgements
Appendix A.  Change log
    A.1.  Changes from -00 to -01
    A.2.  Changes from -01 to -02
10.  References
§  Authors' Addresses
§  Intellectual Property and Copyright Statements

1.  Introduction and problem description

The IDNA specification "Stringprep", [RFC3454] (Hoffman, P. and M. Blanchet, “Preparation of Internationalized Strings ("stringprep"),” December 2002.) makes the following statement in its section 6 on the bidi algorithm, :

3) If a string contains any RandALCat character, a RandALCat character MUST be the first character of the string, and a RandALCat character MUST be the last character of the string.

(A RandAlCat character is a character with unambiguously right-to-left directionality.)

The reasoning behind this prohibition was to ensure that every component of a visually presented domain name has an unambiguously preferred direction. However, this makes certain words in languages written with right-to-left scripts invalid as IDN labels, and in at least one case means that all the words of an entire language are forbidden as IDN labels.

This will be illustrated below with examples taken from the Dhivehi and Yiddish languages, as written with the Thaana and Hebrew scripts, respectively.

The problem may be addressed by more carefully considering the bidi algorithm in Unicode Standard Annex #9 [UAX9] (0, “Unicode Standard Annex #9: The Bidirectional Algorithm, revision 15,” 03 2005.) which states in section 3.3.3 W1: "Examine each non-spacing mark (NSM) in the level run, and change the type of the NSM to the type of the previous character." (See below for some terminology).

Section 3 of UAX9 contains several instructions for determining the directionality of the characters in a string. Some of them (for instance those using explicit embedding) are irrelevant to IDNA because the corresponding codes are not permitted as IDNA input, so a slightly simplified version should be enough for IDNA purposes.

A note on terminology:

In this memo, we use "network order" to describe the sequence of characters as transmitted on the wire or stored in a file; the terms "first", "next" and "previous" are used to refer to the relationship of characters in network order.

We use "display order" to talk about the sequence of characters as imaged on a display medium; the terms "left" and "right" are used to refer to the relationship of characters in display order.

2.  Detailed examples

2.1.  Dhivehi

Dhivehi, the official language of the Maldives, is written with the Thaana script. This displays some of the characteristics of Arabic script, including its directional properties, and the indication of vowels by the diacritical marking of consonantal base characters. This marking is obligatory, and both double vowels and syllable-final consonants are indicated by the marking of special unvoiced characters. Every Dhivehi word therefore ends with a combining mark.

The word for "computer", which is romanized as "konpeetaru", is written with the following sequence of Unicode code points:

U+0786 THAANA LETTER KAAFU (AL)

U+07AE THAANA OBOFILI (NSM)

U+0782 THAANA LETTER NOONU (AL)

U+07B0 THAANA SUKUN (NSM)

U+0795 THAANA LETTER PAVIYANI (AL)

U+07A9 THAANA LETTER EEBEEFILI (AL)

U+0793 THAANA LETTER TAVIYANI (AL)

U+07A6 THAANA ABAFILI (NSM)

U+0783 THAANA LETTER RAA (AL)

U+07AA THANAA UBIUFILI (NSM)

The directionality class of U+07AA in the Unicode database is NSM (non-spacing mark), which is not R or AL; a conformant implementation of the IDNA algorithm will say that "this is not in RandALCat", and refuse to encode the string.

2.2.  Yiddish

Yiddish is one of several languages written with the Hebrew script (others include Hebrew and Ladino). This is basically a consonantal alphabet but Yiddish is written using an extended form that is fully vocalic. The vowels are indicated in several ways, of which one is by repurposing letters that are consonants in Hebrew. Other letters are used both as vowels and consonants, with combining marks used to differentiate between them. Finally, some base characters can indicate several different vowels, which are also disambiguated by combining marks. Marked characters can appear in word-final position and may therefore also be needed at the end of labels. This is not an invariable attribute of all Yiddish strings and there is thus greater latitude here than there is with Dhivehi.

The "YIVO Institute for Jewish Research" is widely known by the acronym of its Yiddish name. This organization maintains a primary reference standard for modern Standard Yiddish orthography, that is also commonly referred to by the same acronym (as the "YIVO Rules"). YIVO is written with the Hebrew letters YOD YOD HIRIQ VAV VAV ALEF QAMATS, where HIRIQ and QAMATS are combining "points":

U+05D9 HEBREW LETTER YOD (R)

U+05B4 HEBREW POINT HIRIQ (NSM)

U+05D5 HEBREW LETTER VAV (R)

U+05D0 HEBREW LETTER ALEF (R)

U+05B8 HEBREW POINT QAMATS (NSM)

The directionality class of U+05B8 HEBREW POINT QAMATS in the Unicode database is NSM, which again causes the IDNA algorithm to reject the string. (It may also be noted that the requisite combined characters also exist in precomposed form at separate positions in the Unicode chart. However, Stringprep also rejects those codepoints, for reasons not discussed here.)

2.3.  Strings with numbers

RFC 3454, in its insistence that the first or last character of a string be category R or AL, prohibited strings that contained right-to-left characters and numbers.

Considering the string ALEF 5 (HEBREW LETTER ALEF + DIGIT FIVE), if we specify that UAX#9 is used to find the directionality of characters, this string will have a consistent direction (R). However, the string 5 ALEF, when embedded in an LTR context, will have the same display order, with a different direction assigned to the number 5. These two display strings are confusable, so we need a rule that permits only one of these in a domain name label.

3.  An expanded justification for the bidi rule

One issue with RFC 3454 was that it did not give an explicit justification for the bidi rule, thus it was hard to tell if a modified rule would continue to fulfil the purpose for which the RFC 3454 rule was written.

This document proposes an explicit justification, for which we think it is possible to test whether or not the modified rule fulfils the justification.

The justification proposed is this:

• No two labels, when presented in visual order, should have the same sequence of characters without also having the same sequence of characters in network order. (This is the criterion that is explicit in RFC 3454).
• In a visual presentation of a string of labels, the characters of each label should remain grouped between the dots delimiting the label components.
• This property should hold true both when the string is embedded in a RTL context and when it's embedded in a LTR context.
• This property should hold true without adding extra formatting, for example bidi control characters, to the string.

Several stronger statements were considered and rejected, because they seem to be impossible to fulfil within the constraints of the Unicode bidirectional algorithm. These include:

• The appearance of a label should be unaffected by its embedding context. This proved impossible even for ASCII labels; the label "123-456" will have a different display order in a RTL context than in a LTR context.
• The sequence of labels should be consistent with network order. This proved impossible - a domain name consisting of the labels (in network order) L1.R1.R2.L2 will be displayed as L1.R2.R1.L2 in an LTR context.

4.  Modification to RFC 3454

If the following modification is made to RFC 3454 section 6, paragraph 4, we believe that the usefulness of the specification for languages written with right-to-left scripts will be significantly improved:

Old text:

[Unicode3.2] defines several bidirectional categories; each character has one bidirectional category assigned to it. For the purposes of the requirements below, an "RandALCat character" is a character that has Unicode bidirectional categories "R" or "AL"; an "LCat character" is a character that has Unicode bidirectional category "L".

New text:

[Unicode3.2] defines several bidirectional categories; each character has one bidirectional category assigned to it.

For characters that have category "R", "AL" or "L", the category is fixed (UAX#9 defines them as having "strong" category); for characters in category EN, ES, ET, AN, CS, NSM, BN, B, S, WS and ON, the category is determined by applying the algorithm described in UAX#9 section 3.3 to the string.

For the purposes of the requirements below, an "RandALCat character" is a character that, after this determination, has Unicode bidirectional categories "R" or "AL"; an "LCat character" is a character that has Unicode bidirectional category "L".

Note that Unicode 5.0 is the current version of Unicode. This fix refers to Unicode 3.2 only, to maintain consistency with the rest of RFC 3454. Nothing here should affect the relationship between Unicode versions and IDNA.

Also, as noted in the introduction, the Unicode UAX#9 algorithm is quite complex. For the purposes of IDNA, a simpler algorithm may be defind that yields the same result within the constraints of this context, but may be easier for people to implement consistently. Such an algorithm may be included in later versions of this memo.

4.1.  Alternative approach

The editors are not entirely happy with the text above. We are considering, instead, a complete replacement for section 6 of RFC 3454.

A first draft of such a section is below.

Conceptually, to verify suitability as a domain name label, one constructs the string consisting of the label preceded and followed by a full stop (U+002E), and executes the Unicode bidirectional algorithm twice, once with <sor> (start of run) and <eor> (end of run) having direction L, and once with them having direction R. (The full stop, being of bidi class CS, is used because it seems likely to show up any problems, and occurs next to labels a lot of the time. Other times, a label is adjacent to an @ sign, a space or another character.)

The following conditions MUST be true in both resulting strings for the string to be acceptable:

• The leftmost and rightmost character of the resulting string in display order must be a full stop (U+002E)
• No non-spacing mark (NSM) can occur in the second position of the string (leftmost in L order, rightmost in R order); that is, no mark can be allowed to attach to the delimiting characters.
• The direction of the leftmost and rightmost characters in the string (the periods) must be either L or R

Note that there is no requirement that the character sequence be the same in the two cases.

All RTL strings permitted by RFC 3454 section 6 will pass this test. Strings that consist of such a string with NSM characters appended to it will also pass this test.

[[NOTE: Not sure if the ALEF 5 vs 5 ALEF issue will be solved by this rule. Test needed.]]

[[NOTE: do we need to require something for the sor=L, eor=R and sor=R, eor=L cases?]]

5.  Other issues in need of resolution

This is not the only issue with right-to-left scripts. Retaining Yiddish for the purposes of further exemplification, its alphabet includes three digraphs that can be encoded both as consecutive instances of the two component characters, and as precomposed ligatures. One of these digraphs also requires additional combined marking. For example, the HEBREW LIGATURE YIDDISH DOUBLE VAV (U+05F0) is orthographically equivalent to, and typographically utterly confusable with, a sequence of two HEBREW LETTER VAV (U+05D5). However, the ligature has no canonical decomposition and is therefore preserved by the IDNA algorithm. These digraphs need to be enumerated and the one form either made invalid for input in the IDNA context, or normalized to the other.

We believe that there is a clear likelihood of similar issues existing with other scripts and languages that are not currently used extensively with IDNs. Careful consideration of all the languages written in a given script, in consultation with all of the corresponding speech communities, is therefore needed before we can say with any degree of certainty that using that script for IDNs is unproblematic.

Another set of issues concerns the proper display of IDNs with a mixture of LTR and RTL labels, or only RTL labels; it is not clear to these authors what the proper display order of the components of a domain name are if the directiion of the components (in network order) is, for instance, FirstRTL.SecondRTL.LTR - is it LTRtsriF.LTRdnoceS.LTR or LTRdnoceS.LTRtsrif.LTR? Again, this memo does not attempt to suggest a solution to this problem.

6.  Backwards compatibility considerations

As with any change to an existing standard, it is important to consider what happens with existing implementations when the change is introduced. The following troublesome cases have been noted:

• Old program used to input the newly allowed string. If the old program checks the input against RFC 3454, the string will not be allowed, and that domain name will remain inaccessible.
• Old program is asked to display the newly allowed string, and checks it against RFC 3454 before displaying. The program will perform some kind of fallback, most likely displaying the Punycode form of the string.
• Old program tries to display the newly allowed string. If the old program has code for displaying the last character of a string that is different from the code used to display the characters in the middle of the string, display may be inconsistent and cause confusion.

One particular example of the last case is if a program chooses to examine the last character (in network order) of a string in order to determine its directionality, rather than its first; if it finds an NSM character and tries to display the string as if it was a left-to-right string, the resulting display may be interesting, but not useful.

The editors believe that these cases will have less harmful impact in practice than continuing to deny the use of words from the languages for which these strings are necessary as IDN labels.

7.  IANA Considerations

This document makes no request of IANA.

Note to RFC Editor: this section may be removed on publication as an RFC.

8.  Security Considerations

This modification will allow some strings to be used in Stringprep contexts that are not allowed today. It is possible that differences in the interpretation of the specification between old and new implementations could pose a security risk, but it is difficult to envision any specific instantiation of this.

Any rational attempt to compute, for instance, a hash over an identifier processed by stringprep would use network order for its computation, and thus be unaffected by the changes proposed here.

While it is not believed to pose a problem, if display routines had been written with specific knowledge of the current Stringprep prohibitions, it is possible that the possible problems noted under "backwards compatibility" could cause new kinds of confusion.

9.  Acknowledgements

While the listed editors held the pen, this document represents the joint work and conclusions of an ad hoc design team. In addition to the editors this consisted of, in alphabetic order, Tina Dam, Patrik Faltstrom, and John Klensin. Many further specific contributions and helpful comments were received from the people listed below, and others who have contributed to the development and use of the IDNA protocols.

The team wishes in particular to thank Roozbeh Pournader for calling its attention to the issue with the Thaana script, and Paul Hoffmann for pointing out the need to be explicit about backwards compatibility considerations.

Appendix A.  Change log

This appendix is intended to be removed when this document is published as an RFC.

A.1.  Changes from -00 to -01

Suggested a possible new algorithm.

Multiple smaller changes.

A.2.  Changes from -01 to -02

Date of publication updated.

Change log added.

10. References

Authors' Addresses

Full Copyright Statement

Copyright © The IETF Trust (2008).

This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.

This document and the information contained herein are provided on an “AS IS” basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.

The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.