Specifying Unicode Character Repertoires in RFCs

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 1 March 2024.¶

Copyright Notice

Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

When a protocol or data format has text fields, that text is normally composed of Unicode [UNICODE] characters, to support use by speakers of all the world's languages. Unfortunately, the Unicode Standard does not define the term "Unicode character" in a way that is useful for technical specifications.¶

Protocols and data formats SHOULD describe exactly which selection of the available Unicode characters are to be used. This document uses the term "character repertoire" to describe such a subset of the Unicode characters. Authors should have a way to concisely and exactly reference a stable specification that identifies a protocol or data format's character repertoire¶

There are several subsets that have been popular choices in code and specification character repertoires. This document describes and names them, and suggests one new one. The goal of this document is to provide a convenient target for cross-reference from other specifications which desire to use one of these character repertoires.¶

2. Character Concepts

The Unicode Standard's definition of "Unicode character" is conceptual. However, each Unicode character is assigned an integer identifier in the range U+0000 through U+10FFFF, and these numbers are used to specify the allowed repertoires of Unicode characters in code and specifications.¶

The numbers assigned to Unicode characters are called “code points”; there are potentially 1,114,112 of them. As of 2023, less than 150,000 characters have had code points assigned. While the inclusion of unassigned code points in text data is undesirable, it is difficult to specify that it should be avoided, because unassigned code points regularly become assigned as new characters are added to Unicode. Fortunately, the occurrence of unassigned code points in texts is generally unlikely to cause software to malfunction.¶

3. Subsets Defined in the Unicode Standard

This section describes popular subsets of the code points that are defined in [UNICODE]. Specifications can refer to these repertoires by the names "Unicode Code Points" and "Unicode Scalar Values".¶

unicode-code-points =
   %x0-10FFFF

unicode-scalar-values =
   %x0-D7FF / %xE000-10FFFF  ; exclude surrogates

4. Other Definitions

This section lists other ways to specify subsets of the code points beyond those provided by the Unicode Standard itself. These subsets may serve as more appropriate character repertoires for some protocols and data formats than those in Section 3, depending on their needs. Specifications can refer to these repertoires by the names "XML Characters" and "Basic Unicode Characters".¶

xml-chars =
   %x9 / %xA / %xD /   ; useful controls
   %x20-D7FF /         ; exclude surrogates
   %xE000-FFFD/        ; exclude FFFE and FFFF nonchars
   %x100000-10FFFF

basic-unichars =
   %x9 / %xA / %xD /             ; useful controls
   %x20-7E /                     ; exclude C1 controls and DEL
   %xA0-D7FF /                   ; exclude surrogates
   %xE000-FDCF                   ; exclude FDD0 nonchars
   %xFDF0-FFFD /                 ; exclude FFFE and FFFF nonchars
   %x1000-1FFFD / %x2000-2FFFD / ; (repeat per plane)
   %x3000-3FFFD / %x4000-4FFFD /
   %x5000-5FFFD / %x6000-6FFFD /
   %x7000-7FFFD / %x8000-8FFFD /
   %x9000-9FFFD / %xA000-AFFFD /
   %xB000-BFFFD / %xC000-CFFFD /
   %xD000-DFFFD / %xE000-EFFFD /
   %xF000-FFFFD / %x10000-10FFFD

5. IANA Considerations

This document makes no requests of IANA.¶

6. Security Considerations

Unicode Security Considerations [TR36] is a wide-ranging survey of the issues implementors should consider while writing software to process Unicode text. Many of the exploits it discusses are aimed at deceiving human readers, but vulnerabilities involving issues such as surrogates and noncharacters are also covered, and in fact can contribute to human-deceiving exploits.¶

Note that the Unicode-character subsets specified in this document include a successively-decreasing number of surrogates and noncharacters, and thus should be less and less susceptible to vulnerabilities. The Section 4.2 subset, "Basic Unicode Characters", excludes all of them.¶

Acknowledgements

Thanks are due to Guillaume Fortin-Debigaré, who filed an Errata Report against RFC8259, The JavaScript Object Notation, noting frequent references to "Unicode characters", when in fact the RFC formally specifies the use of Unicode code points.¶

Authors' Addresses