GeezerVisonThe World Wide Web, restored to 800 pixels of glory. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Home|About|Gaming cabinet|Archives|How it works|Starter sites | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
GeezerVison is showing encoding.spec.whatwg.org — Encoding Standard 1. PrefaceThe UTF-8 encoding is the most appropriate encoding for interchange of Unicode, the universal coded character set. Therefore, for new protocols and formats, as well as existing formats deployed in new contexts, this specification requires (and defines) the UTF-8 encoding. The other (legacy) encodings have been defined to some extent in the past. However, user agents have not always implemented them in the same way, have not always used the same labels, and often differ in dealing with undefined and former proprietary areas of encodings. This specification addresses those gaps so that new user agents do not have to reverse engineer encoding implementations and existing user agents can converge. In particular, this specification defines all those encodings, their algorithms to go from bytes to scalar values and back, and their canonical names and identifying labels. This specification also defines an API to expose part of the encoding algorithms to JavaScript. User agents have also significantly deviated from the labels listed in the IANA Character Sets registry. To stop spreading legacy encodings further, this specification is exhaustive about the aforementioned details and therefore has no need for the registry. In particular, this specification does not provide a mechanism for extending any aspect of encodings. 2. Security backgroundThere is a set of encoding security issues when the producer and consumer do not agree on the encoding in use, or on the way a given encoding is to be implemented. For instance, an attack was reported in 2011 where a Shift_JIS leading byte 0x82 was used to “mask” a 0x22 trailing byte in a JSON resource of which an attacker could control some field. The producer did not see the problem even though this is an illegal byte combination. The consumer decoded it as a single U+FFFD (�) and therefore changed the overall interpretation as U+0022 (") is an important delimiter. Decoders of encodings that use multiple bytes for scalar values now require that in case of an illegal byte combination, a scalar value in the range U+0000 to U+007F, inclusive, cannot be “masked”. For the aforementioned sequence the output would be U+FFFD U+0022. (As an unfortunate exception to this, the gb18030 decoder will “mask” up to one such byte at end-of-queue.) This is a larger issue for encodings that map anything that is an ASCII byte to something that is not an ASCII code point, when there is no leading byte present. These are “ASCII-incompatible” encodings and other than ISO-2022-JP and UTF-16BE/LE, which are unfortunately required due to deployed content, they are not supported. (Investigation is ongoing whether more labels of other such encodings can be mapped to the replacement encoding, rather than the unknown encoding fallback.) An example attack is injecting carefully crafted content into a resource and then encouraging the user to override the encoding, resulting in, e.g., script execution. Encoders used by URLs found in HTML and HTML’s form feature can also result in slight information loss when an encoding is used that cannot represent all scalar values. E.g., when a resource uses the windows-1252 encoding a server will not be able to distinguish between an end user entering “💩” and “💩” into a form. The problems outlined here go away when exclusively using UTF-8, which is one of the many reasons that is now the mandatory encoding for all things. See also the Browser UI chapter. 3. TerminologyThis specification depends on the Infra Standard. [INFRA] Hexadecimal numbers are prefixed with "0x". In equations, all numbers are integers, addition is represented by "+", subtraction by "−", multiplication by "×", integer division by "/" (returns the quotient), modulo by "%" (returns the remainder of an integer division), logical left shifts by "<<", logical right shifts by ">>", bitwise AND by "&", and bitwise OR by "|". For logical right shifts operands must have at least twenty-one bits precision. An I/O queue is a type of list with items of a particular type (i.e., bytes or scalar values). End-of-queue is a special item that can be present in I/O queues of any type and it signifies that there are no more items in the queue. There are two ways to use an I/O queue: in immediate mode, to represent I/O data stored in memory, and in streaming mode, to represent data coming in from the network. Immediate queues have end-of-queue as their last item, whereas streaming queues need not have it, and so their read operation might block. It is expected that streaming I/O queues will be created empty, and that new items will be pushed to it as data comes in from the network. When the underlying network stream closes, an end-of-queue item is to be pushed into the queue. Since reading from a streaming I/O queue might block, streaming I/O queues are not to be used from an event loop. They are to be used in parallel instead. To read an item from an I/O queue ioQueue:
To read a number number of items from ioQueue:
To peek a number number of items from an I/O queue ioQueue:
To push an item item to an I/O queue ioQueue:
To push a sequence of items to an I/O queue ioQueue is to push each item in the sequence to ioQueue, in the given order. To restore an item other than end-of-queue to an I/O queue, perform the list prepend operation. To restore a list of items excluding end-of-queue to an I/O queue, insert those items, in the given order, before the first item in the queue. Inserting the bytes « 0xF0, 0x9F » in an I/O queue « 0x92 0xA9, end-of-queue », results in an I/O queue « 0xF0, 0x9F, 0x92 0xA9, end-of-queue ». The next item to be read would be 0xF0. To convert an I/O queue ioQueue into a list, string, or byte sequence, return the result of reading an indefinite number of items from ioQueue. To convert a list, string, or byte sequence input into an I/O queue:
The Infra standard is expected to define some infrastructure around type conversions. See whatwg/infra issue #319. [INFRA] I/O queues are defined as lists, not queues, because they feature a restore operation. However, this restore operation is an internal detail of the algorithms in this specification, and is not to be used by other standards. Implementations are free to find alternative ways to implement such algorithms, as detailed in Implementation considerations. To obtain a scalar value from surrogates, given a leading surrogate leading and a trailing surrogate trailing, return 0x10000 + ((leading − 0xD800) << 10) + (trailing − 0xDC00). To create a
4. EncodingsAn encoding defines a mapping from a scalar value sequence to a byte sequence (and vice versa). Each encoding has a name, and one or more labels. This specification defines three encodings with the same names as encoding schemes defined in the Unicode standard: UTF-8, UTF-16LE, and UTF-16BE. The encodings differ from the encoding schemes by byte order mark (also known as BOM) handling not being part of the encodings themselves and instead being part of wrapper algorithms in this specification, whereas byte order mark handling is part of the definition of the encoding schemes in the Unicode Standard. UTF-8 used together with the UTF-8 decode algorithm matches the encoding scheme of the same name. This specification does not provide wrapper algorithms that would combine with UTF-16LE and UTF-16BE to match the similarly-named encoding schemes. [UNICODE] 4.1. Encoders and decodersEach encoding has an associated decoder and most of them have an associated encoder. Instances of decoders and encoders have a handler algorithm and might also have state. A handler algorithm takes an input I/O queue and an item, and returns finished, one or more items, error optionally with a code point, or continue. The replacement and UTF-16BE/LE encodings have no encoder. An error mode as used below is " An XML processor would set error mode to " " To process a queue given an encoding’s decoder or encoder instance encoderDecoder, I/O queue input, I/O queue output, and error mode mode:
To process an item given an item item, encoding’s encoder or decoder instance encoderDecoder, I/O queue input, I/O queue output, and error mode mode:
4.2. Names and labelsThe table below lists all encodings and their labels user agents must support. User agents must not support any other encodings or labels. For each encoding, ASCII-lowercasing its name yields one of its labels. Authors must use the UTF-8 encoding and must use its
(ASCII case-insensitive) " New protocols and formats, as well as existing formats deployed in new contexts, must use the
UTF-8 encoding exclusively. If these protocols and formats need to expose the
encoding’s name or label, they must expose it
as " To get an encoding from a string label:
This is a more basic and restrictive algorithm of mapping labels to encodings than section 1.4 of Unicode Technical Standard #22 prescribes, as that is necessary to be compatible with deployed content.
All encodings and their labels are also available as non-normative encodings.json resource. The set of supported encodings is primarily based on the intersection of the sets supported by major browser engines when the development of this standard started, while removing encodings that were rarely used legitimately but that could be used in attacks. The inclusion of some encodings is questionable in the light of anecdotal evidence of the level of use by existing Web content. That is, while they have been broadly supported by browsers, it is unclear if they are broadly used by Web content. However, an effort has not been made to eagerly remove single-byte encodings that were broadly supported by browsers or are part of the ISO 8859 series. In particular, the necessity of the inclusion of IBM866, macintosh, x-mac-cyrillic, ISO-8859-3, ISO-8859-10, ISO-8859-14, and ISO-8859-16 is doubtful for the purpose of supporting existing content, but there are no plans to remove these. The windows-1252 encoding has various labels, such as
" Software that does not follow this standard does not always give the same answers. The root of this is that the original document that specified Latin1 (ISO/IEC 8859-1) did not provide any mappings for bytes in the inclusive ranges 0x00 to 0x1F or 0x7F to 0x9F. Similarly, the original documents that specified ASCII (ISO/IEC 646, among others) did not provide any mappings for bytes in the inclusive range 0x80 to 0xFF. This means different software has chosen different code point mappings for those bytes when asked to use Latin1 or ASCII encodings. Web browsers and browser-compatible software have chosen to map those bytes according to windows-1252, which is a superset of both, and this choice was codified in this standard. Other software throws errors, or uses isomorphic decoding, or other mappings. [ISO8859-1] [ISO646] As such, implementers and developers need to be careful whenever they are using libraries which expose APIs in terms of "Latin1" or "ASCII". It’s very possible such libraries will not give answers in line with this standard, if they have chosen other behaviors for the bytes which were left undefined in the original specifications. 4.3. Output encodingsTo get an output encoding from an encoding encoding:
The get an output encoding algorithm is useful for URL parsing and HTML form submission, which both need exactly this. 5. IndexesMost legacy encodings make use of an index. An index is an ordered list of entries, each entry consisting of a pointer and a corresponding code point. Within an index pointers are unique and code points can be duplicated. An efficient implementation likely has two indexes per encoding. One optimized for its decoder and one for its encoder. To find the pointers and their corresponding code points in an index, let lines be the result of splitting the resource’s contents on U+000A LF. Then remove each item in lines that is the empty string or starts with U+0023 (#). Then the pointers and their corresponding code points are found by splitting each item in lines on U+0009 TAB. The first subitem is the pointer (as a decimal number) and the second is the corresponding code point (as a hexadecimal number). Other subitems are not relevant. To signify changes an index includes an Identifier and a Date. If an Identifier has changed, so has the index. The index code point for pointer in index is the code point corresponding to pointer in index, or null if pointer is not in index. The index pointer for codePoint in index is the first pointer corresponding to codePoint in index, or null if codePoint is not in index. There is a non-normative visualization for each index other than index gb18030 ranges and index ISO-2022-JP katakana. index jis0208 also has an alternative Shift_JIS visualization. Additionally, there is visualization of the Basic Multilingual Plane coverage of each index other than index gb18030 ranges and index ISO-2022-JP katakana. The legend for the visualizations is:
These are the indexes defined by this specification, excluding index single-byte, which have their own table:
The index gb18030 ranges code point for pointer is the return value of these steps:
The index gb18030 ranges pointer for codePoint is the return value of these steps:
The index Shift_JIS pointer for codePoint is the return value of these steps:
The index Big5 pointer for codePoint is the return value of these steps:
All indexes are also available as a non-normative indexes.json resource. (Index gb18030 ranges has a slightly different format here, to be able to represent ranges.) 6. Hooks for standardsThe algorithms defined below (UTF-8 decode, UTF-8 decode without BOM, UTF-8 decode without BOM or fail, and UTF-8 encode) are intended for usage by other standards. For decoding, UTF-8 decode is to be used by new formats. For identifiers or byte sequences within a format or protocol, use UTF-8 decode without BOM or UTF-8 decode without BOM or fail. For encoding, UTF-8 encode is to be used. Standards are to ensure that the input I/O queues they pass to UTF-8 encode (as well as the legacy encode) are effectively I/O queues of scalar values, i.e., they contain no surrogates. These hooks (as well as decode and encode) will block until the input I/O queue has been consumed in its entirety. In order to use the output tokens as they are pushed into the stream, callers are to invoke the hooks with an empty output I/O queue and read from it in parallel. Note that some care is needed when using UTF-8 decode without BOM or fail, as any error found during decoding will prevent the end-of-queue item from ever being pushed into the output I/O queue. To UTF-8 decode an I/O queue of bytes ioQueue given an optional I/O queue of scalar values output (default « »):
To UTF-8 decode without BOM an I/O queue of bytes ioQueue given an optional I/O queue of scalar values output (default « »):
To UTF-8 decode without BOM or fail an I/O queue of bytes ioQueue given an optional I/O queue of scalar values output (default « »):
To UTF-8 encode an I/O queue of scalar values ioQueue given an optional I/O queue of bytes output (default « »), return the result of encoding ioQueue with encoding UTF-8 and output. 6.1. Legacy hooks for standardsStandards are strongly discouraged from using decode, BOM sniff, and encode, except as needed for compatibility. Standards needing these legacy hooks will most likely also need to use get an encoding (to turn a label into an encoding) and get an output encoding (to turn an encoding into another encoding that is suitable to pass into encode). For the extremely niche case of URL percent-encoding, custom encoder error handling is needed. The get an encoder and encode or fail algorithms are to be used for that. Other algorithms are not to be used directly. To decode an I/O queue of bytes ioQueue given a fallback encoding encoding and an optional I/O queue of scalar values output (default « »):
To BOM sniff an I/O queue of bytes ioQueue:
This hook is a workaround for the fact that decode has no way to communicate back to the caller that it has found a byte order mark and is therefore not using the provided encoding. The hook is to be invoked before decode, and it will return an encoding corresponding to the byte order mark found, or null otherwise. To encode an I/O queue of scalar values ioQueue given an encoding encoding and an optional I/O queue of bytes output (default « »):
This is a legacy hook for HTML forms. Layering UTF-8 encode on top is safe as it never triggers errors. [HTML] To get an encoder from an encoding encoding:
To encode or fail an I/O queue of scalar values ioQueue given an encoder instance encoder and an I/O queue of bytes output:
This is a legacy hook for URL percent-encoding. The caller will have to keep an encoder instance alive as the ISO-2022-JP encoder can be in two different states when returning an error. That also means that if the caller emits bytes to encode the error in some way, these have to be in the range 0x00 to 0x7F, inclusive, excluding 0x0E, 0x0F, 0x1B, 0x5C, and 0x7E. [URL] In particular, if upon returning an error the ISO-2022-JP encoder is in the
Roman state, the caller cannot output 0x5C (\) as it will not
decode as U+005C (\). For this reason, applications using encode or fail for unintended
purposes ought to take care to prevent the use of the ISO-2022-JP encoder in combination
with replacement schemes, such as those of JavaScript and CSS, that use U+005C (\) as part of the
replacement syntax (e.g., The return value is either the number representing the code point that could not be encoded or null, if there was no error. When it returns non-null the caller will have to invoke it again, supplying the same encoder instance and a new output I/O queue. 7. APIThis section uses terminology from Web IDL. Browser user agents must support this API. JavaScript implementations should support this API. Other user agents or programming languages are encouraged to use an API suitable to their needs, which might not be this one. [WEBIDL] The following example uses the
The following example decodes an
7.1. Interface mixin
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Code point | Bytes |
|---|---|
| U+E78D | 0xA6 0xD9 |
| U+E78E | 0xA6 0xDA |
| U+E78F | 0xA6 0xDB |
| U+E790 | 0xA6 0xDC |
| U+E791 | 0xA6 0xDD |
| U+E792 | 0xA6 0xDE |
| U+E793 | 0xA6 0xDF |
| U+E794 | 0xA6 0xEC |
| U+E795 | 0xA6 0xED |
| U+E796 | 0xA6 0xF3 |
| U+E81E | 0xFE 0x59 |
| U+E826 | 0xFE 0x61 |
| U+E82B | 0xFE 0x66 |
| U+E82C | 0xFE 0x67 |
| U+E832 | 0xFE 0x6D |
| U+E843 | 0xFE 0x7E |
| U+E854 | 0xFE 0x90 |
| U+E864 | 0xFE 0xA0 |
This asymmetric encoder table preserves compatibility with the GB18030-2005 standard. See also the explanation at index gb18030 ranges.
Let pointer be the index pointer for codePoint in index gb18030.
If pointer is non-null:
Let leading be pointer / 190 + 0x81.
Let trailing be pointer % 190.
Let offset be 0x40 if trailing is less than 0x3F, otherwise 0x41.
Return two bytes whose values are leading and trailing + offset.
Set pointer to the index gb18030 ranges pointer for codePoint.
Let byte1 be pointer / (10 × 126 × 10).
Set pointer to pointer % (10 × 126 × 10).
Let byte2 be pointer / (10 × 126).
Set pointer to pointer % (10 × 126).
Let byte3 be pointer / 10.
Let byte4 be pointer % 10.
Return four bytes whose values are byte1 + 0x81, byte2 + 0x30, byte3 + 0x81, byte4 + 0x30.
Big5’s decoder has an associated Big5 leading, which is a byte, initially 0x00.
Big5’s decoder’s handler, given ioQueue and byte, runs these steps:
If byte is end-of-queue and Big5 leading is not 0x00, then set Big5 leading to 0x00 and return error.
If byte is end-of-queue and Big5 leading is 0x00, then return finished.
If Big5 leading is not 0x00:
Let leading be Big5 leading.
Set Big5 leading to 0x00.
Let pointer be null.
Let offset be 0x40 if byte is less than 0x7F; otherwise 0x62.
If byte is in the range 0x40 to 0x7E, inclusive, or 0xA1 to 0xFE, inclusive, then set pointer to (leading − 0x81) × 157 + (byte − offset).
If there is a row in the table below whose first column is pointer, then return the two code points listed in its second column (the third column is irrelevant):
| Pointer | Code points | Notes |
|---|---|---|
| 1133 | U+00CA U+0304 | Ê̄ (LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND MACRON) |
| 1135 | U+00CA U+030C | Ê̌ (LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND CARON) |
| 1164 | U+00EA U+0304 | ê̄ (LATIN SMALL LETTER E WITH CIRCUMFLEX AND MACRON) |
| 1166 | U+00EA U+030C | ê̌ (LATIN SMALL LETTER E WITH CIRCUMFLEX AND CARON) |
Since indexes are limited to single code points this table is used for these pointers.
Let codePoint be null if pointer is null; otherwise the index code point for pointer in index Big5.
If codePoint is non-null, then return a code point whose value is codePoint.
If byte is an ASCII byte, restore byte to ioQueue.
Return error.
If byte is an ASCII byte, then return a code point whose value is byte.
If byte is in the range 0x81 to 0xFE, inclusive, then set Big5 leading to byte and return continue.
Return error.
Big5’s encoder’s handler, given unused and codePoint, runs these steps:
If codePoint is end-of-queue, then return finished.
If codePoint is an ASCII code point, then return a byte whose value is codePoint.
Let pointer be the index Big5 pointer for codePoint.
If pointer is null, then return error with codePoint.
Let leading be pointer / 157 + 0x81.
Let trailing be pointer % 157.
Let offset be 0x40 if trailing is less than 0x3F, otherwise 0x62.
Return two bytes whose values are leading and trailing + offset.
EUC-JP’s decoder has an associated:
EUC-JP’s decoder’s handler, given ioQueue and byte, runs these steps:
If byte is end-of-queue and EUC-JP leading is not 0x00, then set EUC-JP leading to 0x00 and return error.
If byte is end-of-queue and EUC-JP leading is 0x00, then return finished.
If EUC-JP leading is 0x8E and byte is in the range 0xA1 to 0xDF, inclusive, then set EUC-JP leading to 0x00 and return a code point whose value is 0xFF61 − 0xA1 + byte.
If EUC-JP leading is 0x8F and byte is in the range 0xA1 to 0xFE, inclusive, then set EUC-JP jis0212 to true, set EUC-JP leading to byte, and return continue.
If EUC-JP leading is not 0x00:
Let leading be EUC-JP leading.
Set EUC-JP leading to 0x00.
Let codePoint be null.
If leading and byte are both in the range 0xA1 to 0xFE, inclusive, then set codePoint to the index code point for (leading − 0xA1) × 94 + byte − 0xA1 in index jis0208 if EUC-JP jis0212 is false and in index jis0212 otherwise.
Set EUC-JP jis0212 to false.
If codePoint is non-null, then return a code point whose value is codePoint.
If byte is an ASCII byte, then restore byte to ioQueue.
Return error.
If byte is an ASCII byte, then return a code point whose value is byte.
If byte is 0x8E, 0x8F, or in the range 0xA1 to 0xFE, inclusive, then set EUC-JP leading to byte and return continue.
Return error.
EUC-JP’s encoder’s handler, given unused and codePoint, runs these steps:
If codePoint is end-of-queue, then return finished.
If codePoint is an ASCII code point, then return a byte whose value is codePoint.
If codePoint is U+00A5 (¥), then return byte 0x5C.
If codePoint is U+203E (‾), then return byte 0x7E.
If codePoint is in the range U+FF61 (。) to U+FF9F (゚), inclusive, then return two bytes whose values are 0x8E and codePoint − 0xFF61 + 0xA1.
If codePoint is U+2212 (−), then set it to U+FF0D (-).
Let pointer be the index pointer for codePoint in index jis0208.
If pointer is non-null, it is less than 8836 due to the nature of index jis0208 and the index pointer operation.
If pointer is null, then return error with codePoint.
Let leading be pointer / 94 + 0xA1.
Let trailing be pointer % 94 + 0xA1.
Return two bytes whose values are leading and trailing.
ISO-2022-JP’s decoder has an associated:
ISO-2022-JP’s decoder’s handler, given ioQueue and byte, runs these steps, switching on ISO-2022-JP decoder state:
Based on byte:
Set ISO-2022-JP decoder state to escape start and return continue.
Set ISO-2022-JP output to false and return a code point whose value is byte.
Return finished.
Set ISO-2022-JP output to false and return error.
Based on byte:
Set ISO-2022-JP decoder state to escape start and return continue.
Set ISO-2022-JP output to false and return code point U+00A5 (¥).
Set ISO-2022-JP output to false and return code point U+203E (‾).
Set ISO-2022-JP output to false and return a code point whose value is byte.
Return finished.
Set ISO-2022-JP output to false and return error.
Based on byte:
Set ISO-2022-JP decoder state to escape start and return continue.
Set ISO-2022-JP output to false and return a code point whose value is 0xFF61 − 0x21 + byte.
Return finished.
Set ISO-2022-JP output to false and return error.
Based on byte:
Set ISO-2022-JP decoder state to escape start and return continue.
Set ISO-2022-JP output to false, ISO-2022-JP leading to byte, ISO-2022-JP decoder state to trailing byte, and return continue.
Return finished.
Set ISO-2022-JP output to false and return error.
Based on byte:
Set ISO-2022-JP decoder state to escape start and return error.
Set the ISO-2022-JP decoder state to leading byte.
Let pointer be (ISO-2022-JP leading − 0x21) × 94 + byte − 0x21.
Let codePoint be the index code point for pointer in index jis0208.
If codePoint is null, then return error.
Return a code point whose value is codePoint.
Set the ISO-2022-JP decoder state to leading byte and return error.
Set ISO-2022-JP decoder state to leading byte and return error.
If byte is either 0x24 or 0x28, then set ISO-2022-JP leading to byte, ISO-2022-JP decoder state to escape, and return continue.
If byte is not end-of-queue, then restore byte to ioQueue.
Set ISO-2022-JP output to false, ISO-2022-JP decoder state to ISO-2022-JP decoder output state, and return error.
Let leading be ISO-2022-JP leading and set ISO-2022-JP leading to 0x00.
Let state be null.
If leading is 0x28 and byte is 0x42, then set state to ASCII.
If leading is 0x28 and byte is 0x4A, then set state to Roman.
If leading is 0x28 and byte is 0x49, then set state to katakana.
If leading is 0x24 and byte is either 0x40 or 0x42, then set state to leading byte.
If state is non-null:
Set ISO-2022-JP decoder state and ISO-2022-JP decoder output state to state.
Let output be the value of ISO-2022-JP output.
Set ISO-2022-JP output to true.
If byte is end-of-queue, then restore leading to ioQueue; otherwise, restore « leading, byte » to ioQueue.
Set ISO-2022-JP output to false, ISO-2022-JP decoder state to ISO-2022-JP decoder output state and return error.
The ISO-2022-JP encoder is the only encoder for which the concatenation of multiple outputs can result in an error when run through the corresponding decoder.
Encoding U+00A5 (¥) gives 0x1B 0x28 0x4A 0x5C 0x1B 0x28 0x42. Doing that twice, concatenating the results, and then decoding yields U+00A5 U+FFFD U+00A5.
ISO-2022-JP’s encoder has an associated ISO-2022-JP encoder state which is ASCII, Roman, or jis0208, initially ASCII.
ISO-2022-JP’s encoder’s handler, given ioQueue and codePoint, runs these steps:
If codePoint is end-of-queue and ISO-2022-JP encoder state is not ASCII, then set ISO-2022-JP encoder state to ASCII and return three bytes 0x1B 0x28 0x42.
If codePoint is end-of-queue and ISO-2022-JP encoder state is ASCII, then return finished.
If ISO-2022-JP encoder state is ASCII or Roman, and codePoint is U+000E, U+000F, or U+001B, then return error with U+FFFD (�).
This returns U+FFFD (�) rather than codePoint to prevent attacks.
If ISO-2022-JP encoder state is ASCII and codePoint is an ASCII code point, then return a byte whose value is codePoint.
If ISO-2022-JP encoder state is Roman and codePoint is an ASCII code point, excluding U+005C (\) and U+007E (~), or is U+00A5 (¥) or U+203E (‾):
If codePoint is an ASCII code point, then return a byte whose value is codePoint.
If codePoint is U+00A5 (¥), then return byte 0x5C.
If codePoint is U+203E (‾), then return byte 0x7E.
If codePoint is an ASCII code point, and ISO-2022-JP encoder state is not ASCII, then restore codePoint to ioQueue, set ISO-2022-JP encoder state to ASCII, and return three bytes 0x1B 0x28 0x42.
If codePoint is either U+00A5 (¥) or U+203E (‾), and ISO-2022-JP encoder state is not Roman, then restore codePoint to ioQueue, set ISO-2022-JP encoder state to Roman, and return three bytes 0x1B 0x28 0x4A.
If codePoint is U+2212 (−), then set it to U+FF0D (-).
If codePoint is in the range U+FF61 (。) to U+FF9F (゚), inclusive, then set it to the index code point for codePoint − 0xFF61 in index ISO-2022-JP katakana.
Let pointer be the index pointer for codePoint in index jis0208.
If pointer is non-null, it is less than 8836 due to the nature of index jis0208 and the index pointer operation.
If pointer is null:
If ISO-2022-JP encoder state is jis0208, then restore codePoint to ioQueue, set ISO-2022-JP encoder state to ASCII, and return three bytes 0x1B 0x28 0x42.
Return error with codePoint.
If ISO-2022-JP encoder state is not jis0208, then restore codePoint to ioQueue, set ISO-2022-JP encoder state to jis0208, and return three bytes 0x1B 0x24 0x42.
Let leading be pointer / 94 + 0x21.
Let trailing be pointer % 94 + 0x21.
Return two bytes whose values are leading and trailing.
Shift_JIS’s decoder has an associated Shift_JIS leading, which is a byte, initially 0x00.
Shift_JIS’s decoder’s handler, given ioQueue and byte, runs these steps:
If byte is end-of-queue and Shift_JIS leading is not 0x00, then set Shift_JIS leading to 0x00 and return error.
If byte is end-of-queue and Shift_JIS leading is 0x00, then return finished.
If Shift_JIS leading is not 0x00:
Let leading be Shift_JIS leading.
Set Shift_JIS leading to 0x00.
Let pointer be null.
Let offset be 0x40 if byte is less than 0x7F; otherwise 0x41.
Let leadingOffset be 0x81 if leading is less than 0xA0; otherwise 0xC1.
If byte is in the range 0x40 to 0x7E, inclusive, or 0x80 to 0xFC, inclusive, then set pointer to (leading − leadingOffset) × 188 + byte − offset.
If pointer is in the range 8836 to 10715, inclusive, then return a code point whose value is 0xE000 − 8836 + pointer.
This is interoperable legacy from Windows known as EUDC.
Let codePoint be null if pointer is null; otherwise the index code point for pointer in index jis0208.
If codePoint is non-null, then return a code point whose value is codePoint.
If byte is an ASCII byte, then restore byte to ioQueue.
Return error.
If byte is an ASCII byte or 0x80, then return a code point whose value is byte.
If byte is in the range 0xA1 to 0xDF, inclusive, then return a code point whose value is 0xFF61 − 0xA1 + byte.
If byte is in the range 0x81 to 0x9F, inclusive, or 0xE0 to 0xFC, inclusive, then set Shift_JIS leading to byte and return continue.
Return error.
Shift_JIS’s encoder’s handler, given unused and codePoint, runs these steps:
If codePoint is end-of-queue, then return finished.
If codePoint is an ASCII code point or U+0080, then return a byte whose value is codePoint.
If codePoint is U+00A5 (¥), then return byte 0x5C.
If codePoint is U+203E (‾), then return byte 0x7E.
If codePoint is in the range U+FF61 (。) to U+FF9F (゚), inclusive, then return a byte whose value is codePoint − 0xFF61 + 0xA1.
If codePoint is U+2212 (−), then set it to U+FF0D (-).
Let pointer be the index Shift_JIS pointer for codePoint.
If pointer is null, then return error with codePoint.
Let leading be pointer / 188.
Let leadingOffset be 0x81 if leading is less than 0x1F; otherwise 0xC1.
Let trailing be pointer % 188.
Let offset be 0x40 if trailing is less than 0x3F; otherwise 0x41.
Return two bytes whose values are leading + leadingOffset and trailing + offset.
EUC-KR’s decoder has an associated EUC-KR leading, which is a byte, initially 0x00.
EUC-KR’s decoder’s handler, given ioQueue and byte, runs these steps:
If byte is end-of-queue and EUC-KR leading is not 0x00, then set EUC-KR leading to 0x00 and return error.
If byte is end-of-queue and EUC-KR leading is 0x00, then return finished.
If EUC-KR leading is not 0x00:
Let leading be EUC-KR leading.
Set EUC-KR leading to 0x00.
Let pointer be null.
If byte is in the range 0x41 to 0xFE, inclusive, then set pointer to (leading − 0x81) × 190 + (byte − 0x41).
Let codePoint be null if pointer is null; otherwise the index code point for pointer in index EUC-KR.
If codePoint is non-null, then return a code point whose value is codePoint.
If byte is an ASCII byte, then restore byte to ioQueue.
Return error.
If byte is an ASCII byte, then return a code point whose value is byte.
If byte is in the range 0x81 to 0xFE, inclusive, then set EUC-KR leading to byte and return continue.
Return error.
EUC-KR’s encoder’s handler, given unused and codePoint, runs these steps:
If codePoint is end-of-queue, then return finished.
If codePoint is an ASCII code point, then return a byte whose value is codePoint.
Let pointer be the index pointer for codePoint in index EUC-KR.
If pointer is null, then return error with codePoint.
Let leading be pointer / 190 + 0x81.
Let trailing be pointer % 190 + 0x41.
Return two bytes whose values are leading and trailing.
The replacement encoding exists to prevent certain attacks that abuse a mismatch between encodings supported on the server and the client.
replacement’s decoder has an associated replacement error returned, which is a boolean, initially false.
replacement’s decoder’s handler, given unused and byte, runs these steps:
If byte is end-of-queue, then return finished.
If replacement error returned is false, then set replacement error returned to true and return error.
Return finished.
UTF-16BE/LE is UTF-16BE or UTF-16LE.
A byte order mark has priority over a label as it has been found to be more accurate in deployed content. Therefore it is not part of the shared UTF-16 decoder algorithm, but rather the decode algorithm.
shared UTF-16 decoder has an associated:
shared UTF-16 decoder’s handler, given ioQueue and byte, runs these steps:
If byte is end-of-queue and either UTF-16 leading byte or UTF-16 leading surrogate is non-null, then set UTF-16 leading byte and UTF-16 leading surrogate to null, and return error.
If byte is end-of-queue and UTF-16 leading byte and UTF-16 leading surrogate are null, then return finished.
If UTF-16 leading byte is null, then set UTF-16 leading byte to byte and return continue.
Let codeUnit be the result of:
(UTF-16 leading byte << 8) + byte.
(byte << 8) + UTF-16 leading byte.
Set UTF-16 leading byte to null.
If UTF-16 leading surrogate is non-null:
Let leadingSurrogate be UTF-16 leading surrogate.
Set UTF-16 leading surrogate to null.
If codeUnit is a trailing surrogate, then return a scalar value from surrogates given leadingSurrogate and codeUnit.
Let byte1 be codeUnit >> 8.
Let byte2 be codeUnit & 0x00FF.
Let bytes be a list of two bytes whose values are byte1 and byte2, if is UTF-16BE decoder is true; otherwise byte2 and byte1.
If codeUnit is a leading surrogate, then set UTF-16 leading surrogate to codeUnit and return continue.
If codeUnit is a trailing surrogate, then return error.
Return code point codeUnit.
UTF-16BE’s decoder is shared UTF-16 decoder with its is UTF-16BE decoder set to true.
"utf-16" is a label for UTF-16LE to deal with
deployed content.
UTF-16LE’s decoder is shared UTF-16 decoder.
While technically this is a single-byte encoding, it is defined separately as it can be implemented algorithmically.
x-user-defined’s decoder’s handler, given unused and byte, runs these steps:
If byte is end-of-queue, then return finished.
If byte is an ASCII byte, then return a code point whose value is byte.
Return a code point whose value is 0xF780 + byte − 0x80.
x-user-defined’s encoder’s handler, given unused and codePoint, runs these steps:
If codePoint is end-of-queue, then return finished.
If codePoint is an ASCII code point, then return a byte whose value is codePoint.
If codePoint is in the range U+F780 to U+F7FF, inclusive, then return a byte whose value is codePoint − 0xF780 + 0x80.
Return error with codePoint.
Browsers are encouraged to not enable overriding the encoding of a resource. If such a feature is nonetheless present, browsers should not offer UTF-16BE/LE as an option, due to the aforementioned security issues. Browsers should also disable this feature if the resource was decoded using UTF-16BE/LE.
Instead of supporting I/O queues with arbitrary restore, the decoders for encodings in this standard could be implemented with:
The ability to unread the current byte.
A single-byte buffer for gb18030 (an ASCII byte) and ISO-2022-JP (0x24 or 0x28).
For gb18030 when hitting a bogus byte while gb18030 third is not 0x00, gb18030 second could be moved into the single-byte buffer to be returned next, and gb18030 third would be the new gb18030 first, checked for not being 0x00 after the single-byte buffer was returned and emptied. This is possible as the range for the first and third byte in gb18030 is identical.
The ISO-2022-JP encoder needs ISO-2022-JP encoder state as additional state, but other than that, none of the encoders for encodings in this standard require additional state or buffers.
There have been a lot of people that have helped make encodings more interoperable over the years and thereby furthered the goals of this standard. Likewise many people have helped making this standard what it is today.
With that, many thanks to Adam Rice, Alan Chaney, Alexander Shtuchkin, Allen Wirfs-Brock, Andreu Botella, Aneesh Agrawal, Arkadiusz Michalski, Asmus Freytag, Ben Noordhuis, Bnaya Peretz, Boris Zbarsky, Bruno Haible, Cameron McCormack, Charles McCathieNeville, Christopher Foo, CodifierNL, David Carlisle, Domenic Denicola, Dominique Hazaël-Massieux, Doug Ewell, Erik van der Poel, 譚永鋒 (Frank Yung-Fong Tang), Glenn Maynard, Gordon P. Hemsley, Henri Sivonen, Ian Hickson, J. King, James Graham, Jeffrey Yasskin, John Tamplin, Joshua Bell, 村井純 (Jun Murai), 신정식 (Jungshik Shin), Jxck, 강 성훈 (Kang Seonghoon), 川幡太一 (Kawabata Taichi), Ken Lunde, Ken Whistler, Kenneth Russell, 田村健人 (Kent Tamura), Leif Halvard Silli, Luke Wagner, Maciej Hirsz, Makoto Kato, Mark Callow, Mark Crispin, Mark Davis, Martin Dürst, Masatoshi Kimura, Mattias Buelens, Ms2ger, Nigel Megitt, Nigel Tao, Norbert Lindenberg, Øistein E. Andersen, Peter Krefting, Philip Jägenstedt, Philip Taylor, Richard Ishida, Robbert Broersma, Robert Mustacchi, Ryan Dahl, Sam Sneddon, Shawn Steele, Simon Montagu, Simon Pieters, Simon Sapin, Stephen Checkoway, 寺田健 (Takeshi Terada), Vyacheslav Matva, Wolf Lammen, and 成瀬ゆい (Yui Naruse) for being awesome.
This standard is written by Anne van Kesteren (Apple, [email protected]). The API chapter was initially written by Joshua Bell (Google).
Copyright © WHATWG (Apple, Google, Mozilla, Microsoft). This work is licensed under a Creative Commons Attribution 4.0 International License. To the extent portions of it are incorporated into source code, such portions in the source code are licensed under the BSD 3-Clause License instead.
This is the Living Standard. Those interested in the patent-review version should view the Living Standard Review Draft.