All About Alphabets and Such

Alphabets, character sets, Java charsets, code pages, collators, encoders and decoders can be confusing, especially when overlaying the COBOL notion of these with the Java world’s view. Hopefully this will shed some light on the topic, at least in the context of P3/COBOL.

First, a few distinctions and definitions. Let’s start with COBOL’s Alphabets and Codesets.

Terms

Codeset, Alphanumeric, National, Charset, Native

A codeset is the correspondence between the algebraic value of a character in memory and the actual character (figure) it represents. The COBOL program has two codesets, the alphanumeric codeset and the national codeset. The alphanumeric codeset is used to represent characters in alphanumeric (including alphabetic and alphanumeric-edited, as well as numeric-edited) data items and literals. Since the stored form of these characters are always 8-bit bytes (in all COBOL dialects), the alphanumeric codeset establishes the byte value of each character that can be represented in the codeset.

Most COBOL systems use a variant of the ASCII codeset as the native codeset for programs. Mainframe COBOL systems often use an EBCDIC codeset natively, with an option to use an ASCII codeset. This means that for an ASCII codeset, the value of a character in memory is equal to its ASCII code point. In fact, the ASCII variant that is most often used is the ISO-8859-1 character set, which includes all of the ASCII characters and, in addition, all of the characters needed for most Western languages. In this codeset, all 256 of the possible byte values are assigned to specific ISO-8859 code points. From now on, I’ll refer to the selected codeset of the COBOL program as the native codeset. The set of all characters coded by this codeset will be the native charset (more on that later).

Alphabet, Collation, Input Map, Output Map, Ordinal

In COBOL, an alphabet is a combination of three transformations (mappings) of character code values in the native codeset. First, it provides a collation. A collation is simply the position of each character in the codeset relative to every other character in the codeset. Most often a collation determines how a string of characters coded according to the codeset will sort against other strings. The collation can be thought of as a matrix of 256 collation values, each of which determines where the character corresponding to the index of each value will sort. In addition the collation determines two other maps of significance to COBOL, the input map and the output map. The input map determines what codeset value (or ordinal) to assign for an arbitrary incoming character value, and the output map determines what outgoing value to use for each alphanumeric byte value.

Program Collating Sequence

It should be clear that a COBOL alphabet is not the same as its native codeset. In fact, the alphabet describes how to manipulate and compare items stored according to the native codeset, without regard to what that codeset represents. The COBOL programmer may define as many alphabets as they wish, and may choose one of them to determine the program collating sequence. This collation is used for all comparisons between alphanumeric characters and strings of alphanumeric characters in the native codeset. If no collating sequence is specified, the default collation is used. This is the lexicographic collating sequence that results from assigning each code point its ordinal value as its collating position.

National Codeset, Alphabet, Collating Sequence

The national codeset in COBOL is required to contain the alphanumeric codeset as a subset. It may, and always does, contain many more characters, however. In P3/COBOL, the national codeset contains all of the characters that can be present in a Java String. This corresponds to the Unicode characters that can be represented in 16 bits (two bytes). As a result, the native national codeset is encoded into a pair of bytes in memory as the UTF-16 code point of the corresponding character.

A national alphabet is simply an alphabet that comprises a collation for some or all of the characters in the native national codeset. The input and output maps for this alphabet provide mappings from and to an 8-bit encoding of characters in the codeset.

A string of alphanumeric characters can always be transformed into an equivalent string of national characters, but a national string cannot always be transformed into an equivalent string of alphanumeric characters. The COBOL rules regarding the interaction of national and alphanumeric data items reflect this fact. For example, a comparison of an alphanumeric item to a national item is defined as the comparison of the alphanumeric item converted to a national equivalent with the other national item.

National Data in P3/COBOL

P3/COBOL implements the national data facilities defined by the 2014 COBOL standard, including elementary and group national usage, national literals, national alphabets and symbolic characters, and operations involving both alphanumeric (DISPLAY) and national data.

Declaring National Data

Elementary items may use USAGE NATIONAL or NATIONAL-EDITED, and a PICTURE containing N describes national character positions. A group may specify GROUP-USAGE IS NATIONAL; subordinate items then inherit national usage unless an explicitly permitted declaration says otherwise. P3/COBOL diagnoses subordinate declarations that conflict with the group's usage.

01  customer-name group-usage is national.
    05  family-name pic n(30).
    05  given-name  pic n(20).

Each national character position occupies two bytes and represents a UTF-16 code unit. Consequently, LENGTH reports national character positions while BYTE-LENGTH reports the corresponding storage size in bytes.

DISPLAY and NATIONAL Interaction

P3/COBOL applies the standard conversion rules when national and alphanumeric values interact. A comparison between a DISPLAY value and a national value converts the DISPLAY operand to its national representation before comparing. Moving national data to DISPLAY may require replacement when a character has no representation in the selected native charset.

National data maps naturally to Java String values when COBOL and Java interoperate. Intrinsic functions implemented for national conversion include CHAR-NATIONAL, DISPLAY-OF, and NATIONAL-OF; common character functions also accept national arguments where indicated. See the intrinsic function summary for argument, result, and implementation information.

National Alphabets and Symbolic Characters

An ALPHABET clause may define an alphabet FOR NATIONAL, and a program may select separate collating sequences for alphanumeric and national data. National alphabets operate over the 65,536 UTF-16 code-unit positions rather than the 256 positions available to an alphanumeric alphabet.

A SYMBOLIC CHARACTERS FOR NATIONAL clause defines national symbolic characters, optionally relative to a named national alphabet. Their ordinal values range from 1 through 65,536. These definitions participate in literals, comparisons, and other contexts in the same manner as alphanumeric symbolic characters, but retain national type.

Locale Support Boundary

P3/COBOL provides limited, platform-supported locale collation services for alphanumeric and national values. It does not implement the complete standard LOCALE IS ACTIVE framework, and the LOCALE-COMPARE, LOCALE-DATE, LOCALE-TIME, and LOCALE-TIME-FROM-SECONDS intrinsic functions are currently recognized but not implemented. Comparisons and string operations on UTF-8 data use strict binary collation.

Source Program Format

Every COBOL program begins as a source program. Since this source program is stored on some form of persistent storage (e.g., a disk), the characters comprising the source are encoded according to some algorithm. If they are encoded in a loss-less coding such as UTF-8, every character that is possible to represent in the Unicode set can be represented and passed through to the COBOL compiler that will process the source. If they are encoded in a coding such as ISO 8859-1 (aka Latin-1), then any characters that are also in the Latin-1 set will be accurately passed on the compiler, but any other characters will be mapped onto a different character that is in the Latin-1 set (such as “?”, for example). Therefore, the choice of editor and the mode in which it maintains and stores the source code characters is of vital importance if any unusual (non-ASCII) characters are used. In fact, any editor or tool that processes the source code can (inadvertently) change the characters in the source, thus corrupting it and, in some cases, render it invalid as COBOL compiler input. To avoid this, it is important to realize the coding of the source and to be sure that any tool used to read and output the source has a correct view of its coding, both for input and output. In any case, P3/COBOL has options to control how the compiler determines the coding of the source files, and interprets them correctly based on that determination. By default, it will attempt to “sniff out” the correct coding based on a sampling of the first part of the source code file. This will usually result in a correct selection for the common codings of 8859-1 and ASCII, but may make an incorrect determination if the sample source does not contain some non-ASCII characters that occur in the full source. If this is the case, an option is provided to set the default character set to choose if the sniffing does not come to a conclusive result. Additionally, there is an option to force the setting of the source coding to the selected default value, regardless of the sniffing result. One or both of these means can allow P3/COBOL to correctly interpret the source code for any source code format.

Once the source format is correctly known to the compiler, considerations of the actual characters used in the source may be made.

Characters in the Source Context

In P3/COBOL source code, the interpretation and semantics associated with a specific character is a function of its context in the program.

Reserved Words

All reserved words use only characters from the ASCII set, so there is no coding issue possible.

Comments, Sequence Area, Identification Area

Any characters may be used as commentary in a P3/COBOL program as long as the end of the commentary can be unambiguously determined according the COBOL syntax rules.

Literals

Character literals used in the COBOL program have special significance with respect to the choice of source codeset. If a literal is not explicitly identified as a national literal (e.g., is of the form "..." or '...'), and it does not contain any characters that are not present in the program’s native codeset, it is encoded by the compiler as a native alphanumeric string. Remember, this has nothing to do with how the source itself coded the character since the compiler has converted that coding into an internal 16-bit character code that can represent any and all characters that could possibly be in the source string.

If the literal contains at least one character that cannot be represented in the native character set, the string will be considered to be a national string, and the resulting string will be used in the context of the literal as if it had been explicitly designated as a national string.

If the literal is implicitly or explicitly designated as a national string, the interpretation of the characters varies according to the use in the source program. If the national string is used in a VALUE clause specifying the initial value of an alphanumeric item, the characters will be converted as if an elementary MOVE from a national item to the alphanumeric item were performed. If the national string is used in a comparison with an alphanumeric item, the alphanumeric item is converted to a national string and the comparison is then made using the national character set.

Note, however, that ASCII control characters (especially "new line" and "carriage return") should not be used in their "raw" form inside COBOL literals, since they can cause the compiler to interpret them not as characters inside a literal, but rather the abnormal end of the literal. If you need to include these characters in a literal value, you may use a national literal with the Java escape code for the character instead.

For example, this is a new line \u000A, and this is one also \n will be interpreted as a national string containing two new line characters by the P3/COBOL compiler. The Java escapes may be used anywhere a normal character can appear in a national literal string.

User-defined Words

Most of the Unicode characters that correspond to letters or numbers (including non-Latin symbols) are allowed in user-defined words. This means that variable names, for example, may use Kanji symbols if the source character set allows those characters to be represented. Anything that is not considered "special" by COBOL is permitted.

Characters in the Runtime Context

When the P3/COBOL-compiled Java application is running, there are two character sets of significance: the set of characters in the native code set and the set in the national code set. The national code set is the same as that at compile-time, namely, all 16-bit Unicode Basic Multiligual Plane (BMP) code points. This is also the code set used when interacting with other Java components that expect the Java String type argument or result. The native code set corresponds to characters that can be encoded and decoded by a Java Charset capable of mapping the 8-bit alphanumeric codes used by the COBOL program into or out of the Java String characters. Any of several native code sets may be used, but if the program was compiled for a specific set (this is usually the case), then the runtime must be configured for a compatible native charset. The most common choices for the native charset are one of the variants of the ISO 8859 encoding or one of the variants of IBM's EBCDIC encodings. In some cases, the basic ASCII set is used, as are some of those corresponding to the Windows/DOS code pages such as "1252".

Whatever the choice, there is always a single "native" representation of characters in alphanumeric (USAGE DISPLAY) for a COBOL run unit. Equally important, however, is the collating sequence used by a program to compare alphanumeric strings, including those represented by literal values at compile time.

Collation

COBOL has a very precise way of specifying and using different ways of collating character strings. They are described by the COBOL ALPHABET clause in the ENVIRONMENT DIVISION of the program. If a particular alphabet is to be used to define the default collating sequence for the program (the "program collating sequence"), it can be specified in the PROGRAM COLLATING SEQUENCE IS ... clause of the PROGRAM-ID paragraph. If specified there, the collating sequence defined by the named alphabet will be used for all alphanumeric comparisons of DISPLAY data during the execution of the program, and any contained nested programs that do not themselves specify a program collating sequence. If no collating sequence clause is used, the program and its nested components will use the standard collating sequence, which is that of the native charset taken in ordinal order. I.e., each character will have a collation value equal to its ordinal value.

Input-output

When COBOL data is read from or written to external sources and sinks, the native runtime charset and operative collating sequence potentially affect the operation in several ways. First, any DISPLAY data in the records being written will be encoded in the native charset. This means each 8-bit byte of data corresponds to the Unicode character determined by the native charset encoding. For ASCII, this means every byte will correspond to the characters in the first 128 Unicode code points, which happen to be the ASCII 7-bit characters. If the native charset is an ISO-8859 variant, the encoded bytes will mostly correspond to the first 256 Unicode codepoints, with the possible exception of the currency character and, in some of the variants, a few other characters. If the native charset of the program is any 1:1 reversible charset (i.e., one that maps all 256 characters to unique Unicode codepoints), any DISPLAY character can decoded into a Java (Unicode) string character, and the character can be re-encoded into a byte by the same charset, and the result in memory will be the same. This is the case with default charset options, and is the case unless a non-1:1 charset is chosen for the native charset.

Non-1:1 Charsets

A non-1:1 charset might be appropriate in some cases. Perhaps the most common is that when an escape code encoding is required. An example of this is the SHIFT-JIS coding for representing the common characters in the Japanese language. In this case, the source code of the program, as well as the desired native charset could be SHIFT-JIS. If so, then not all characters are coded in a single byte. If this is the case for the native charset, the program must be aware of this property. Namely, that the X in the picture clause does not necessarily represent a single character, but that two or more X characters in the picture might be required. As long as the program is aware of this, and is coded accordingly, everything should be all right. Display coded characters will be converted to string characters according to the SHIFT-JIS encoding, and they will be converted from strings in the same way. Alphanumeric comparisons, however, will not necessarily yield the intended result due to the fact that a single character may not represent a character in the charset.

Another potential non-1:1 charset that might be used is UTF-8. UTF-8 is the standard way of encoding any Unicode character into and out of a stream of 8-bit bytes. In this encoding, all of the ASCII characters are identical to the 7-bit ASCII representation. Code points above 128 are used to define multi-byte sequences that represent all the other characters in the Unicode set, including all the 32-bit code points (even beyond those that are in the Java character set). If this charset is chosen for native COBOL charset, it will require the program to be aware of the character encoding in a manner similar to SHIFT-JIS.

Note, however, it is not necessary to choose a native UTF-8 charset just because the source program contains non-ISO-8859 characters. Quite often the source will be encoded in UTF-8, and will contain multi-byte characters that are entirely within the capability of a 1:1 charset encoding such as ISO-8859. When this is the case, the 1:1 encoding should be chosen over UTF-8 for the native charset, as this will allow the program logic itself to be ignorant of the actually encoding. The source charset may be set with the <force-default-source-charset> option, if the automatic detection of the source charset does not correctly choose the actual encoding. This is usually the case if some of the non-ASCII characters in the source appear as the incorrect character in the listing or in the editor, or if unexplainable syntax errors are indicated when the program is compiled. In many cases, however, such behavior can be due to an incorrect setting of a prior editor or tool that processed the source, in which case there is nothing that can be done to restore the correct characters without going back to the uncorrupted original source.

File Data

When COBOL records are read from or written to a COBOL file (Sequential, Relative, or Indexed organization), the data is neither encoded nor decoded. All of the 8-bit DISPLAY-usage items are represented on the external media in exactly the same form as in memory, regardless of the character sets or encodings external to the COBOL run unit. This means that, for example, a COBOL program running with a native charset of EBCDIC will write files containing EBCDIC characters, even when the operating system and everything else in the file system is using a non-EBCDIC character set. This will result in files that cannot be correctly read or edited by most available non-COBOL tools. As long as only EBCDIC COBOL programs are reading and processing the files, however, all is well.

If the program collating sequence is EBCDIC, but the native charset is ASCII (or ISO-8859), for example, the files will be read and written with DISPLAY data encoded as ASCII, and this incompatibility will not occur.

File and Key Collating Sequence and Charset

COBOL permits the collating sequence of key to be specified in the program as different from the program collating sequence. If this is done, it affects the way keys may be used to address the records, but it has no effect on the charset used on the file to encode the data.

In Summary

• Make sure the COBOL source is correctly encoded
• Make sure P3/COBOL is configured to interpret the source in its proper encoded form
• Make sure the P3/COBOL native charset is set to one that has all the characters needed
• Make sure the program collating sequence is set if the native charset collating sequence does not match the original data (for example, data that has been converted from EBCDIC to 8859)
• Make sure the non-COBOL tools are set to interpret the source and data according to the native charset chosen
• And, finally, if it doesn't look right in an editor either the editor is using the wrong charset or the file has already been corrupted; in either case don't save it until you know which it is!