P3/COBOL Indexed File System

P3/COBOL includes a complete implementation of the COBOL indexed file standards, including file locking and sharing, and both single and multiple record locking. This implementation incorporates a multi-version concurrency control (MVCC) storage management system. This storage system is implemented entirely in Java, is very high performance, and includes capabilities for both single and multi-file volumes, journaling and recovery, hot backup, transactional behavior and many other modern features. P3/COBOL makes this storage system directly available to the COBOL programmer using standard syntax and semantics as well as to the Java programmer using a Java object-oriented API.

File System Organization

The P3/COBOL file system is organized as one or more volumes, each containing one or more files. The volumes may be located anywhere, including different directories or even different computer systems. The only requirement is that they be accessible on the computer system(s) that are using the data. A built-in indexed file server is provided by P3/COBOL, and it can manage access to indexed files shared by multiple processes or computer systems connected to a common TCP/IP network. The server(s) may be started independently from the command line, or a daemon process, or can be started automatically from the COBOL application for convenience.

Configuring the Storage System

When the first :term:MVCC-based indexed file is opened for a distinct volume file path, an instance of the storage subsystem for that volume is initialized. This instance then continues to serve any indexed file requests for that volume until there are no files open on that volume or all run units accessing file(s) on that volume terminates. During the initialization process, several characteristics of the MVCC subsystem configuration may be specified. If no configuration is supplied, a default that stores all indexed files in a single volume of pre-determined characteristics is used.

Default Configuration

The default configuration will result in a single volume for each file named the same as the file's external name as specified by the COBOL SELECT statement. The file within the volume will also have the same name.

Configuring the Database

If a different configuration is desired, for example a different location for the data volume(s), the filesystem configuration options may be specified using the <file-association> option element. This element may contain a <volume-configuration> element, which applies file system options to any indexed file selected by the file association. Using this configuration, the name and location of the volume can be specified, the compression and size of the buffer pool can be set, and other details of operation can be adjusted. The available property names and meaning are outlined there. Some of the more likely configuration examples for a COBOL application are described as follows.

Associating Files With Volumes

By using the runtime configuration file’s <file-associations> option element, it is possible to group specific files with specific volumes. Volumes can be backed up and restored as an independent unit. For that reason, care should be taken when this is determined for a production system as it can be difficult to change this association once data is stored in the volumes. Although the runtime configuration itself is immutable after initialization, the applicable file association—and therefore its preconfigured volume options—is selected dynamically whenever a file is opened. This is accomplished with the <volume-configuration> element within the <file-association> element.

The format of the volume configuration element is:

<volume-configuration>
    <option key="name" value="volume name (indexed_files.mvc default)" />
    <option key="path" value="volume directory path (working directory default)"/>
    <option key="enable-compression" value="true or false (true default)"/>
    <option key="cache-size" value="size of this volume's buffer cache (in megabytes, 16 default)"/>
    <option key="page-split-size" value="size (in bytes) to trigger a page split (4096 default)"/>
    <option key="auto-flush-buffer-size" value="number of bytes to buffer before automatically flushing data to storage medium (1024 default)"/>
    <option key="auto-compact-fill-fraction" value="the % utilization below which automatic compaction will occur (50 default)"/>
    <option key="auto-flush" value="true (default) or false; allows automatic flushing to the storage medium"/>
    <option key="compact" value="false (default) or true; allows automatic compaction of storage at file close time"/>
    <option key="read-only" value="false (default) or true; declares volume as read-only and not modifiable"/>
    <option key="retention-time" value="integer milliseconds or -1 (default is 45000); sets the time period to wait before reusing discarded chunks"/>
</volume-configuration>

See Configuring File Associations for more information on this.

In order to specify a location for the volume other than the default, your volume configuration might be similar to this one:

<volume-configuration>
    <option key="name" value="myvolumename" />
    <option key="path" value="{file_dir}/"/>
</volume-configuration>

In almost all cases the default values for the other options are sufficient. If a large number of volumes are to be used, and some of them are not heavily used, the cache-size option may be used to reduce the amount of heap memory used by the volume(s).

In the above case, the path to all the files is set with an internal property. This file_dir variable can be any name, and you can use as many of them as you wish. They are dereferenced using the {...} notation, as can be seen above.

Selecting Volume Configuration by File Name

Runtime and volume options cannot be modified after runtime initialization. This does not require every indexed file to use the same fixed volume configuration.

The <file-associations> configuration acts as a routing table. Each <file-association> may contain its own <volume-configuration>. Whenever an indexed file is opened, P3/COBOL compares its file name with the configured associations, in order, and selects the first applicable association whose <name-pattern> matches.

This allows characteristics such as volume location, read-only access, retention time, compression, and cache size to be selected as a function of the file name without changing the initialized runtime options. Capture groups from the matching pattern may also be used as replacement markers in the external name and volume options.

<file-associations>
    <!-- Archive files use a read-only archive volume. -->
    <file-association appliesTo="INDEXED">
        <name-pattern>^ARCHIVE-(.*)$</name-pattern>
        <external-name>{1}</external-name>
        <volume-configuration>
            <option key="name" value="archive.mvc"/>
            <option key="path" value="/data/archive"/>
            <option key="read-only" value="true"/>
        </volume-configuration>
    </file-association>
 
    <!-- Other indexed files use the active volume. -->
    <file-association appliesTo="INDEXED">
        <name-pattern>^(.*)$</name-pattern>
        <external-name>{1}</external-name>
        <volume-configuration>
            <option key="name" value="active.mvc"/>
            <option key="path" value="/data/active"/>
            <option key="read-only" value="false"/>
        </volume-configuration>
    </file-association>
</file-associations>

For example, opening ARCHIVE-CUSTOMERS selects the first association, maps the external file name to CUSTOMERS, and uses the read-only archive volume. Opening CUSTOMERS selects the catch-all association and uses the writable active volume.

Because association order is significant, place specific patterns before general patterns such as .*.

File Migration

In some cases, the indexed data files to be used by the program are those produced by another COBOL system or sequential files containing the data with which to populate corresponding indexed files. This often occurs when P3/COBOL is replacing another system, and the files have not been migrated off-line to the standard P3/COBOL (MVCC) format. In many of these cases, P3/COBOL offers an automatic "hot" migration capability. In order for automatic migration to take place, the file being referenced must have a file-association that specifies a volume-configuration with a path and name that differs from the exisiting pre-P3/COBOL file. In other words, the file to be migrated must not have exactly the same path as the volume specified. If a file exists at the path specified in the <migration-configuration> option element and the P3/COBOL volume configured does not exist, P3/COBOL will attempt to convert the file to P3 format and populate the new volume the first time it is accessed by a P3/COBOL program. If successful, the new file residing in the P3/COBOL MVCC system will be used for all further access by P3/COBOL.

This will enable the direct automatic migration of RM/COBOL and Fujitsu NetCobol indexed files, sequential organization files, and Micro Focus variable binary sequential files. These are the only file-types currently supported by this automatic migration capability.

An example of a file association designed to automatically migrate files (e.g., RM/COBOL or Fujitsu NetCobol files) is:

<!-- this is an example of configuring indexed files for the new file system, and migrating from another file system -->
<file-associations>
    <file-association appliesTo="INDEXED">
        <name-pattern>(.*)[.]dat$</name-pattern> <!-- matches any files ending in .dat -->
        <external-name>{0}</external-name><name-pattern>.*</name-pattern> <!-- use the whole matched name as the file name -->
 
        <!-- this configures where the file volumes are (one per file in this case) -->
        <volume-configuration>
            <!-- this is the name of the volume(file) desired ("{0}" is the capture group in the name-pattern, in this case, the whole name -->
            <option key="name" value="{0}.p3inx"/>
            <!-- this is the directory to put the migrated files in and to use for the data files from that point on -->
            <option key="path" value="data"/>
        </volume-configuration>
        <migration-configuration>
            <!-- this says where to find the files to migrate when they are opened for the first time. After that this will not be used -->
            <option key="migrate-automatically-from" value="data_old/{0}"/>
        </migration-configuration>
    </file-association>
</file-associations>

Upgrading Existing MVCC Volumes

P3/COBOL 25.1 uses a newer H2 MVStore on-disk format for MVCC volumes (typically .mvc files). Volumes written by P3/COBOL 25.0, or by earlier builds that used the previous format, cannot be opened directly by the current storage manager until they are rewritten.

Automatic rewrite on open

By default, the first open of a legacy volume rewrites it in place:

1. A Fletcher-32 checksum of the live v1 volume is computed.
2. A byte-for-byte durable copy is written as <volume>.bak_<fletcher> (reused when an existing backup already matches that checksum), fsynced, and verified as legacy-readable before any cutover.
3. The volume contents are exported with an isolated 25.0-compatible reader into a temporary v2 file.
4. The golden backup is re-checked, then the temporary v2 file replaces the live path.
5. Open continues on the rewritten volume. The INFO log records the backup path and Fletcher value.

Automatic rewrite requires the legacy reader jars (h2.jar and legacy-mvstore-reader.jar) to be discoverable next to p3cobol.jar, under lib/ / lib/legacy-mvstore/, or via the system properties documented under Storage Manager Configuration.

To refuse automatic rewrite and require an offline conversion instead:

-Dcom.turrettech.p3cobol.lib.files.internal.h2mv.MvStorageManager.autoRewriteLegacyVolumes=false

Offline rewrite and undelete

Use the volume utility when you prefer a controlled conversion before production open, or when you need to restore a logical file name for an anonymous file map:

• rewrite-volume — convert a legacy volume in place or to a new path (writes a verified .bak_<fletcher> golden v1 copy before cutover; automatically undeletes anonymous maps so the migrated file can be opened by name)
• undelete — restore p3.name entries for anonymous (deleted) file maps on an existing volume

This upgrade path is distinct from foreign-file migration (RM/COBOL, NetCobol, sequential sources into a new MVCC volume). Foreign migration continues to create current-format volumes; only existing P3 .mvc volumes from the prior H2 era need rewrite.

File Sharing

File sharing is selected for access to a given indexed file by coding a SHARING clause in the file control entry that describes the file connector or in an OPEN statement that opens the file connector. A sharing mode specified in an OPEN statement overrides a sharing mode specified in a file control entry. Permitted sharing and input-output operations are further specified by the i-o modes of OPEN statements.

Each OPEN statement succeeds or fails on the basis of the most restrictive sharing mode and i-o mode already in effect for the file as well as the sharing mode and i-o mode specified for that OPEN statement. The permitted sharing, then, is dynamic based on the sharing and i-o modes of the on-going opening and closing of file connectors. When a shared file is already open, a subsequent OPEN statement that specifies the OUTPUT open mode for that file will be unsuccessful.

The specification for the OPEN statement in Table 20, Opening available shared files that are currently open by another file connector, should be reviewed for an understanding of the detailed interaction of opening request and sharing modes. The following is a simplified description:

The default if neither the OPEN statement nor the file control entry specifies the sharing mode is determined by the compiler configuration option default-sharing. If this option is not specified, the default sharing mode will be determined by the OPEN mode of the file according to the table below.

Locking

Record locking

Record locking gives application developers the ability to control concurrent access to logical records in a shared file. Concurrent access is possible when more than one file connector opens a file. In order to guarantee the integrity of records in a file, it is usually necessary for applications to restrict access to records being updated or deleted - but this depends on the nature of the application. The mechanism for restricting access to a record is called a record lock.

P3/COBOL provides two modes of record locking - automatic locking and manual locking - that can be selected by coding a LOCK MODE clause in a file control entry. The following combinations of access mode and lock mode are provided:

When sharing WITH NO OTHER is specified, record locking is not applicable and use of any syntax that requests record locking is ignored.

Automatic locking

LOCK MODE AUTOMATIC indicates that the runtime system will take care of locking records. When automatic locking is in effect, the time at which locks are acquired and released is defined by the COBOL specification, rather than controlled by the application. At the same time, however, other file connectors can access the file with either manual locking or automatic locking. It is easier for an application to control concurrent access to records with automatic locking than with manual locking, but it might not meet the needs of all applications.

Single record or multiple record automatic locking can be selected. With single record automatic locking, a file connector has a lock on one record at a time. The successful execution of a READ statement establishes a lock on the newly read record. The execution of a READ, REWRITE, WRITE, DELETE, UNLOCK, or CLOSE statement releases a lock on a previously-locked record.

With multiple record automatic locking, all newly read records are automatically locked and the lock for a record is retained until the record is deleted, or an UNLOCK statement is executed for the file connector, or the file connector is closed. The following illustrates a file-control paragraph that defines a file connector (my- file), where the file (accounts) resides on a mass storage device. SHARING WITH READ ONLY indicates that the file can be shared with other file connectors as long as those files are open for READ ONLY. The file connector my-file is not itself restricted to read only.

FILE-CONTROL.
SELECT my-file ASSIGN TO accounts
...
SHARING WITH READ ONLY
LOCK MODE IS AUTOMATIC WITH LOCK ON MULTIPLE RECORDS.

Single record automatic locking is selected by the following lock mode clause:

LOCK MODE IS AUTOMATIC

The automatic acquisition and release of record locks is summarized in the table, Summary of Record Lock Acquisition and Release

Manual locking

LOCK MODE MANUAL indicates that whether and when locks are acquired and released is completely under control of the application; thus, data integrity is completely under application control. Manual locking provides the flexibility to select locking only for records that require it and to release locks at convenient times. The use of manual locking requires careful design of all the applications that share access to a given file.

An application can manually lock a single record at a time or multiple records at a time. With single record manual locking, each record lock is established by specifying the LOCK phrase on an I/O statement and the lock is automatically released on the next execution of a READ, REWRITE, WRITE, DELETE, UNLOCK, or CLOSE statement -- in the same manner as for single record automatic locking. With multiple record manual locking, each record lock is established by specifying the LOCK phrase on an I/O statement and all locks are held until explicitly released. Successful execution of an UNLOCK statement or a CLOSE statement releases all locks. Successful execution of an I/O statement with a NO LOCK phrase releases a lock on the record that is processed by that I/O statement. Successful execution of a DELETE statement releases a lock on the deleted record.

Additional options on I/O statements allow for selective actions that are not normally needed, but may be useful in special circumstances:

READ ... ADVANCING ON LOCK

Locked records are skipped and the next unlocked record in sequential order is retrieved.

READ ... IGNORING LOCK

A record is retrieved even if it is locked.

The following illustrates a file-control paragraph that defines a file connector (my-file), where the file (accounts) resides on a mass storage device. SHARING WITH ALL OTHER indicates that the file can be shared with other file connectors and any of them can update records.

FILE-CONTROL.
SELECT my-file ASSIGN TO accounts
...
SHARING WITH ALL OTHER
LOCK MODE IS MANUAL WITH LOCK ON MULTIPLE RECORDS.

The manual locking and unlocking of records is controlled by options on I/O statements, by the UNLOCK statement, and by the CLOSE statement, as summarized in the following table, Summary of Record Lock Acquisition and Release.

Summary of Record Lock Acquisition and Release

Retry

The RETRY phrase gives the capability of indicating a willingness to wait to obtain a locked file or record. The RETRY phrase may specify a waiting period in seconds or the number of times to retry the operation. The number of times to retry can be a number or the word FOREVER, which implies that the operating environment attempts to retry the operation until some external occurrence terminates the task or the locked record or file becomes available. If the lock condition exists throughout, or if there is a lock but no RETRY phrase is specified, the operating environment reports the file sharing conflict condition or record operation conflict condition by setting the I-O status value associated with the file connector to the appropriate value.

Multi-JVM File Sharing

A COBOL indexed file volume can be accessed by only a single process (i.e., single instance of the JVM) at one time. For many web-based multi-user applications this is sufficient since there is a single JVM running all of the services for the web application server. In some cases, however, more than one JVM on the same or different computers requires simultaneous access to one or more COBOL indexed files. This would occur in most distributed web server environments, where multiple web servers are accessing a pool of data files, or in the case of traditional multi-user client-based applications that share files on remote server computers. In these cases P3/COBOL provides a remote file server that can coordinate safe access to multiple files. This server is included with the standard P3/COBOL command-line package, and can be started both automatically and manually.

Configuring the Server

In order to use the remote file server, a special element must be specified in the <file-association> element (see Configuring File Associations) for any files that are desired to be accessed via the server. This configuration element declares that any files included in the containing association are to be accessed by the specified server, and the options provided in the configuration provide the name of the host system upon which the server is running, the port number the server is using for connections to clients, the maximum time to allow for connecting to the server, and the maximum time to allow for any file operation to be processed by the server. Specifically, an example of the configuration element and options are as follows:

<remote-server-configuration>
    <option key="host-name" value="myserver" />
    <option key="port" value="1234" />
    <option key="connect-timeout" value="10000" />
    <option key="operation-timeout" value="1000" />
    <option key="is-daemon" value="false"/>
    <option key="idle-timeout" value="60"/>
</remote-server-configuration>

All of the option elements are optional, with the following default values used if omitted:

Starting the Server

The file server may be started in one of two ways.

If the host-name specified (either explicitly or by default) is "localhost", and there is no file server already running in the JVM on the specified port, a file server is automatically started. If this happens, the server will remain running until there are no run units in the JVM still accessing files through the specified port, or the JVM terminates.

If the conditions for automatic starting of the intended server are not present or desired, the file server can be started manually from the command line on any computer(s) with network connectivity to the client machines. This is done in whatever manner is preferred, but for example the following java command could be used:

java -cp $P3_HOME/lib/p3cobol.jar com.turrettech.p3cobol.lib.files.indexed.IndexedFileServer

This assumes that the P3_HOME environment variable has been set to the location of the installed P3/COBOL command-line package (see P3_HOME).

The file server can be stopped by terminating the process started as described above.

Server Considerations

The MVCC storage manager that is used by the default P3/COBOL file connector requires exclusive access to the volume file(s) for any JVM instance that requires access to that file(s). This means that if COBOL files are to be shared among multiple JVM instances, either on a single machine or on a network of machines, the Indexed File Server must be used. Additionally, only one file server instance may be used to access a set of files on the same volume file. Multiple servers may be used when the files accessed through each server are never accessed through any other server at the same moment, however. In other words, you may have non-overlapping groups of shared files open simultaneously by multiple servers, but you may not have the same file or files opened by multiple servers simultaneously. There is no restriction regarding multiple file associations referring to multiple servers or mixtures of server-based files and directly accessed files.

When using the Application Server (see App Server) a built-in file server is used for all programs running under a single server. That server will also serve other accesses from other JVM instances and/or machines if the configuration file allows it.