File Storage Structure

Overview

The File Storage Structure has been updated to an indexed directory structure to avoid performance degradation as disk usage grows. Existing files will be moved to the new directory structure and attached to entities. Temporary files will be cleaned up, and files no longer being used will be archived. New files will be placed appropriately and attached to entities as they are used, and moved as needed.  

To learn more about Archiving and File Deletion, see Archive Entities and File Archiving and Deletion.

Indexed Directory Structure

A name-based directory structure is used for the File Storage. Files are stored based on the first two characters of their names. 

New Core Directory Structure

• <FileStorageLocation>\<InstanceName>
  • \archive
    • \files-to-check
    • \36 directories (0-9) and (a-z)
      • \36 directories (0-9) and (a-z)
  • \FlowData
    • \36 directories (0-9) and (a-z)
      • \36 directories (0-9) and (a-z)
  • \temp
    • \36 directories (0-9) and (a-z)
      • \36 directories (0-9) and (a-z)

Indexed paths by file type

Each file is stored in a two-level-indexed directory based on the first two characters of its filename. Each level includes 36 possible values: 0-9 and a-z.

There is a Boot Initializable Job named FileStorageDirectoryCreationJob that runs during Decisions Services startup and creates the new directory structure if it does not already exist. This job will not move any files.

Moving Old Files to New Directory Structure

Once the new directory structure is created, existing files can be safely moved to the appropriate directory, as defined above, based on their names. In addition to moving the files, files in use by In Flight and Completed Flows need to be marked as such. This is accomplished by adding a json.info file to each file that identifies its association. 

The movement and association of existing files will be handled by a Thread Job named FileStorageReorganizationJob. This job will run one minute after the FileStorageDirectoryCreationJob to ensure that the new directories are created before trying to move files into them. The Thread Job will not be added back to the queue to run again unless Decisions Services is restarted; at that time, it will run 1 minute after the restart.

It is important to note that the FileStorageReorganizationJob will ONLY process files if there are directories that exist that match any of the old file structures.

Info File Format Update (9.20+)

As of 9.20, file metadata is stored in a JSON-based info file named <FileId>-json.info. In versions prior to 9.20, the info file name included both the file ID and the original file name (for example, <FileId>_<OriginalFileName>.<ext>.info).

Existing files transition to the -json.info format when file storage jobs run and when files are accessed.

File and info file format prior to 9.20.

File and info file format as of 9.20.

Cleanup Temp Files and Move Files to Archive

At midnight each day, a job runs with the sole responsibility of cleaning up Temporary Files and moving Files associated with Completed Entities to an Archive Directory. This will be a Thread Job named FileStorageCleanupJob and, as indicated, will run at midnight daily.

The responsibilities and functions of the job are as follows.

• Read the json.info files for all files in
  • <FileStorageLocation>\<InstanceName>\temp
  • <FileStorageLocation>\<InstanceName>\FlowData
• For json.info files in <FileStorageLocation>\<InstanceName>\temp
  • Examine the <CleanUpDate> field and determine if the file can be deleted.
• For json.info files in <FileStorageLocation>\<InstanceName>\FlowData 
  • Examine the <EntityAssociation> fields
    • For each one, examine the <CompletedDate> field
  • If all <EntityAssociation> entries have a <CompletedDate>
    • Move the file to the appropriate directory
      • <FileStorageLocation>\<InstanceName>\archive\?\?
      • The file will be stored based on the file name, which in almost all cases will start with a ULID. Users updating from a version prior to v8.10 will see files using the older GUID format. For example, ULID: 01ARZ3NDEKTSV4RRFFQ69G5FAV.txt, GUID: 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
        • <FileStorageLocation>\<InstanceName>\archive\5\a
    • Update the <PathBeforeArchive> in the json.info file to provide traceability for the file's origin.

Feature Changes