File Storage Structure

Overview

The File Storage Structure has been updated to an indexed directory structure to not hinder performance as the disk usage grows. Existing files will be moved to the new directory structure and attached to entities. Temporary files will be cleaned up and file no longer being used will be archived. New files will be placed appropriately and attached to entities as they become used, and move as needed.  

Indexed Directory Structure

A named-based directory structure is used for the File Storage. Files are stored based on the first two characters of the file name. 

New Core Directory Structure

• <FileStorageLocation>\<InstanceName>
  • \archive
    • \files-to-check
    • \36 directories (0-9) and (a-z)
      • \36 directories (0-9) and (a-z)
  • \FlowData
    • \36 directories (0-9) and (a-z)
      • \36 directories (0-9) and (a-z)
  • \temp
    • \36 directories (0-9) and (a-z)
      • \36 directories (0-9) and (a-z)

New Temporary Files

• <FileStorageLocation>\<InstanceName>\temp\?\?
• The file will be stored based on the file name, which in almost all cases will start with a GUID. For example 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
  • <FileStorageLocation>\<InstanceName>\temp\5\a

New Files Associated with Flows

• <FileStorageLocation>\<InstanceName>\FlowData\?\?
• The file will be stored based on the file name, which in almost all cases will start with a GUID. For example 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
  • <FileStorageLocation>\<InstanceName>\FlowData\5\a

New Files Associated with Assignments

• <FileStorageLocation>\<InstanceName>\FlowData\?\?
• The file will be stored based on the file name, which in almost all cases will start with a GUID. For example 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
  • <FileStorageLocation>\<InstanceName>\FlowData\5\a

New Files Associated with Entities

• <FileStorageLocation>\<InstanceName>\FlowData\?\?
• The file will be stored based on the file name, which in almost all cases will start with a GUID. For example 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
  • <FileStorageLocation>\<InstanceName>\FlowData\5\a

New Archived Files

• <FileStorageLocation>\<InstanceName>\archive\?\?
• <FileStorageLocation>\<InstanceName>\archive\files-to-check
  • This directory will hold files that we cannot find any entity associations for.
• The file will be stored based on the file name, which in almost all cases will start with a GUID. For example 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
  • <FileStorageLocation>\<InstanceName>\archive\5\a

New Persistent Files

This directory was added to handle files that will not have a .info file and will not be cleaned up. The File Storage Cleanup Job and Archive Entity Jobs ignore all files under the persistent directory.

• <FileStorageLocation>\<InstanceName>\persistent\?\?
• The file will be stored based on the file name, which in almost all cases will start with a GUID. For example 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
  • <FileStorageLocation>\<InstanceName>\persistent\5\a

There is a Boot Initializable Job named FileStorageDirectoryCreationJob which will run on Decisions Services startup and create this new directory structure if they do not already exist. This job will not move any files.

JavaScript Files

In v8.19 and higher JavaScript files are located below. In prior versions to v8.19 in MT environments renaming a .js file in one Tenant would rename that file in all Tenants.

• <FileStorageLocation>\JavaScriptControlFiles
• For MT Environments the Tenants the File Storage is: <FileStorageLocation>\<InstanceName>\JavascriptControlFiles

Moving Old Files to New Directory Structure

Once the new directory structure is created, existing files can be safely moved into the appropriate directory as defined above based on the name. In addition to moving the files, files in use by In Flight and Completed Flows need to be marked as such. This move is accomplished through the addition of a .info file, which will accompany every file and identify what the file is associated with. 

The movement and association of existing files will be handled by a Thread Job named FileStorageReorganizationJob. This job will run 1 minute after the FileStorageDirectoryCreationJob to ensure that the new directories are created before trying to move files into them. The Thread Job will not get added back to the queue to be run again unless Decisions Services is restarted, at which time, it will run one minute after the restart.

It is important to note that the FileStorageReorganizationJob will ONLY process files if there are directories that exist that match any of the old file structures.

Temporary Files

• Existing Temporary Files
  • <FileStorageLocation>\dd-mm-YYYY
  • <FileStorageLocation>\<InstanceName\dd-mm-YYYY
• Location To Be Moved To
  • <FileStorageLocation>\<InstanceName>\temp\?\?
  • The file will be stored based on the file name, which in almost all cases will start with a GUID. For example: 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
    • <FileStorageLocation>\<InstanceName>\temp\5\a
  • If one is not present, a .info file will be generated for the file with an expiration date of 1 day. This will be detailed further later in the document.

Files Associated with Flows

• Existing Files Associated with Flows
  • <FileStorageLocation>\FlowData\dd-mm-YYYY
  • <FileStorageLocation>\<InstanceName\FlowData\dd-mm-YYYY
• Location To Be Moved To
  • <FileStorageLocation>\<InstanceName>\FlowData\?\?
  • The file will be stored based on the file name, which in almost all cases will start with a GUID. For example: 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
    • <FileStorageLocation>\<InstanceName>\FlowData\5\a
  • If one is not present, a .info file will be generated for the file with the current Entities, Flows, and Assignments the file is associated with. This will be detailed further later in the document.

Files Associated with Assignments

• Existing Files Associated with Assignments
  • <FileStorageLocation>\FlowData\dd-mm-YYYY
  • <FileStorageLocation>\<InstanceName\FlowData\dd-mm-YYYY
• Location To Be Moved To
  • <FileStorageLocation>\<InstanceName>\FlowData\?\?
  • The file will be stored based on the file name, which in almost all cases will start with a GUID. For example 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
    • <FileStorageLocation>\<InstanceName>\FlowData\5\a
  • If one is not present, a .info file will be generated for the file with the current Entities, Flows, and Assignments the file is associated with. This will be detailed further later in the document.

Files Associated with Entities

• Existing Files Associated with Entities
  • <FileStorageLocation>\dd-mm-YYYY
  • <FileStorageLocation>\<InstanceName\temp\dd-mm-YYYY
  • <FileStorageLocation>\<InstanceName\FlowData\dd-mm-YYYY
• Location To Be Moved To
  • <FileStorageLocation>\<InstanceName>\temp\?\?
  • <FileStorageLocation>\<InstanceName>\FlowData\?\?
  • The file will be stored based on the file name, which in almost all cases will start with a GUID. For example 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
    • <FileStorageLocation>\<InstanceName>\temp\5\a
    • <FileStorageLocation>\<InstanceName>\FlowData\5\a
  • If one is not present, a .info file will be generated for the file with the current Entities, Flows, and Assignments the file is associated with. This will be detailed further later in the document.

Archived Files

• Existing Archived Files
  • <FileStorageLocation>\<InstanceName\archive\dd-mm-YYYY
  • <FileStorageLocation>\<InstanceName\archive\dd-mm-YYYY\files-to-check
  • Files present in this directory are files that were previously processed and we could not determine if they were connected to anything, so they were placed in this directory to be manually deleted later in order to avoid losing important files.
• Location To Be Moved To
  • <FileStorageLocation>\<InstanceName>\archive\?\?
  • <FileStorageLocation>\<InstanceName>\archive\files-to-check
    • All files stored in any files-to-check-directory will be moved into this single files-to-check directory.
  • The file will be stored based on the file name, which in almost all cases will start with a GUID. For example 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
    • <FileStorageLocation>\<InstanceName>\archive\5\a
  • These files will already have a .info file associated with them as they have already been processed in a previous version.

Cleanup Temp Files and Move Files to Archive

At midnight daily, a job will run with the sole responsibility of cleaning up Temporary Files and moving Files associated with Completed Entities to an Archive Directory. This will be a Thread Job named FileStorageCleanupJob and as indicated, will run at midnight daily.

The responsibilities and functions of the job are as follows.

• Read the .info files for all files in
  • <FileStorageLocation>\<InstanceName>\temp
  • <FileStorageLocation>\<InstanceName>\FlowData
• For .info files in <FileStorageLocation>\<InstanceName>\temp
  • Examine the <CleanUpDate> field and determine if the file can be deleted.
• For .info files in <FileStorageLocation>\<InstanceName>\FlowData 
  • Examine the <EntityAssociation> fields
    • For each one, examine the <CompletedDate> field
  • If all <EntityAssociation> entries have a <CompletedDate>
    • Move the file to the appropriate directory
      • <FileStorageLocation>\<InstanceName>\archive\?\?
      • The file will be stored based on the file name, which in almost all cases will start with a GUID. For example 5a4154e9-c0ce-49bb-83ed-1db7524c8d61_test.txt
        • <FileStorageLocation>\<InstanceName>\archive\5\a
    • Update the <PathBeforeArchive> in the .info file so we have traceability into where the file came from.

File Handling

Decisions controls newly created files from the beginning of their lifecycle, a file will be moved between directories as its state changes, rather than duplicated in multiple locations. The .info file will also be updated as the file state changes to reflect new entity associations. The entity associations will not be overwritten, they will be updated or added to.

Temp Files

There are several places where Temporary Files are created within Decisions. Previously, these files were duplicated any time any of the below actions were executed. These files are reused or rewritten if they have been deleted.

• Any time a File is added to a Form via a File Upload Control.
• Any time a document is uploaded.
• Any time a File is defined as a Constant in a Flow.
• Any time a Flow Step using a File Constant is clicked on and the Step Properties are shown.
• Any time an Icon or Image is used in the Folder Look and Feel
• Any time an Icon or Image is used on a Step
• Any time an Icon or Image is used on any Image Form Control (Image Controls, Button Controls, etc.)

These files will be created with <CleanupDate> in the .info file for 1 day. If they are accessed within that window, the <CleanupDate> will be updated.

The FileStorageCleanupJob will delete these files if it runs after the <CleanupDate>. 

If the File is attempted to be accessed again and has been deleted, it will be rewritten.

Files Associated with Flows

Any time a file becomes part of Flow State Data, it is considered to be associated with that Flow. This means that the file in question must live alongside the Flow for the duration of the Flow’s life cycle. There are several places in Decisions where a File will become associated with a Flow.

• Any time a File is added to a Form via a File Upload Control and then subsequently used in the Flow.
• Any time a Step with a File Constant is defined and used in a Flow.
• Any time a File is used in a Flow in ANY way.

If these files were previously Temporary Files, the Temporary File will get moved into the appropriate directory under FlowData, its .info file will have a new <EntityAssociation> added, and its <CleanupDate> will be set to null.

The .info file will also contain the date and time on which the file was associated with the entity in the <AssociatedDate> field.

As mentioned previously, the FileStorageCleanupJob will examine these .info files to see if the Flow has been completed before acting on the file. If all Flows associated with this file have been completed, the file will be moved into the appropriate archive directory.

Files Associated with Assignments

Any time a file becomes part of a Form Assignment, it is considered to be associated with that Assignment in addition to the Flow it’s already been associated with. This means that the file in question must live alongside the Flow and Assignment for the duration of both the Flow’s and Assignment’s life cycles. There are several places in Decisions where a File will become associated with an Assignment.

• Any time a File is added to an Assigned Form via a File Upload Control.
• Any time a Step with a File Constant is defined and used on an Assigned Form in a Flow.
• Any time a File is used in an Assigned Form in a Flow in ANY way.

If these files were previously Temporary Files, the Temporary File will get moved into the appropriate directory under FlowData, its .info file will have a new <EntityAssociation> added, and its <CleanupDate> will be set to null.

Files Associated with Entities

Data Structures, Flow Execution Extensions, etc. can be created with Properties of Type File Reference. Any time a new Entity is created that contains a File Reference Property as part of a flow, the file must be associated with the Entity. 

If these files were previously Temporary Files, the Temporary File will get moved into the appropriate directory under FlowData, its .info file will have a new <EntityAssociation> added, and its <CleanupDate> will be set to null.

The .info file will also contain the date and time on which the file was associated with the entity in the <AssociatedDate> field.

As mentioned previously, the FileStorageCleanupJob will examine these .info files to see if the Flow has been completed and the Entity has been deleted before acting on the file. If all Flows have been completed and all Entities have been deleted, the file will be moved into the appropriate archive directory and the <PathBeforeArchive> field will be set in the .info file.

Files in the Persistent Directory

These files will not have an associated .info file because Decisions does not know what entity to attach them to and they are not temporary files. These files will be ignored by the File Cleanup Job and Archive Entity Job.

File Archiving

In order to free up space on the disk, the existing Archive Job (ArchiveEntityJob) can be configured to include the deletion of Files from the <FileStorageLocation>\<InstanceName>\archive\?\? directory structure. This job is currently implemented to run every 24 hours.

1. Navigate to System > Settings and click Archive Settings.
   
   
   

   Setting		Description
   File Archive Settings
   	Archive Files	If set to true, files will be moved from the FlowData directory to the Archive directory. (if false it will not go through the FlowData directory)
   Default Archive Days
   	Days	Allows the user to set the number of days to archive
   Archive Settings For Types
   	Archive Settings For Types	Allow user to configure the archive settings for a specified type

2. Click ADD under the Archive Settings For Types field.
   
3. On the Add Archive Settings For Types window, select DecisionsFramework.Design.Flow.Service.Execution.FlowStateStorageData under Type Name and enter "30" under Days as the number of days the Flow State Storage Data should be stored before it’s deleted (its corresponding files will also be deleted).
   
4. Click OK then click SAVE.

Feature Change