FormKiQ Enterprise Reference Guide
1. Introduction
FormKiQ Enterprise is made up of Enterprise Add-Ons that are available for FormKiQ Core, bringing enterprise-level features and functionality that may be required for some organizations.
NOTE: we have included the full Enterprise API Reference as a section below the FormKiQ Core API Reference, while including a link to the specific module APIs in the reference sections below.
FormKiQ Enterprise Add-On Modules include:
✅ Document Email Processing Module
✅ Fulltext Search Module
✅ Network Share and Sync Module
✅ Intelligent Document Processing Module
✅ Document Archive and Lifecycle Module
To learn more about FormKiQ Core, please visit https://github.com/formkiq/formkiq-core.
To learn more about FormKiQ Enterprise, please visit https://formkiq.com.
2. Tag Schema Module
✅ Enforce Required and Optional Tag Keys/Values
✅ Extend Search Functionality through Composite Keys for Tags
✅ Optionally Return Tags while using the POST /search API Endpoint
2.1. Overview
Tag Schema is a FormKiQ Enterprise Add-On Module (for the FormKiQ Core Headless Document Management System) that provides the enforcement of a tag schema on specific documents and enhances search functionality through composite keys for tags.
Use Cases:
-
If you have multiple input methods, such as having more than one application using the API, you can ensure that your documents are properly and consistently tagged no matter the source
-
If you expect your application(s) to search often using two tag keys, e.g., customerID and projectID, you can automate the creation and maintenance of composite keys to search in one request
With the Tag Schema Add-On Module, you can create different sets of required or optional keys, i.e., the Tag Schemas, and each schema can be associated to any number of documents. You can also specify both a default value and a list of allowed values for both required and optional keys.
Each tag schema can have up to 5 required and 10 optional keys, for a total of 15; you should consider prioritizing tags used for searches in cases where your document content may have more than five required key/value pairs, vs. attempting to store all document key/value pairs as tags. In cases where you expect to search for more than 15 keys, we recommend our Fulltext Search Module.
A tag schema can specify any of the following options as its rules:
-
only documents with the required tag keys can be accepted, with no optional keys allowed
-
only documents with the required tag keys can be accepted, with only the specified optional keys allowed
-
only documents with the required tag keys can be accepted, with any other keys being optional
-
documents have no required tag keys, but only the specified optional keys allowed
-
documents have no required or optional tag keys specified, and the schema is only used to create composite keys for searches (see below)
When a document is added with an associated tag schema, the document will be rejected if it does not conform to the rules of the schema.
When searching for documents, can you request that the tags in the schema are returned by using an optional boolean parameter, includeTagsFromSchema. Only documents with tag schemas will return their tags directly within search results. If a tag allows any number of optional tags (i.e., if allowAdditionalTags is not disabled in the schema), this optional parameter will only return the required tags. This is done to maintain a reasonable response size for these requests.
A tag schema also allows the creation of composite keys, i.e, compositeKeys, which allow for the searching of 2-3 tags at once. This allows FormKiQ to maintain its high scalability while remaining cost efficient.
There is no requirement that these composite keys be made up of required or specified optional tags, so you will want to ensure that you keep this in mind when creating any composite keys. Up to five separate composite keys can be created for a specific tag schema, while each composite key combines two or three tag keys. For more complex searches, we recommend our Fulltext Search Module.
NOTE: when using composite keys involving three tag keys, searching for these keys requires that all three tag keys are specified; if you require an additional search for only two of the three keys, you would need to create a separate composite key consisting of only those two tag keys.
2.2. API
The Tag Schema Module adds API endpoints to create, retrieve, and delete tag schemas. As this module interacts with the FormKiQ Core API Endpoints, a tagSchemaId will also be available on most document endpoints, such as POST /documents/.
You can view the full list of Tag Schema Module endpoints in the Enterprise API Reference.
3. Fulltext Search Module
✅ Extend Search Functionality, including multiple tag parameters and fulltext queries, through OpenSearch
✅ Optionally Include OCR Results in fulltext queries
3.1. Overview
Fulltext Search is a FormKiQ Enterprise Add-On Module (for the FormKiQ Core Headless Document Management System) that enhances built-in search functionality by integrating OpenSearch.
3.2. Installation
On your FormKiQ Enterprise page, you’ll find CloudFormation single-click installation links.
The Fulltext Search module uses the AWS OpenSearch service. The OpenSearch service can be inside of a VPC or outside of a VPC. FormKiQ provides methods for doing either type of installation.
If you are NOT using a VPC, you can skip the Install VPC section.
3.2.1. Install VPC (recommended)
To install OpenSearch inside of a VPC, the first step is to use the provided VPC Creation Script link to create a VPC preconfigure to work with FormKiQ.
The VPC CloudFormation script, will create a AWS VPC specifically designed to work with FormKiQ. Multiple Public / Private Subnets will be created across multiple availability zones to provide the best reliability possible.
You can customize the CIDR ranges to meet your network requirements.
NOTE: Remember the Stack Name you used to install the VPC, as you’ll need it in the next section.
3.2.2. Install OpenSearch
On your FormKiQ Enterprise page, you’ll find links to create OpenSearch using single-click installation links.
When configurating the OpenSearch WITH VPC, you’ll need to enter the VpcStackName which is the name of the VPC Stack you created above. The script will use outputs from the previous script in its configuration. You’ll also need to select the same private subnet CIDRs used during the creation of the VPC.
It is also important that your AppEnvironment match exactly the same string used in your FormKiQ installation.
NOTE: Remember the Stack Name you used to install OpenSearch, as you’ll need it in the next section.
3.2.3. Update FormKiQ
After installing OpenSearch, you’ll need to enable the OpenSearch module in FormKiQ. This is done by selecting your FormKiQ Stack from the CloudFormation console and click the Update Stack.
On the CloudFormation configuration page, enter the Stack Names of the OpenSearch installation and optionally the VPC installation.
CloudFormation will then update FormKiQ installation to enable OpenSearch.
3.3. API
The Fulltext Search Module adds several new endpoints, including the following API endpoint to the Documents section of the API:
-
PUT /documents/{DocumentId}/fulltext- Replaces the specified document’s fulltext metadata (if exists)
NOTE: all FormKiQ API endpoints are case-sensitive
The module also enables a new JSON body format for POST /search, allowing for fulltext and multi-clause queries on existing document metadata, including tags and OCR results.
You can view the full list of Fulltext Search Module endpoints in the Enterprise API Reference.
4. Custom Domains Module
✅ Customize your FormKiQ API and Console URLs with your domain
4.1. Overview
Custom Domains is a FormKiQ Enterprise Add-On Module (for the FormKiQ Core Headless Document Management System) that enables custom domains support for FormKiQ console, and API endpoints.
FormKiQ out of the box uses the AWS default domains for the Amazon CloudFront (XXXXXXX.cloudfront.net.) and Amazon API Gateway (https://XXXXXXX.execute-api.region.amazonaws.com)
The module uses Amazon Route 53 and AWS Certificate Manager to create domains and auto-renewing SSL certificates. Then adds the certificates and configures the domains to API Gateway and the FormKiQ console.
Use Cases:
-
If you would like to share the FormKiQ API or FormKiQ Console internally to users, and would prefer that one of your domains or subdomains be used instead of the less-readable CloudFront and API Gateway domains
-
If you are exposing any FormKiQ URLs externally and would prefer that external users remain unaware that you use AWS for your document management
5. Antivirus Module
✅ Automatically scan every newly added document
✅ Automatically tag document with scan status and timestamp
5.1. Overview
Antivirus is a module for the FormKiQ Headless Document Management System that provides document security by detecting trojans, viruses, malware & other malicious threats for every newly created document, using the ClamAV antivirus engine.
Using FormKiQ’s Document Events, each newly-created document is automatically added to a ClamAV Scanning SQS queue. An AWS Lambda function for ClamAV virus scanning pulls each document from the SQS queue and scans it for trojans, viruses, malware, and other malicious threats. The results from the scan are added as S3 Tags to the document, and are also attached to the document in DynamoDB. In addition, a message is posted to an ClamAV Scan Events SNS topic which allows other systems to be notified of the document’s scan status.
Use Cases:
-
If you have received certain compliance goals that require antivirus scanning of all documents used by the organization, this module can meet those goals without any additional effort
-
If you receive documents from external sources, whether through the FormKiQ API’s public endpoints or optional Enterprise Add-On Modules such as Document Email Processing, antivirus scanning will help to safeguard both your documents within FormKiQ and any documents you export from FormKiQ
6. Document OCR Module
✅ Use Amazon Textract to extract text and structured table or form data from documents
6.1. Overview
Document OCR is a FormKiQ Enterprise Add-On Module (for the FormKiQ Core Headless Document Management System) that provides optical character recognition (OCR) for the extraction of text and structured table or form data from documents.
The module uses Amazon Textract as its OCR engine. Amazon Textract uses advanced machine learning (ML) algorithms to identify, understand, and extract text and data from documents, including from forms and tables.
Use Case:
-
If you would like to add metadata for document retrieval, this module can provide that metadata from scanned documents; you can then create tags (in the case of form or table data) or enable the entire OCR result to be searched using the Fulltext Search module
6.2. API
The Document OCR Module adds endpoints to request a document OCR processing, to retrieve the result, and to delete.
You can view the full list of Document OCR Module endpoints in the Enterprise API Reference.
7. Full Encryption Module
Full Encryption is a FormKiQ Enterprise Add-On Module (for the FormKiQ Core Headless Document Management System) that enables both in-transit and at-rest encryption for the AWS services used by FormKiQ.
The module uses the AWS Key Management Service (AWS KMS) service to create and manage cryptographic keys and control the use of the keys. AWS KMS integrates with all of the services used by FormKiQ:
As an option, FormKiQ provides AWS KMS integration with AWS CloudTrail, which delivers log files to your designated Amazon S3 bucket. By using CloudTrail, you can monitor and investigate how and when your KMS keys have been used and by which service or user. This integration if turned off by default, to avoid unexpected CloudTrail costs.
Use Case:
-
If you have received certain compliance goals that require encryption of all documents used by the organization, this module can meet those goals without any additional effort
8. Single Sign-On and Custom JWT Authorizer Module
Single Sign-On and Custom JWT Authorizer is a FormKiQ Enterprise Add-On Module (for the FormKiQ Core Headless Document Management System) that enables the ability to customize the authorizer used for FormKiQ console, and API endpoints.
FormKiQ by default uses Amazon Cognito as the JWT authorizer.
Using this module the authorizer can be changed to another authorizer such as:
Use Case:
-
If you would like to maintain your ActiveDirectory or Google Apps login from your organization into the FormKiQ API or Console, this module allows that with minimal configuration
8.1. Integration Instructions - Microsoft Active Directory Federation Services
8.1.1. Retrieve Your Amazon Cognito Info
In order to configure your Windows Server AD FS, you will need to retrieve your FormKiQ Pool ID and Amazon Cognito Domain.
Go to Amazon Cognito in your AWS Management Console. Under "Settings", you can retrieve your User Pool ID:
You can then retrieve your Cognito Domain under "App Integration >> Domain Name":
8.1.2. Open AD FS Management in Windows Server Manager
NOTE: Ensure that
-
AD FS server(s) has https network access to
amazoncognito.com -
When using multiple AD FS servers, apply the following changes to all servers
Go to "Server Manager" and select "AD FS":
Choose the "Tools" menu, and select "AD FS Management":
8.1.3. Adding Relying Party Trust Management
Select "Relying Party Trusts" and choose the "Add Relying Party Trust Wizard". Choose the default "Claims aware" option:
For "Select Data Source", choose the option "Enter data about the relying party manually":
For "Specify Display Name", enter a descriptive name, such as "FormKiQ ADFS Login":
For "Configure URL", choose "Enable support for the SAML 2.0 WebSSO protocol", and for the form field below, replace "yourDomainPrefix" with your Amazon Cognito User Pool’s Domain Prefix (retrieved as the first step), and replace region with the User Pool’s AWS Region (for example, "us-east-1"):
For "Configure Identifiers", you need to supply the "Relying party trust identifier". Enter "urn:amazon:cognito:sp:*yourUserPoolID*" as the URN, but with your Cognito User Pool (e.g., "us-east-1_g2zxiEbac"), instead of "yourUserPoolId":
You can now click through the default options for the remainder of the Wizard and click "Finish".
8.1.4. Editing the Relying Party Trust Claim Issuance Policy
You should now see your new Relying Party Trust listed with the Display Name that you provided. You can select this trust:
Choose "Edit Claim Issuance Policy":
You can now add a new Claim Issuance Policy Rule for the Windows account name:
Next, create a rule for the email:
8.1.5. Signing in with AD FS
You can test out your AD sign in with the FormKiQ Console:
You will then be redirected to an Active Directory server and prompted to choose your Corporate ID:
You can then securely sign in with your Active Directory credentials:
8.1.6. User Group Access
FormKiQ access is determined by the group(s) the user belongs to, see Multi-Tenant
The only difference is all user groups need to be prefixed with formkiq_.
At a minimum you need to create a group called formkiq_default which will give users read / write access to the default siteId.
8.2. OpenSearch Dashboards/Kibana using SAML Authentication
SAML authentication for OpenSearch Dashboards lets you use your existing identity provider to offer single sign-on (SSO) for Dashboards/Kibana on Amazon OpenSearch Service.
Start by visiting the AWS OpenSearch Console and selecting the domain to change the authentication source.
Select the Security configuration tab and then the Edit button.
First, Disable Cognito Authentication
Next, Enable SAML authentication
After enabling SAML authentication, you are provided the following configuration urls:
-
Service provider entity ID
-
IdP-initiated SSO URL
-
SP-initiated SSO URL
Using the above urls to create a new application with SAML support in your IdP. To configure authentication through your IdP’s application directory, use the IdP-initiated SSO URL. To configure authentication through the OpenSearch Dashboards/Kibana URL, use the SP-initiated SSO URL.
The following link provides additional information on configuring SAML authentication for OpenSearch Dashboards.
9. FileSync CLI Module
FileSync CLI is a FormKiQ Enterprise Add-On Module (for the FormKiQ Core Headless Document Management System) that enables syncing of documents from a local file system to FormKiQ installation.
9.1. CloudFormation installation
The FileSync CLI needs to have permissions configured to give it access to FormKiQ. This can be done via the CloudFormation script found in the FileSync CLI .zip release file.
After installation on the if you look at the Outputs tab of the CloudFormation Stack you’ll get the Access/Secret Key the CLI needs to use.
9.2. Parameters
The FileSync CLI top level parameters are shown below.
usage: fk
--configure configure AWS credentials
--sync sync files with FormKiQ
--watch watch directorie(s) for file changes and syncs with
FormKiQ9.3. Configure
The configure parameter allows configuring the AWS credentials for the CLI.
usage: fk --configure
--access-key <arg> AWS Access Key
--endpoint-url <arg> Override command's default URL with the given
URL
--region <arg> The AWS region FormKiQ is installed in
--secret-key <arg> AWS Secret KeyUsing the Access/Secret Key created in the section above. Use those values to configure the CLI tool.
fk --configure --access-key <ACCESS_KEY> --secret-key <SECRET_KEY> --region <AWS_REGION>9.4. Sync / Watch
The CLI can be used used sync a directory with FormKiQ or to watch a directory for changes and sync those changes with FormKiQ.
The required parameters are:
Argument |
Description |
|
The FormKiQ’s AppEnvironment configured during installation |
|
Directory to sync/watch changes |
|
virtual folder name/path to organize documents within FormKiQ |
usage: fk --sync / --watch
-a,--appenvironment <arg> FormKiQ AppEnvironment
--actions <arg> Actions to perform on file (OCR / FULLTEXT)
-d,--dir <arg> Directory to watch for changes
--dry-run show what would have been transferred
--max-file-queue maximum number of files to queue (default:
500)
--mtime <arg> file modification time subtracted from the
initialization time (IE: --mtime -5 find
files modified last 5 minutes to sync)
-r,--recursive recurse into directories
--siteId <arg> FormKiQ Site Id to Sync files to
-v,--verbose increase verbosity
--virtualfolder <arg> A virtual folder name/path to organize
documents within FormKiQ9.5. Usage
The following are example commands of fk usage.
Basic Directory Syncing
fk --sync -a dev --virtualfolder windowsShare -d /documents --verboseSync files modified in the last 24 hours
fk --sync -a dev --virtualfolder windowsShare -d /documents --verbose --mtime 0Sync files modified more than 7 days ago
fk --sync -a dev --virtualfolder windowsShare -d /documents --verbose --mtime 7Sync files modified in the last 30 days
fk --sync -a dev --virtualfolder windowsShare -d /documents --verbose --mtime -30Sync the /documents directory and performs the OCR & Fulltext actions to all documents
fk --sync -a dev --virtualfolder windowsShare --actions ocr,fulltext -d /documents --verboseBasic Directory Watching
fk --watch -a dev --virtualfolder windowsShare -d /documents --verbose10. Appendix
Please check out the website for further links, and to contact.