AI Common

Updated on 02 Jan 2024
2 Minutes to read

Print
Share
Dark
Light

Article summary

Did you find this summary helpful?

Thank you for your feedback

Module Details
Core or Github Module	Core
Requires Restart?	No
Steps Exposed	Yes
Settings Location	AI → Postgres AI → Text Manipulation

Overview

The AI Common Module is essential for utilizing various AI integrations within the platform. This is a shared module between other AI modules and provides necessary functionalities, tools, and interfaces as a foundational layer for integrating different AI services.

Available Steps

Installing the Module exposes the following steps found in the toolbox at AI → Postgres and AI → Text Manipulation.

Postgres

Step Name	Description
Ensure Postgres Vector Extension	This step configures the PostgreSQL database to support vector data storage. It allows setting table and column names to store vector data in PostgreSQL. This step ensures that the necessary extensions are in place for optimal vector data management.
Query Table with Vector Data	This step facilitates querying a PostgreSQL table with vector data. It takes as input the 'Vector Storage Info' and the specific vector to find, which includes a JSON Representation, ReferenceID, and TokenCount. This step is useful for retrieving relevant information based on vector data stored in the PostgreSQL database.
Save Vector Embedding to Postgres	This step allows the storage of vector embeddings in a PostgreSQL table. It takes the 'Vector Storage Info' and the list of vectors to store as input. This step is valuable for saving embeddings generated from various sources into the PostgreSQL database, providing a centralized location for vector data.

Text Manipulation

Step Name	Description
Escape JSON in Text	This step takes in text input containing JSON content and escapes special characters, ensuring proper JSON format. This is useful with textual data that includes JSON content to prevent parsing errors or unintended behavior.
Prepare Text for Embeddings	This step is intended for text preprocessing for natural language processing (NLP) tasks. It aims to optimize text data for embedding models.
Remove Control Characters From Text	This step helps to process text inputs and eliminate any control characters present in the text. Control characters are non-printable characters that can cause text processing, display, or interpretation issues. This step ensures that the output text is free from such characters, promoting cleaner and more readable text.
Remove HTML Tags From Text	This step removes HTML tags from the text, ensuring only plain text is left. This is important when dealing with content extracted from web pages or HTML-formatted text.
Split Long Text into Chunks	This step helps break long text into manageable chunks. Useful for data exceeding maximum size and processing limits. This is particularly helpful when dealing with text data that exceeds a specified maximum size, preventing issues related to processing limitations or model constraints.