AI Common
  • 02 Jan 2024
  • 2 Minutes to read
  • Dark
    Light

AI Common

  • Dark
    Light

Article summary

Module Details

Core or Github ModuleCore
Requires Restart?No
Steps ExposedYes
Settings Location
  • AI → Postgres
  • AI → Text Manipulation

Overview

The AI Common Module is essential for utilizing various AI integrations within the platform. This is a shared module between other AI modules and provides necessary functionalities, tools, and interfaces as a foundational layer for integrating different AI services. 




Available Steps

Installing the Module exposes the following steps found in the toolbox at AI → Postgres and AI → Text Manipulation.

Postgres

Step NameDescription
Ensure Postgres Vector ExtensionThis step configures the PostgreSQL database to support vector data storage. It allows setting table and column names to store vector data in PostgreSQL. This step ensures that the necessary extensions are in place for optimal vector data management.
Query Table with Vector DataThis step facilitates querying a PostgreSQL table with vector data. It takes as input the 'Vector Storage Info' and the specific vector to find, which includes a JSON Representation, ReferenceID, and TokenCount. This step is useful for retrieving relevant information based on vector data stored in the PostgreSQL database.
Save Vector Embedding to PostgresThis step allows the storage of vector embeddings in a PostgreSQL table. It takes the 'Vector Storage Info' and the list of vectors to store as input. This step is valuable for saving embeddings generated from various sources into the PostgreSQL database, providing a centralized location for vector data.

Text Manipulation

Step NameDescription
Escape JSON in TextThis step takes in text input containing JSON content and escapes special characters, ensuring proper JSON format. This is useful with textual data that includes JSON content to prevent parsing errors or unintended behavior.
Prepare Text for EmbeddingsThis step is intended for text preprocessing for natural language processing (NLP) tasks. It aims to optimize text data for embedding models.
Remove Control Characters From TextThis step helps to process text inputs and eliminate any control characters present in the text. Control characters are non-printable characters that can cause text processing, display, or interpretation issues. This step ensures that the output text is free from such characters, promoting cleaner and more readable text.
Remove HTML Tags From TextThis step removes HTML tags from the text, ensuring only plain text is left. This is important when dealing with content extracted from web pages or HTML-formatted text.
Split Long Text into ChunksThis step helps break long text into manageable chunks. Useful for data exceeding maximum size and processing limits. This is particularly helpful when dealing with text data that exceeds a specified maximum size, preventing issues related to processing limitations or model constraints.



Was this article helpful?