Ingestion

Document Ingestion and Management

Ingest Files

Ingest files or directories into your R2R system:

const files = [
  { path: 'path/to/file1.txt', name: 'file1.txt' },
  { path: 'path/to/file2.txt', name: 'file2.txt' }
];
const metadatas = [{ key1: 'value1' }, { key2: 'value2' }];

const ingestResponse = await client.ingestFiles(files, {
  metadatas,
  user_ids: ['user-id-1', 'user-id-2'],
});

response

object

The response from the R2R system after ingesting the files.

[{'message': 'Ingestion task queued successfully.', 'task_id': '6e27dfca-606d-422d-b73f-2d9e138661b4', 'document_id': 'c3291abf-8a4e-5d9d-80fd-232ef6fd8526'}, ...]

files

Array<string | File | { path: string; name: string }>

required

An array of file paths, File objects, or objects with path and name properties to ingest.

options

object

metadatas

Record<string, any>

An optional array of metadata objects corresponding to each file.

document_ids

Array<string>

An optional array of document IDs to assign to the ingested files.

user_ids

Array<string | null>

An optional array of user IDs associated with the ingested files.

chunking_config

Optional[Union[dict, ChunkingConfig]]

The chunking config override parameter enables developers to customize their R2R chunking strategy at runtime.

provider

str

default: "unstructured_local"

Which chunking provider to use. Options are “r2r”, “unstructured_local”, or “unstructured_api”.

method

str

default: "by_title"

Which chunking method to apply. Options are “by_title”, “basic”, “recursive”, or “character”.

chunk_size

int

default: "512"

The average size of chunks, in tokens.

chunk_overlap

int

default: "20"

The default overlap between chunks.

max_chunk_size

Optional[int]

default: "None"

Sets a maximum size on output chunks.

combine_under_n_chars

Optional[int]

Combine chunks smaller than this number of characters.

max_characters

Optional[int]

Maximum number of characters per chunk.

coordinates

bool

default: "False"

Whether to include coordinates in the output.

encoding

Optional[str]

Encoding to use for text files.

extract_image_block_types

Optional[list[str]]

Types of image blocks to extract.

gz_uncompressed_content_type

Optional[str]

Content type for uncompressed gzip files.

hi_res_model_name

Optional[str]

Name of the high-resolution model to use.

include_orig_elements

Optional[bool]

default: "False"

Whether to include original elements in the output.

include_page_breaks

bool

Whether to include page breaks in the output.

languages

Optional[list[str]]

List of languages to consider for text processing.

multipage_sections

bool

default: "True"

Whether to allow sections to span multiple pages.

new_after_n_chars

Optional[int]

Start a new chunk after this many characters.

ocr_languages

Optional[list[str]]

Languages to use for OCR.

output_format

str

default: "application/json"

Format of the output.

overlap

int

default: "0"

Number of characters to overlap between chunks.

overlap_all

bool

default: "False"

Whether to overlap all chunks.

pdf_infer_table_structure

bool

default: "True"

Whether to infer table structure in PDFs.

similarity_threshold

Optional[float]

Threshold for considering chunks similar.

skip_infer_table_types

Optional[list[str]]

Types of tables to skip inferring.

split_pdf_concurrency_level

int

default: "5"

Concurrency level for splitting PDFs.

split_pdf_page

bool

default: "True"

Whether to split PDFs by page.

starting_page_number

Optional[int]

Page number to start processing from.

strategy

str

default: "auto"

Strategy for processing. Options are “auto”, “fast”, or “hi_res”.

chunking_strategy

Optional[str]

default: "by_title"

Strategy for chunking. Options are “by_title” or “basic”.

unique_element_ids

bool

default: "False"

Whether to generate unique IDs for elements.

xml_keep_tags

bool

default: "False"

Whether to keep XML tags in the output.

Update Files

Update existing documents:

const files = [
  { path: '/path/to/updated_file1.txt', name: 'updated_file1.txt' }
];
const document_ids = ['document-id-1'];
const updateResponse = await client.updateFiles(files, {
  document_ids,
  metadatas: [{ key: 'updated_value' }] // to overwrite the existing metadata
});

response

object

The response from the R2R system after updating the files.

  {'results': {'processed_documents': [{'id': '9f375ce9-efe9-5b57-8bf2-a63dee5f3621', 'title': 'aristotle_v2.txt'}], 'failed_documents': [], 'skipped_documents': []}}

files

Array<File | { path: string; name: string }>

required

An array of File objects or objects with path and name properties to update.

options

object

required

document_ids

Array<string>

required

An array of document IDs corresponding to the files being updated.

metadatas

Array<Record<string, any>>

An optional array of metadata objects for the updated files.

chunking_config

Record<string, any>

The chunking config override parameter enables developers to customize their R2R chunking strategy at runtime.

Documents Overview

Retrieve high-level document information, restricted to user files, except when called by a superuser where it will then return results from over all users:

const documentsOverview = await client.documentsOverview();

response

Array<object>

An array of objects containing document information.

[
  {
    'document_id': '93123a68-d668-51de-8291-92162730dc87',
    'version': 'v1',
    'size_in_bytes': 73353,
    'metadata': {},
    'status': 'success',
    'user_id': '2acb499e-8428-543b-bd85-0d9098718220',
    'title': 'aristotle.txt',
    'created_at': '2024-07-21T20:09:14.218741Z',
    'updated_at': '2024-07-21T20:09:14.218741Z',
    'metadata': {'x': 'y'}
  },
  ...
]

document_ids

Array<string>

An optional array of document IDs to filter the overview.

Document Chunks

Fetch chunks for a particular document:

const documentId = '93123a68-d668-51de-8291-92162730dc87';
const chunks = await client.documentChunks(documentId);

response

Array<object>

An array of objects containing chunk information.

[{
  'text': 'Aristotle[A] (Greek: Ἀριστοτέλης Aristotélēs, pronounced [aristotélɛːs]; 384–322 BC) was an Ancient Greek philosopher and polymath...',
  'user_id': '2acb499e-8428-543b-bd85-0d9098718220',
  'document_id': 'db02076e-989a-59cd-98d5-e24e15a0bd27',
  'extraction_id': 'aeba6400-1bd0-5ee9-8925-04732d675434',
  'fragment_id': 'f48bcdad-4155-52a4-8c9d-8ba06e996ba3'
  'metadata': {'title': 'aristotle.txt', 'version': 'v0', 'chunk_order': 0}}
},
...]

document_id

string

required

The ID of the document to retrieve chunks for.

Delete Documents

Delete a document by its ID:

const deleteResponse = await client.delete({ document_id: "91662726-7271-51a5-a0ae-34818509e1fd" });

filters

{ [key: string]: string | string[] }

required

A list of logical filters to perform over input documents fields which identifies the unique set of documents to delete (e.g., {"document_id": {"$eq": "db02076e-989a-59cd-98d5-e24e15a0bd27"}}). Logical operations might include variables such as "user_id" or "title" and filters like neq, gte, etc.

Getting Started

Setup

Deployment

Deep Dives

Document Ingestion and Management

Ingest Files

Update Files

Documents Overview

Document Chunks

Delete Documents

Getting Started

Setup

Deployment

Deep Dives

​Document Ingestion and Management

​Ingest Files

​Update Files

​Documents Overview

​Document Chunks

​Delete Documents

Document Ingestion and Management

Ingest Files

Update Files

Documents Overview

Document Chunks

Delete Documents