Logging

Overview

Users deploying R2R into production settings benefit from robust, persistant logging. R2R supports this via Victorialogs, open source user-friendly database for logs from VictoriaMetrics.

Victorialogs ships by default with the full version of R2R and hosts a UI to view your logs at http://localhost:9428/select/vmui.

Accessing Logs

VictoriaLogs UI

The easiest way to view logs is through the VictoriaLogs UI:

1

Navigate to http://localhost:9428/select/vmui.

2

Use the query box to search for specific log entries.

3

Adjust the time range as needed using the time controls

Common Query Examples

Here are some useful queries for finding specific log information:

1
# View all logs
2
*
3
4
# View logs with [ERROR] tag 
5
{log=~"\\[ERROR\\].*"}
6
7
# View logs with error-related content
8
{log=~".*error.*"}
9
{log=~".*exception.*"}
10
{log=~".*traceback.*"}
11
{log=~".*failed.*"}
12
13
# View logs with warning content
14
{log=~".*WARNING.*"}
15
16
# View logs about a specific process
17
{log=~".*ingestion.*"}
18
19
# View specific error types
20
{log=~".*HTTPException.*"}
21
{log=~".*ValueError.*"}
22
23
# View Azure OpenAI-related errors
24
{log=~".*OpenAI.*"}

Troubleshooting Common Issues

No Logs Showing Up

If you don’t see any logs:

1. Increase the time range - logs might be outside your current time window
2. Check if Fluent Bit is running: docker ps | grep fluent-bit
3. Check VictoriaLogs is running: docker ps | grep victoria-logs
4. Verify your R2R container is properly configured for logging

Understanding Error Logs

When you see an error in the logs, it typically follows this pattern:

1. Error message with timestamp
2. A traceback showing the sequence of function calls
3. The specific error and its cause

Look for the actual error message at the bottom of a traceback to understand the root cause.

Advanced Configuration

Customizing Fluent Bit

If you need to customize how logs are collected and processed, you can modify the Fluent Bit configuration:

1. Create/edit the fluent-bit.conf file in your ./fluent-bit directory
2. Restart the Fluent Bit container: docker restart docker-fluent-bit-1

Setting Up Grafana for Log Visualization

For more advanced visualization, you can connect Grafana to VictoriaLogs:

1. Access Grafana at http://localhost:3001

2. Add a new VictoriaLogs data source:

   • Go to Configuration > Data Sources > Add data source
   • Select “VictoriaMetrics Logs”
   • Set URL to http://victoria-logs:9428
   • Save and test the connection

3. Create a new dashboard with a Logs panel

4. Configure the panel to query logs using the same query syntax as in the VictoriaLogs UI

Retention Policy

By default, logs are retained for 60 days as configured in the Docker Compose file:

1
victoria-logs:
2
  image: victoriametrics/victoria-logs:v1.10.1-victorialogs
3
  command: -storageDataPath=/data -retentionPeriod=60d

To change the retention period, modify the -retentionPeriod parameter and restart the container.

Log Format

Each log entry contains:

• _time: Timestamp of the log
• container_name: Source container
• log: The actual log message
• Additional metadata

When searching logs, you’ll typically want to search for content in the log field.