R2R Service Configuration Troubleshooting Guide: Postgres, Hatchet

This guide addresses common configuration problems for Postgres, and Hatchet services in R2R deployments.

Postgres Configuration Issues

1. Connection Failures

Symptom: R2R cannot connect to Postgres database.

Possible Causes and Solutions:

a) Incorrect connection string:

• Verify the DATABASE_URL environment variable.
• Ensure it follows the format: postgres://user:password@host:port/dbname

b) Network issues:

• Check if Postgres is running and accessible from the R2R container.
• Verify network settings in Docker Compose file.

c) Authentication problems:

• Confirm that the username and password in the connection string are correct.
• Check Postgres logs for failed authentication attempts.

2. pgvector Extension Missing

Symptom: Vector operations fail or R2R reports missing pgvector functionality.

Solution:

• Ensure you’re using the pgvector/pgvector image instead of the standard Postgres image.
• Verify that the pgvector extension is created in your database:

  1
  CREATE EXTENSION IF NOT EXISTS vector;

3. Insufficient Resources

Symptom: Postgres performance issues or crashes under load.

Solution:

• Adjust resource limits in Docker Compose:

  1
  postgres:
  2
    deploy:
  3
      resources:
  4
        limits:
  5
          cpus: '2'
  6
          memory: 4G

• Tune Postgres configuration parameters like shared_buffers, effective_cache_size, etc.

Hatchet Configuration Issues

1. RabbitMQ Connection Problems

Symptom: Hatchet cannot connect to RabbitMQ.

Solution:

• Verify RabbitMQ connection string:

  1
  environment:
  2
    - SERVER_TASKQUEUE_RABBITMQ_URL=amqp://user:password@hatchet-rabbitmq:5672/

• Ensure RabbitMQ service is healthy and accessible.

2. Hatchet API Key Issues

Symptom: R2R cannot authenticate with Hatchet.

Solution:

• Check the Hatchet API key generation process:

  1
  setup-token:
  2
    command: >
  3
      sh -c "
  4
        TOKEN=$(/hatchet/hatchet-admin token create --config /hatchet/config --tenant-id your-tenant-id)
  5
        echo $TOKEN > /hatchet_api_key/api_key.txt
  6
      "

• Verify that R2R is correctly reading the API key:

  1
  r2r:
  2
    environment:
  3
      - HATCHET_CLIENT_TOKEN=${HATCHET_CLIENT_TOKEN}
  4
    command: >
  5
      sh -c '
  6
        export HATCHET_CLIENT_TOKEN=$(cat /hatchet_api_key/api_key.txt)
  7
        exec your_r2r_command
  8
      '

3. Hatchet Engine Health Check Failures

Symptom: Hatchet Engine service fails health checks.

Solution:

• Verify Hatchet Engine configuration:

  1
  hatchet-engine:
  2
    environment:
  3
      - SERVER_GRPC_BROADCAST_ADDRESS=host.docker.internal:7077
  4
      - SERVER_GRPC_BIND_ADDRESS=0.0.0.0
  5
    healthcheck:
  6
      test: ["CMD", "curl", "-f", "http://localhost:7077/health"]
  7
      interval: 10s
  8
      timeout: 5s
  9
      retries: 5

• Check Hatchet Engine logs for startup errors.
• Ensure all required environment variables are set correctly.

General Troubleshooting Tips

1. Check Logs: Always start by examining the logs of the problematic service:

   docker-compose logs postgres
   docker-compose logs hatchet-engine

2. Verify Network Connectivity: Ensure services can communicate:

   docker-compose exec r2r ping postgres
   docker-compose exec r2r ping hatchet-engine

3. Check Resource Usage: Monitor CPU and memory usage:

   docker stats

4. Recreate Containers: Sometimes, recreating containers can resolve issues:

   docker-compose up -d --force-recreate <service-name>

5. Verify Volumes: Ensure data persistence by checking volume mounts:

   docker volume ls
   docker volume inspect <volume-name>

6. Environment Variables: Double-check all environment variables in your .env file and docker-compose.yml.

By following this guide, you should be able to diagnose and resolve most configuration issues related to Postgres, and Hatchet in your R2R deployment. If problems persist, consider seeking help from the R2R community or support channels.