Integration Troubleshooting

This guide provides step-by-step instructions for diagnosing and resolving common integration, database connection, storage access, and webhook delivery issues.

---

1. Database Connection Failures

When configuring or running a Database Connector, you may encounter connectivity errors.

Common Error Messages & Resolutions

🔴 Connection Timeout (timeout=20s)

• Possible Cause: A network firewall or security group is blocking outbound traffic from the AIIA worker container to the target database server.
• Troubleshooting Steps:
  1. Verify the hostname and port are correct.
  2. Log in to the host machine running AIIA and test basic TCP connectivity:

     nc -zvw3 <target-host> <target-port>

  3. Ensure that your organization's network security team has whitelisted the outbound IP address of the AIIA worker node for the specific database port.

🔴 Connection Refused

• Possible Cause: The database service is not running on the target server, or it is not listening on the specified network interface.
• Troubleshooting Steps:
  1. Verify the database service status on the target host.
  2. Confirm the database configuration permits remote TCP/IP connections (e.g., check postgresql.conf's listen_addresses = '*' for PostgreSQL).

🔴 FATAL: password authentication failed for user "..."

• Possible Cause: Incorrect username or password configured in the connector.
• Troubleshooting Steps:
  1. Re-enter the password in the Edit Connector form.
  2. Verify that the user credentials work by logging in manually from another client using the same user.
  3. Ensure the database user has been granted SELECT privileges on the target schema tables.

---

2. Object Storage (S3 / MinIO) Failures

If evidence file uploads fail or S3 data refresh jobs throw errors:

Common Error Messages & Resolutions

🔴 AccessDenied (403): User is not authorized to perform: s3:GetObject

• Possible Cause: The AWS IAM policy or MinIO access policy associated with the API Key/Secret Key does not have sufficient bucket permissions.
• Troubleshooting Steps:
  1. Verify the IAM policy allows the following actions on the target bucket:

     {
       "Version": "2012-10-17",
       "Statement": [
         {
           "Effect": "Allow",
           "Action": [
             "s3:GetObject",
             "s3:PutObject",
             "s3:ListBucket"
           ],
           "Resource": [
             "arn:aws:s3:::your-bucket-name",
             "arn:aws:s3:::your-bucket-name/*"
           ]
         }
       ]
     }

🔴 SignatureDoesNotMatch

• Possible Cause: Incorrect API Key or Secret Key configured in the connector, or the system time on the AIIA worker host is out of sync.
• Troubleshooting Steps:
  1. Re-configure the access keys in the connector.
  2. Verify clock synchronization on the host machine using NTP:

     chronyc sources -v || ntpq -p

     [!IMPORTANT] S3 API requests will fail if the client clock deviates by more than 5 minutes from the S3 server clock.

---

3. Webhook Delivery Failures

If external systems are not receiving event notifications:

Common Error Messages & Resolutions

🔴 Status Code: 401 Unauthorized

• Possible Cause: The webhook signature verification is failing on the receiver side.
• Troubleshooting Steps:
  1. Verify that the secret key configured on the receiving server matches the signing secret displayed in the AIIA Admin → Webhooks panel.
  2. Check for clock skew issues. If the receiver rejects requests where t is more than 5 minutes old, ensure both servers sync via NTP.
  3. Print and inspect the computed signature on the receiver side to check for format differences.

🔴 Status Code: 403 Forbidden

• Possible Cause: The receiver's firewall or application gateway is blocking requests from the AIIA worker IP.
• Troubleshooting Steps:
  1. Check the receiver's security logs.
  2. Add the outbound IP of the AIIA worker node to the receiver's IP whitelist.

🔴 Connection Timeout / Host Unreachable

• Possible Cause: The receiver endpoint is offline, or the URL is misconfigured.
• Troubleshooting Steps:
  1. Verify the webhook URL is correct and starts with https:// (production) or http:// (internal testing).
  2. Ensure the target domain resolves correctly from the AIIA worker container.

---

4. Data Refresh Latency & Queue Backlog

If a data refresh job is stuck in the Pending status:

• Possible Cause: The Celery/worker queue is overloaded with long-running tasks, or the worker service is offline.
• Troubleshooting Steps:
  1. Check worker service status in Docker:

     docker compose ps worker

  2. Inspect worker logs for active processing or errors:

     docker compose logs worker --tail=100

  3. Check the active task queue depth using the Redis CLI:

     docker compose exec redis redis-cli LLEN celery

     If the depth is high, consider allocating additional worker instances or optimizing connector schedules.