Data Refresh Jobs
Data Refresh Jobs represent individual sync operations executed manually or automatically by the AIIA scheduler to pull fresh records from configured connectors.
Every time a connector runs, a Data Refresh Job (JobRun) is instantiated to track the execution steps, record counts, execution duration, and log any database or API connection errors.
Sync Execution Flow
The sync process is executed asynchronously via background workers (Redis + Worker service):
Monitoring Job Runs
Administrators can monitor the status of all data refresh activities in the Admin → Connectors → Job History section.
Job Statuses
Refresh jobs cycle through the standard execution statuses:
| Status Badge | Description | Required Action |
|---|---|---|
| Pending | Job is queued in Redis, waiting for an available background worker slot. | None. Usually transitions in < 5 seconds. |
| Running | The worker is actively fetching and processing data from the external source. | None. You can monitor the live record count. |
| Completed | Job finished successfully; raw data was parsed, matched, and committed. | None. Displays total ingested records. |
| Failed | A network, authentication, or protocol failure blocked the run. | Check credential configuration, verify firewall ports, and click Retry. |
| Error | Connection succeeded, but data validation failed (e.g., mismatching column type). | Inspect the exception traceback, correct mappings, and re-run. |
| Cancelled | The job was terminated manually by an administrator during execution. | Re-dispatch if needed. |
Manual Execution vs. Scheduled Runs
Scheduled Syncs
If a connector is configured with a CRON schedule, the scheduler daemon automatically pushes a new job to the worker queue at the specified interval.
- Example:
0 2 * * *(Runs every night at 2:00 AM, pulling transaction data from SAP).
Triggering Manual Syncs
To force an immediate sync (e.g., after updating database credentials or during an active audit field investigation):
- Navigate to Admin → Connectors.
- Locate the target connector in the list.
- Click the Sync Now (
sync) icon button. - The system will queue a job, and the status badge will transition to Pending → Running.
Log Analysis & Troubleshooting
For every failed or error-state job, AIIA provides a detailed log log viewer:
[2026-05-24 02:00:03] INFO: Starting data refresh job run #14802 (SAP ERP DB)
[2026-05-24 02:00:04] INFO: Connecting to postgresql://sap_read@10.140.2.14:5432/production...
[2026-05-24 02:00:25] ERROR: Connection timeout. Unable to reach 10.140.2.14:5432 after 20 seconds.
[2026-05-24 02:00:25] FATAL: Job aborted with state FAILED.
Common Failures & Resolutions
| Issue | Root Cause | Resolution |
|---|---|---|
| Connection Timeout | Firewall blocking traffic between AIIA worker and target system. | Allow outbound connections from AIIA worker IP to target host/port. |
| Invalid Credentials | Credentials changed on target database or expired S3 access keys. | Update configuration settings in the connector and run a Connection Test. |
| Column Mismatch | Schema migration on the external DB changed expected types or names. | Update the connector mappings or adjust the custom SQL query configured. |
| Out of Memory (OOM) | Target query returned too many rows without chunking. | Optimize the custom SQL query or configure partition limits. |