Xray · see through your fleet
1h 6h 24h 48h 3d
custom ▾
live

Logs

Live Cloud Logging query (nothing stored) · master · last 6h · newest first · page 1 (200/page) · 200 entries in 4457 ms all app neo4j weaviate time relative absolute

reset
15s ago2026-07-27 02:14:51 UTC DEBUG docproc Pulling messages
time2026-07-27 02:14:51 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:14:51.168Z",
  "version": "0.1.0"
}
15s ago2026-07-27 02:14:51 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:14:51 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:14:51.020Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
16s ago2026-07-27 02:14:50 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:14:50 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:14:50.945Z",
  "version": "0.1.0"
}
16s ago2026-07-27 02:14:50 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:14:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:14:50.166Z",
  "version": "0.1.0"
}
16s ago2026-07-27 02:14:50 UTC DEBUG docproc Recovery fill
time2026-07-27 02:14:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:14:50.166Z",
  "version": "0.1.0"
}
31s ago2026-07-27 02:14:35 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:14:35 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:14:35.487Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
31s ago2026-07-27 02:14:35 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:14:35 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:14:35.405Z",
  "version": "0.1.0"
}
35s ago2026-07-27 02:14:31 UTC DEBUG docproc Pulling messages
time2026-07-27 02:14:31 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:14:31.630Z",
  "version": "0.1.0"
}
36s ago2026-07-27 02:14:30 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:14:30 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:14:30.626Z",
  "version": "0.1.0"
}
36s ago2026-07-27 02:14:30 UTC DEBUG docproc Recovery fill
time2026-07-27 02:14:30 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:14:30.626Z",
  "version": "0.1.0"
}
47s ago2026-07-27 02:14:19 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:14:19 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:14:19.919Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
47s ago2026-07-27 02:14:19 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:14:19 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:14:19.848Z",
  "version": "0.1.0"
}
55s ago2026-07-27 02:14:11 UTC DEBUG docproc Pulling messages
time2026-07-27 02:14:11 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:14:11.102Z",
  "version": "0.1.0"
}
56s ago2026-07-27 02:14:10 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:14:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:14:10.097Z",
  "version": "0.1.0"
}
56s ago2026-07-27 02:14:10 UTC DEBUG docproc Recovery fill
time2026-07-27 02:14:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:14:10.097Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:14:04 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:14:04 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:14:04.395Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
1m ago2026-07-27 02:14:04 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:14:04 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:14:04.314Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:51 UTC DEBUG docproc Pulling messages
time2026-07-27 02:13:51 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:13:51.554Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:50 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:13:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:13:50.552Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:50 UTC DEBUG docproc Recovery fill
time2026-07-27 02:13:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:13:50.551Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:48 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:13:48 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:13:48.795Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:48 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:13:48 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:13:48.723Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:33 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:13:33 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:13:33.283Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:33 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:13:33 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:13:33.210Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:32 UTC DEBUG docproc Pulling messages
time2026-07-27 02:13:32 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:13:32.053Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:31 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:13:31 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:13:31.049Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:31 UTC DEBUG docproc Recovery fill
time2026-07-27 02:13:31 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:13:31.048Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:17 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:13:17 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:13:17.781Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:17 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:13:17 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:13:17.700Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:12 UTC DEBUG docproc Pulling messages
time2026-07-27 02:13:12 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:13:12.507Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:11 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:13:11 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:13:11.502Z",
  "version": "0.1.0"
}
1m ago2026-07-27 02:13:11 UTC DEBUG docproc Recovery fill
time2026-07-27 02:13:11 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:13:11.502Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:13:02 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:13:02 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:13:02.247Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
2m ago2026-07-27 02:13:02 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:13:02 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:13:02.178Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:52 UTC DEBUG docproc Pulling messages
time2026-07-27 02:12:52 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:12:52.030Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:50 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:12:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:12:50.996Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:50 UTC DEBUG docproc Recovery fill
time2026-07-27 02:12:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:12:50.996Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:46 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:12:46 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:12:46.708Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:46 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:12:46 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:12:46.602Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:31 UTC DEBUG docproc Pulling messages
time2026-07-27 02:12:31 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:12:31.583Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:31 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:12:31 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:12:31.156Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:31 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:12:31 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:12:31.088Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:30 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:12:30 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:12:30.581Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:30 UTC DEBUG docproc Recovery fill
time2026-07-27 02:12:30 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:12:30.580Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:15 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:12:15 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:12:15.551Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:15 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:12:15 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:12:15.478Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:11 UTC DEBUG docproc Pulling messages
time2026-07-27 02:12:11 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:12:11.418Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:10 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:12:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:12:10.413Z",
  "version": "0.1.0"
}
2m ago2026-07-27 02:12:10 UTC DEBUG docproc Recovery fill
time2026-07-27 02:12:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:12:10.413Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:59 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:11:59 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:11:59.962Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:59 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:11:59 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:11:59.888Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:51 UTC DEBUG docproc Pulling messages
time2026-07-27 02:11:51 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:11:51.908Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:50 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:11:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:11:50.904Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:50 UTC DEBUG docproc Recovery fill
time2026-07-27 02:11:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:11:50.904Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:44 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:11:44 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:11:44.429Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:44 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:11:44 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:11:44.354Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:31 UTC DEBUG docproc Pulling messages
time2026-07-27 02:11:31 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:11:31.355Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:30 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:11:30 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:11:30.352Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:30 UTC DEBUG docproc Recovery fill
time2026-07-27 02:11:30 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:11:30.352Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:28 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:11:28 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:11:28.848Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:28 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:11:28 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:11:28.772Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:12 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:11:12 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:11:12.994Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:12 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:11:12 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:11:12.921Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:10 UTC DEBUG docproc Pulling messages
time2026-07-27 02:11:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:11:10.835Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:09 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:11:09 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:11:09.831Z",
  "version": "0.1.0"
}
3m ago2026-07-27 02:11:09 UTC DEBUG docproc Recovery fill
time2026-07-27 02:11:09 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:11:09.831Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:57 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:10:57 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:10:57.512Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:57 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:10:57 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:10:57.436Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:50 UTC DEBUG docproc Pulling messages
time2026-07-27 02:10:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:10:50.332Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:49 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:10:49 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:10:49.329Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:49 UTC DEBUG docproc Recovery fill
time2026-07-27 02:10:49 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:10:49.328Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:41 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:10:41 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:10:41.938Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:41 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:10:41 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:10:41.868Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:30 UTC DEBUG docproc Pulling messages
time2026-07-27 02:10:30 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:10:30.784Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:29 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:10:29 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:10:29.745Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:29 UTC DEBUG docproc Recovery fill
time2026-07-27 02:10:29 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:10:29.745Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:26 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:10:26 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:10:26.415Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:26 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:10:26 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:10:26.328Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:10 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:10:10 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:10:10.876Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:10 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:10:10 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:10:10.808Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:10 UTC DEBUG docproc Pulling messages
time2026-07-27 02:10:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:10:10.251Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:09 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:10:09 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:10:09.246Z",
  "version": "0.1.0"
}
4m ago2026-07-27 02:10:09 UTC DEBUG docproc Recovery fill
time2026-07-27 02:10:09 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:10:09.246Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:55 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:09:55 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:09:55.371Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:55 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:09:55 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:09:55.296Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:49 UTC DEBUG docproc Pulling messages
time2026-07-27 02:09:49 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:09:49.693Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:48 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:09:48 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:09:48.688Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:48 UTC DEBUG docproc Recovery fill
time2026-07-27 02:09:48 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:09:48.688Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:39 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:09:39 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:09:39.779Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:39 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:09:39 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:09:39.655Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:29 UTC DEBUG docproc Pulling messages
time2026-07-27 02:09:29 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:09:29.651Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:28 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:09:28 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:09:28.647Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:28 UTC DEBUG docproc Recovery fill
time2026-07-27 02:09:28 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:09:28.647Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:24 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:09:24 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:09:24.220Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:24 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:09:24 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:09:24.150Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:10 UTC DEBUG docproc Pulling messages
time2026-07-27 02:09:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:09:10.122Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:09 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:09:09 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:09:09.120Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:09 UTC DEBUG docproc Recovery fill
time2026-07-27 02:09:09 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:09:09.120Z",
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:08 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:09:08 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:09:08.695Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
5m ago2026-07-27 02:09:08 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:09:08 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:09:08.623Z",
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:53 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:08:53 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:08:53.206Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:53 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:08:53 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:08:53.134Z",
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:50 UTC DEBUG docproc Pulling messages
time2026-07-27 02:08:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:08:50.103Z",
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:49 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:08:49 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:08:49.099Z",
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:49 UTC DEBUG docproc Recovery fill
time2026-07-27 02:08:49 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:08:49.098Z",
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:37 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:08:37 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:08:37.695Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:37 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:08:37 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:08:37.629Z",
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:30 UTC DEBUG docproc Pulling messages
time2026-07-27 02:08:30 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:08:30.055Z",
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:29 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:08:29 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:08:29.050Z",
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:29 UTC DEBUG docproc Recovery fill
time2026-07-27 02:08:29 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:08:29.050Z",
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:22 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:08:22 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:08:22.173Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:22 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:08:22 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:08:22.106Z",
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:10 UTC DEBUG docproc Pulling messages
time2026-07-27 02:08:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:08:10.568Z",
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:09 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:08:09 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:08:09.565Z",
  "version": "0.1.0"
}
6m ago2026-07-27 02:08:09 UTC DEBUG docproc Recovery fill
time2026-07-27 02:08:09 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:08:09.565Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:08:06 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:08:06 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:08:06.680Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
7m ago2026-07-27 02:08:06 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:08:06 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:08:06.601Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:51 UTC DEBUG docproc Pulling messages
time2026-07-27 02:07:51 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:07:51.055Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:50 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:07:50 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:07:50.798Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:50 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:07:50 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:07:50.716Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:50 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:07:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:07:50.052Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:50 UTC DEBUG docproc Recovery fill
time2026-07-27 02:07:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:07:50.052Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:35 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:07:35 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:07:35.235Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:35 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:07:35 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:07:35.122Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:31 UTC DEBUG docproc Pulling messages
time2026-07-27 02:07:31 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:07:31.471Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:30 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:07:30 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:07:30.466Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:30 UTC DEBUG docproc Recovery fill
time2026-07-27 02:07:30 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:07:30.466Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:19 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:07:19 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:07:19.669Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:19 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:07:19 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:07:19.598Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:11 UTC DEBUG docproc Pulling messages
time2026-07-27 02:07:11 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:07:11.949Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:10 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:07:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:07:10.945Z",
  "version": "0.1.0"
}
7m ago2026-07-27 02:07:10 UTC DEBUG docproc Recovery fill
time2026-07-27 02:07:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:07:10.945Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:07:04 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:07:04 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:07:04.133Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
8m ago2026-07-27 02:07:04 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:07:04 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:07:04.046Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:52 UTC DEBUG docproc Pulling messages
time2026-07-27 02:06:52 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:06:52.390Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:51 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:06:51 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:06:51.388Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:51 UTC DEBUG docproc Recovery fill
time2026-07-27 02:06:51 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:06:51.387Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:48 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:06:48 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:06:48.564Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:48 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:06:48 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:06:48.493Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:33 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:06:33 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:06:33.032Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:32 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:06:32 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:06:32.953Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:32 UTC DEBUG docproc Pulling messages
time2026-07-27 02:06:32 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:06:32.880Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:31 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:06:31 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:06:31.878Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:31 UTC DEBUG docproc Recovery fill
time2026-07-27 02:06:31 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:06:31.878Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:17 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:06:17 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:06:17.244Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:17 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:06:17 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:06:17.161Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:12 UTC DEBUG docproc Pulling messages
time2026-07-27 02:06:12 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:06:12.337Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:11 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:06:11 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:06:11.333Z",
  "version": "0.1.0"
}
8m ago2026-07-27 02:06:11 UTC DEBUG docproc Recovery fill
time2026-07-27 02:06:11 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:06:11.333Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:06:01 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:06:01 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:06:01.655Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
9m ago2026-07-27 02:06:01 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:06:01 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:06:01.579Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:52 UTC DEBUG docproc Pulling messages
time2026-07-27 02:05:52 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:05:52.825Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:51 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:05:51 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:05:51.821Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:51 UTC DEBUG docproc Recovery fill
time2026-07-27 02:05:51 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:05:51.821Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:46 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:05:46 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:05:46.102Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:46 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:05:46 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:05:46.009Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:32 UTC DEBUG docproc Pulling messages
time2026-07-27 02:05:32 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:05:32.287Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:31 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:05:31 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:05:31.284Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:31 UTC DEBUG docproc Recovery fill
time2026-07-27 02:05:31 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:05:31.284Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:30 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:05:30 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:05:30.538Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:30 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:05:30 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:05:30.458Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:15 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:05:15 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:05:15.039Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:14 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:05:14 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:05:14.958Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:11 UTC DEBUG docproc Pulling messages
time2026-07-27 02:05:11 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:05:11.753Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:10 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:05:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:05:10.747Z",
  "version": "0.1.0"
}
9m ago2026-07-27 02:05:10 UTC DEBUG docproc Recovery fill
time2026-07-27 02:05:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:05:10.747Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:59 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:04:59 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:04:59.464Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:59 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:04:59 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:04:59.384Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:51 UTC DEBUG docproc Pulling messages
time2026-07-27 02:04:51 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:04:51.239Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:50 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:04:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:04:50.234Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:50 UTC DEBUG docproc Recovery fill
time2026-07-27 02:04:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:04:50.234Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:43 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:04:43 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:04:43.844Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:43 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:04:43 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:04:43.766Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:30 UTC DEBUG docproc Pulling messages
time2026-07-27 02:04:30 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:04:30.681Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:29 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:04:29 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:04:29.679Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:29 UTC DEBUG docproc Recovery fill
time2026-07-27 02:04:29 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:04:29.679Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:28 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:04:28 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:04:28.298Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:28 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:04:28 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:04:28.219Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:12 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:04:12 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:04:12.804Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:12 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:04:12 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:04:12.726Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:11 UTC DEBUG docproc Pulling messages
time2026-07-27 02:04:11 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:04:11.172Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:10 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:04:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:04:10.168Z",
  "version": "0.1.0"
}
10m ago2026-07-27 02:04:10 UTC DEBUG docproc Recovery fill
time2026-07-27 02:04:10 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:04:10.168Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:57 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:03:57 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:03:57.328Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:57 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:03:57 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:03:57.241Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:50 UTC DEBUG docproc Pulling messages
time2026-07-27 02:03:50 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:03:50.636Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:49 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:03:49 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:03:49.632Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:49 UTC DEBUG docproc Recovery fill
time2026-07-27 02:03:49 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:03:49.632Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:41 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:03:41 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:03:41.773Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:41 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:03:41 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:03:41.693Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:30 UTC DEBUG docproc Pulling messages
time2026-07-27 02:03:30 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:03:30.094Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:29 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:03:29 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:03:29.091Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:29 UTC DEBUG docproc Recovery fill
time2026-07-27 02:03:29 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:03:29.091Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:25 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:03:25 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:03:25.083Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:25 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:03:25 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:03:25.012Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:09 UTC DEBUG docproc Pulling messages
time2026-07-27 02:03:09 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:240
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pulling messages
raw payload (11 fields)
{
  "caller": "handlers/pubsub_handler.py:240",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "max_messages": 1.0,
  "message": "Pulling messages",
  "pull_timeout_s": 60.0,
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:03:09.563Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:09 UTC DEBUG docproc Reported pending workload
time2026-07-27 02:03:09 UTC
severityDEBUG
servicedocproc
sourcejobs/hpa_pending_workload.py:176
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Reported pending workload
raw payload (15 fields)
{
  "backlog_capped": 0.0,
  "caller": "jobs/hpa_pending_workload.py:176",
  "cluster": "master",
  "component": "hpa_pending_workload",
  "environment": "DEVELOPMENT",
  "in_progress_count": 0.0,
  "message": "Reported pending workload",
  "num_replicas": 1.0,
  "recovery_workload": 0.0,
  "service": "docproc",
  "signal": 0.0,
  "timestamp": "2026-07-27T02:03:09.546Z",
  "total_workload": 0.0,
  "unacked": 0.0,
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:09 UTC DEBUG docproc Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
time2026-07-27 02:03:09 UTC
severityDEBUG
servicedocproc
sourcejobs/cloud_monitoring_metrics.py:296
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0
raw payload (7 fields)
{
  "caller": "jobs/cloud_monitoring_metrics.py:296",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "Pub/Sub num_unacked result: project_id=tilda-trial-automation subscription_id_in_filter=tilda-document-master-docproc-sub num_series=1 total=0",
  "service": "docproc",
  "timestamp": "2026-07-27T02:03:09.472Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:08 UTC DEBUG docproc Pull returned 0 messages
time2026-07-27 02:03:08 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:284
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Pull returned 0 messages
raw payload (9 fields)
{
  "caller": "handlers/pubsub_handler.py:284",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Pull returned 0 messages",
  "service": "docproc",
  "subscription_name": "tilda-document-master-docproc-sub",
  "timestamp": "2026-07-27T02:03:08.561Z",
  "version": "0.1.0"
}
11m ago2026-07-27 02:03:08 UTC DEBUG docproc Recovery fill
time2026-07-27 02:03:08 UTC
severityDEBUG
servicedocproc
sourcehandlers/pubsub_handler.py:429
clustermaster
version0.1.0
poddocproc-7bc76669-ljh8h
message
Recovery fill
raw payload (13 fields)
{
  "adopt_list_len": 0.0,
  "adopt_started": 0.0,
  "backlog_list_len": 0.0,
  "backlog_started": 0.0,
  "caller": "handlers/pubsub_handler.py:429",
  "cluster": "master",
  "component": "pubsub",
  "environment": "DEVELOPMENT",
  "message": "Recovery fill",
  "remaining_slots": 10.0,
  "service": "docproc",
  "timestamp": "2026-07-27T02:03:08.561Z",
  "version": "0.1.0"
}

Paginated with Cloud Logging page cursors, so you can walk the whole 6h window — not just the newest page. Click any filter by value to scope the view (resets to page 1). Paste a trace id to see every log line across all services for that one request.