← services

Trace flow

99ba360a8893d1fcd789d11f35936e01 · live fetch 3016 ms · tilda-trial-automation

Waterfall

Request span + Neo4j queries by start offset (reconstructed from log timestamps — approximate)

readwrite

Queries in this trace 5

Query Offset Elapsed R/W

Query	Offset	R/W
`MATCH (n:WorkOrder {tildaid: $node_id}) RETURN n`	—	read
`MATCH (wo:WorkOrder {tildaid: $wo_id}) OPTIONAL MATCH (wo)-[:WORK_ORDER_TO_ACTIVITY]->(a:WorkOrderActivity {activity_type: 'U_REPLY'}) RETURN wo.createdat AS created_at, max(a.timestamp) AS latest_reply_ts`	—	read
`MATCH (w:WorkOrder {tildaid: $work_order_id})-[:WORK_ORDER_TO_ACTIVITY]->(a:WorkOrderActivity) WHERE 1=1 AND a.activity_type IN $activity_types AND a.status IN $statuses RETURN count(a) as total`	—	read
MATCH (w:WorkOrder {tildaid: $work_order_id})-[:WORK_ORDER_TO_ACTIVITY]->(a:WorkOrderActivity) WHERE 1=1 AND a.activity_type IN $activity_types AND a.status IN $statuses OPTIONAL MATCH (d:Document) WHERE d.tildaid IN COALESCE(a.docIds, []) WITH w, a, COLLECT(DISTINCT {tildaid: d.tildaid, fileName: d.fileName, contentType: d.contentType}) as documents // Compute latest AI_MSG draft id per work order for gating REGENERATE OPTIONAL MATCH (w)-[:WORK_ORDER_TO_ACTIVITY]->(ai:WorkOrderActivity) WHERE ai.activity_type = 'AI_MSG' // any status: a newer SENT/QUEUED must supersede an older FindingsChanged draft, else REGENERATE stays stuck WITH w, a, documents, max(coalesce(ai.updatedat, ai.createdat)) AS latest_any_ts, collect({id: ai.tildaid, ts: coalesce(ai.updatedat, ai.createdat), status: ai.status, ignoreType: ai.ignoreType}) AS ai_msgs WITH a, documents, CASE WHEN size(ai_msgs) = 0 THEN false ELSE any(x IN ai_msgs WHERE x.id = a.tildaid AND x.ts = latest_any_ts) END AS is_latest_draft RETURN a, CASE WHEN size(documents) > 0 AND documents[0].tildaid IS NOT NULL THEN documents ELSE [] END AS documents, is_latest_draft ORDER BY a.createdat DESC SKIP $skip LIMIT $limit	—	read
`MATCH (w:WorkOrder {tildaid: $work_order_id})<-[:FACT_TO_WORK_ORDER]-(f:Fact) WHERE f.delpending = false RETURN f ORDER BY f.createdat ASC`	—	read

MATCH (n:WorkOrder {tildaid: $node_id}) RETURN n

—

0 ms

read


        MATCH (wo:WorkOrder {tildaid: $wo_id})
        OPTIONAL MATCH (wo)-[:WORK_ORDER_TO_ACTIVITY]->(a:WorkOrderActivity {activity_type: 'U_REPLY'})
        RETURN wo.createdat AS created_at, max(a.timestamp) AS latest_reply_ts

—

0 ms

read


        MATCH (w:WorkOrder {tildaid: $work_order_id})-[:WORK_ORDER_TO_ACTIVITY]->(a:WorkOrderActivity)
        WHERE 1=1 AND a.activity_type IN $activity_types AND a.status IN $statuses 
        RETURN count(a) as total

—

0 ms

read


        MATCH (w:WorkOrder {tildaid: $work_order_id})-[:WORK_ORDER_TO_ACTIVITY]->(a:WorkOrderActivity)
        WHERE 1=1 AND a.activity_type IN $activity_types AND a.status IN $statuses 
        OPTIONAL MATCH (d:Document)
        WHERE d.tildaid IN COALESCE(a.docIds, [])
        WITH w, a, COLLECT(DISTINCT {tildaid: d.tildaid, fileName: d.fileName, contentType: d.contentType}) as documents
        // Compute latest AI_MSG draft id per work order for gating REGENERATE
        OPTIONAL MATCH (w)-[:WORK_ORDER_TO_ACTIVITY]->(ai:WorkOrderActivity)
        WHERE ai.activity_type = 'AI_MSG'  // any status: a newer SENT/QUEUED must supersede an older FindingsChanged draft, else REGENERATE stays stuck
        WITH w, a, documents,
             max(coalesce(ai.updatedat, ai.createdat)) AS latest_any_ts,
             collect({id: ai.tildaid, ts: coalesce(ai.updatedat, ai.createdat), status: ai.status, ignoreType: ai.ignoreType}) AS ai_msgs
        WITH a, documents,
             CASE
               WHEN size(ai_msgs) = 0 THEN false
               ELSE any(x IN ai_msgs WHERE x.id = a.tildaid AND x.ts = latest_any_ts)
             END AS is_latest_draft
        RETURN a,
               CASE WHEN size(documents) > 0 AND documents[0].tildaid IS NOT NULL THEN documents ELSE [] END AS documents,
               is_latest_draft
        ORDER BY a.createdat DESC
        SKIP $skip
        LIMIT $limit

—

0 ms

read


            MATCH (w:WorkOrder {tildaid: $work_order_id})<-[:FACT_TO_WORK_ORDER]-(f:Fact)
            WHERE f.delpending = false
            RETURN f
            ORDER BY f.createdat ASC

—

0 ms

read

Logs in this trace 10

1h ago DEBUG virtual-assistant ✅ Org 2d88eda5-bcfa-4565-af90-db111ea04aa3 summary: processed=1, reminders_sent=0, moved_to_need_attention=0

time2026-07-27 00:07:09 UTC

severityDEBUG

servicevirtual-assistant

sourcejobs/daily_work_order_cron.py:377

clustermaster

org2d88eda5-bcfa-4565-af90-db111ea04aa3

versionunknown

trace99ba360a8893d1fcd789d11f35936e01

span2b7f93492c031438

podvirtual-assistant-65b975d66b-l8wpw

message

✅ Org 2d88eda5-bcfa-4565-af90-db111ea04aa3 summary: processed=1, reminders_sent=0, moved_to_need_attention=0

raw payload (8 fields)

{
  "caller": "jobs/daily_work_order_cron.py:377",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "\u2705 Org 2d88eda5-bcfa-4565-af90-db111ea04aa3 summary: processed=1, reminders_sent=0, moved_to_need_attention=0",
  "org_alias": "2d88eda5-bcfa-4565-af90-db111ea04aa3",
  "service": "virtual-assistant",
  "timestamp": "2026-07-27T00:07:09.599Z",
  "version": "unknown"
}

1h ago DEBUG virtual-assistant [reminder] wo=442a7419-9334-4479-aa7e-c607afaee462 mode=RECURRING should_send=False escalate=False override=none reason='RECURRING not due:...

time2026-07-27 00:07:09 UTC

severityDEBUG

servicevirtual-assistant

sourcejobs/daily_work_order_cron.py:244

clustermaster

org2d88eda5-bcfa-4565-af90-db111ea04aa3

versionunknown

trace99ba360a8893d1fcd789d11f35936e01

span2b7f93492c031438

podvirtual-assistant-65b975d66b-l8wpw

message

[reminder] wo=442a7419-9334-4479-aa7e-c607afaee462 mode=RECURRING should_send=False escalate=False override=none reason='RECURRING not due: now-base=921m < step=1440m'

raw payload (8 fields)

{
  "caller": "jobs/daily_work_order_cron.py:244",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "[reminder] wo=442a7419-9334-4479-aa7e-c607afaee462 mode=RECURRING should_send=False escalate=False override=none reason=\u0027RECURRING not due: now-base=921m \u003c step=1440m\u0027",
  "org_alias": "2d88eda5-bcfa-4565-af90-db111ea04aa3",
  "service": "virtual-assistant",
  "timestamp": "2026-07-27T00:07:09.599Z",
  "version": "unknown"
}

1h ago INFO virtual-assistant MATCH (w:WorkOrder {tildaid: $work_order_id})<-[:FACT_TO_WORK_ORDER]-(f:Fact) WHERE f.delpending = false ...

time2026-07-27 00:07:09 UTC

severityINFO

servicevirtual-assistant

clustermaster

org2d88eda5-bcfa-4565-af90-db111ea04aa3

trace99ba360a8893d1fcd789d11f35936e01

span2b7f93492c031438

podserver-2-dev-0

query

MATCH (w:WorkOrder {tildaid: $work_order_id})<-[:FACT_TO_WORK_ORDER]-(f:Fact)
            WHERE f.delpending = false
            RETURN f
            ORDER BY f.createdat ASC

raw payload (15 fields)

{
  "allocatedBytes": 776.0,
  "cpu": 0.0,
  "elapsedTimeMs": 0.0,
  "executionPlanCacheKeyHash": "23A43F51",
  "id": "24641167",
  "pageFaults": 0.0,
  "pageHits": 7.0,
  "planning": 0.0,
  "query": "\n            MATCH (w:WorkOrder {tildaid: $work_order_id})\u003c-[:FACT_TO_WORK_ORDER]-(f:Fact)\n            WHERE f.delpending = false\n            RETURN f\n            ORDER BY f.createdat ASC\n            ",
  "queryParameters": "{work_order_id: \u0027442a7419-9334-4479-aa7e-c607afaee462\u0027}",
  "runtime": "pipelined",
  "source": "bolt-session\tbolt\tneo4j-python/5.28.0 Python/3.12.13-final-0 (linux)\t\tclient/10.129.4.156:38312\tserver/10.173.0.8:7687\u003e",
  "timestamp": "2026-07-27T00:07:09.594Z",
  "transactionId": 12025257.0,
  "waiting": 0.0
}

1h ago INFO virtual-assistant MATCH (w:WorkOrder {tildaid: $work_order_id})-[:WORK_ORDER_TO_ACTIVITY]->(a:WorkOrderActivity) WHERE 1=1 AND a.activity_type IN...

time2026-07-27 00:07:09 UTC

severityINFO

servicevirtual-assistant

clustermaster

org2d88eda5-bcfa-4565-af90-db111ea04aa3

trace99ba360a8893d1fcd789d11f35936e01

span2b7f93492c031438

podserver-2-dev-0

query

MATCH (w:WorkOrder {tildaid: $work_order_id})-[:WORK_ORDER_TO_ACTIVITY]->(a:WorkOrderActivity)
        WHERE 1=1 AND a.activity_type IN $activity_types AND a.status IN $statuses 
        OPTIONAL MATCH (d:Document)
        WHERE d.tildaid IN COALESCE(a.docIds, [])
        WITH w, a, COLLECT(DISTINCT {tildaid: d.tildaid, fileName: d.fileName, contentType: d.contentType}) as documents
        // Compute latest AI_MSG draft id per work order for gating REGENERATE
        OPTIONAL MATCH (w)-[:WORK_ORDER_TO_ACTIVITY]->(ai:WorkOrderActivity)
        WHERE ai.activity_type = 'AI_MSG'  // any status: a newer SENT/QUEUED must supersede an older FindingsChanged draft, else REGENERATE stays stuck
        WITH w, a, documents,
             max(coalesce(ai.updatedat, ai.createdat)) AS latest_any_ts,
             collect({id: ai.tildaid, ts: coalesce(ai.updatedat, ai.createdat), status: ai.status, ignoreType: ai.ignoreType}) AS ai_msgs
        WITH a, documents,
             CASE
               WHEN size(ai_msgs) = 0 THEN false
               ELSE any(x IN ai_msgs WHERE x.id = a.tildaid AND x.ts = latest_any_ts)
             END AS is_latest_draft
        RETURN a,
               CASE WHEN size(documents) > 0 AND documents[0].tildaid IS NOT NULL THEN documents ELSE [] END AS documents,
               is_latest_draft
        ORDER BY a.createdat DESC
        SKIP $skip
        LIMIT $limit

raw payload (15 fields)

{
  "allocatedBytes": 3272.0,
  "cpu": 0.0,
  "elapsedTimeMs": 0.0,
  "executionPlanCacheKeyHash": "2A13D7A9",
  "id": "24641166",
  "pageFaults": 0.0,
  "pageHits": 10.0,
  "planning": 0.0,
  "query": "\n        MATCH (w:WorkOrder {tildaid: $work_order_id})-[:WORK_ORDER_TO_ACTIVITY]-\u003e(a:WorkOrderActivity)\n        WHERE 1=1 AND a.activity_type IN $activity_types AND a.status IN $statuses \n        OPTIONAL MATCH (d:Document)\n        WHERE d.tildaid IN COALESCE(a.docIds, [])\n        WITH w, a, COLLECT(DISTINCT {tildaid: d.tildaid, fileName: d.fileName, contentType: d.contentType}) as documents\n        // Compute latest AI_MSG draft id per work order for gating REGENERATE\n        OPTIONAL MATCH (w)-[:WORK_ORDER_TO_ACTIVITY]-\u003e(ai:WorkOrderActivity)\n        WHERE ai.activity_type = \u0027AI_MSG\u0027  // any status: a newer SENT/QUEUED must supersede an older FindingsChanged draft, else REGENERATE stays stuck\n        WITH w, a, documents,\n             max(coalesce(ai.updatedat, ai.createdat)) AS latest_any_ts,\n             collect({id: ai.tildaid, ts: coalesce(ai.updatedat, ai.createdat), status: ai.status, ignoreType: ai.ignoreType}) AS ai_msgs\n        WITH a, documents,\n             CASE\n               WHEN size(ai_msgs) = 0 THEN false\n               ELSE any(x IN ai_msgs WHERE x.id = a.tildaid AND x.ts = latest_any_ts)\n             END AS is_latest_draft\n        RETURN a,\n               CASE WHEN size(documents) \u003e 0 AND documents[0].tildaid IS NOT NULL THEN documents ELSE [] END AS documents,\n               is_latest_draft\n        ORDER BY a.createdat DESC\n        SKIP $skip\n        LIMIT $limit\n        ",
  "queryParameters": "{limit: 1, statuses: [\u0027DRAFT\u0027], skip: 0, work_order_id: \u0027442a7419-9334-4479-aa7e-c607afaee462\u0027, activity_types: [\u0027AI_MSG\u0027]}",
  "runtime": "pipelined",
  "source": "bolt-session\tbolt\tneo4j-python/5.28.0 Python/3.12.13-final-0 (linux)\t\tclient/10.129.4.156:38312\tserver/10.173.0.8:7687\u003e",
  "timestamp": "2026-07-27T00:07:09.587Z",
  "transactionId": 12025256.0,
  "waiting": 0.0
}

1h ago INFO virtual-assistant MATCH (w:WorkOrder {tildaid: $work_order_id})-[:WORK_ORDER_TO_ACTIVITY]->(a:WorkOrderActivity) WHERE 1=1 AND a.activity_type IN...

time2026-07-27 00:07:09 UTC

severityINFO

servicevirtual-assistant

clustermaster

org2d88eda5-bcfa-4565-af90-db111ea04aa3

trace99ba360a8893d1fcd789d11f35936e01

span2b7f93492c031438

podserver-2-dev-0

query

MATCH (w:WorkOrder {tildaid: $work_order_id})-[:WORK_ORDER_TO_ACTIVITY]->(a:WorkOrderActivity)
        WHERE 1=1 AND a.activity_type IN $activity_types AND a.status IN $statuses 
        RETURN count(a) as total

raw payload (15 fields)

{
  "allocatedBytes": 488.0,
  "cpu": 0.0,
  "elapsedTimeMs": 0.0,
  "executionPlanCacheKeyHash": "FA66F084",
  "id": "24641165",
  "pageFaults": 0.0,
  "pageHits": 11.0,
  "planning": 0.0,
  "query": "\n        MATCH (w:WorkOrder {tildaid: $work_order_id})-[:WORK_ORDER_TO_ACTIVITY]-\u003e(a:WorkOrderActivity)\n        WHERE 1=1 AND a.activity_type IN $activity_types AND a.status IN $statuses \n        RETURN count(a) as total\n        ",
  "queryParameters": "{statuses: [\u0027DRAFT\u0027], work_order_id: \u0027442a7419-9334-4479-aa7e-c607afaee462\u0027, activity_types: [\u0027AI_MSG\u0027]}",
  "runtime": "pipelined",
  "source": "bolt-session\tbolt\tneo4j-python/5.28.0 Python/3.12.13-final-0 (linux)\t\tclient/10.129.4.156:38312\tserver/10.173.0.8:7687\u003e",
  "timestamp": "2026-07-27T00:07:09.583Z",
  "transactionId": 12025256.0,
  "waiting": 0.0
}

1h ago INFO virtual-assistant MATCH (wo:WorkOrder {tildaid: $wo_id}) OPTIONAL MATCH (wo)-[:WORK_ORDER_TO_ACTIVITY]->(a:WorkOrderActivity {activity_type:...

time2026-07-27 00:07:09 UTC

severityINFO

servicevirtual-assistant

clustermaster

org2d88eda5-bcfa-4565-af90-db111ea04aa3

trace99ba360a8893d1fcd789d11f35936e01

span2b7f93492c031438

podserver-2-dev-0

query

MATCH (wo:WorkOrder {tildaid: $wo_id})
        OPTIONAL MATCH (wo)-[:WORK_ORDER_TO_ACTIVITY]->(a:WorkOrderActivity {activity_type: 'U_REPLY'})
        RETURN wo.createdat AS created_at, max(a.timestamp) AS latest_reply_ts

raw payload (15 fields)

{
  "allocatedBytes": 928.0,
  "cpu": 0.0,
  "elapsedTimeMs": 0.0,
  "executionPlanCacheKeyHash": "4C51129A",
  "id": "24641164",
  "pageFaults": 0.0,
  "pageHits": 13.0,
  "planning": 0.0,
  "query": "\n        MATCH (wo:WorkOrder {tildaid: $wo_id})\n        OPTIONAL MATCH (wo)-[:WORK_ORDER_TO_ACTIVITY]-\u003e(a:WorkOrderActivity {activity_type: \u0027U_REPLY\u0027})\n        RETURN wo.createdat AS created_at, max(a.timestamp) AS latest_reply_ts\n        ",
  "queryParameters": "{wo_id: \u0027442a7419-9334-4479-aa7e-c607afaee462\u0027}",
  "runtime": "pipelined",
  "source": "bolt-session\tbolt\tneo4j-python/5.28.0 Python/3.12.13-final-0 (linux)\t\tclient/10.129.4.156:38312\tserver/10.173.0.8:7687\u003e",
  "timestamp": "2026-07-27T00:07:09.576Z",
  "transactionId": 12025255.0,
  "waiting": 0.0
}

1h ago INFO virtual-assistant MATCH (n:WorkOrder {tildaid: $node_id}) RETURN n

time2026-07-27 00:07:09 UTC

severityINFO

servicevirtual-assistant

clustermaster

org2d88eda5-bcfa-4565-af90-db111ea04aa3

trace99ba360a8893d1fcd789d11f35936e01

span2b7f93492c031438

podserver-2-dev-0

query

MATCH (n:WorkOrder {tildaid: $node_id}) 
RETURN n

raw payload (15 fields)

{
  "allocatedBytes": 312.0,
  "cpu": 0.0,
  "elapsedTimeMs": 0.0,
  "executionPlanCacheKeyHash": "E0146DD7",
  "id": "24641163",
  "pageFaults": 0.0,
  "pageHits": 10.0,
  "planning": 0.0,
  "query": "MATCH (n:WorkOrder {tildaid: $node_id}) RETURN n",
  "queryParameters": "{node_id: \u0027442a7419-9334-4479-aa7e-c607afaee462\u0027}",
  "runtime": "pipelined",
  "source": "bolt-session\tbolt\tneo4j-python/5.28.0 Python/3.12.13-final-0 (linux)\t\tclient/10.129.4.156:38312\tserver/10.173.0.8:7687\u003e",
  "timestamp": "2026-07-27T00:07:09.569Z",
  "transactionId": 12025254.0,
  "waiting": 0.0
}

1h ago DEBUG virtual-assistant [daily_cron] Starting processing for work order 442a7419-9334-4479-aa7e-c607afaee462

time2026-07-27 00:07:09 UTC

severityDEBUG

servicevirtual-assistant

sourcejobs/daily_work_order_cron.py:135

clustermaster

org2d88eda5-bcfa-4565-af90-db111ea04aa3

versionunknown

trace99ba360a8893d1fcd789d11f35936e01

span2b7f93492c031438

podvirtual-assistant-65b975d66b-l8wpw

message

[daily_cron] Starting processing for work order 442a7419-9334-4479-aa7e-c607afaee462

raw payload (8 fields)

{
  "caller": "jobs/daily_work_order_cron.py:135",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "[daily_cron] Starting processing for work order 442a7419-9334-4479-aa7e-c607afaee462",
  "org_alias": "2d88eda5-bcfa-4565-af90-db111ea04aa3",
  "service": "virtual-assistant",
  "timestamp": "2026-07-27T00:07:09.565Z",
  "version": "unknown"
}

1h ago DEBUG virtual-assistant [daily_cron] About to process work order 442a7419-9334-4479-aa7e-c607afaee462

time2026-07-27 00:07:09 UTC

severityDEBUG

servicevirtual-assistant

sourcejobs/daily_work_order_cron.py:368

clustermaster

org2d88eda5-bcfa-4565-af90-db111ea04aa3

versionunknown

trace99ba360a8893d1fcd789d11f35936e01

span2b7f93492c031438

podvirtual-assistant-65b975d66b-l8wpw

message

[daily_cron] About to process work order 442a7419-9334-4479-aa7e-c607afaee462

raw payload (8 fields)

{
  "caller": "jobs/daily_work_order_cron.py:368",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "[daily_cron] About to process work order 442a7419-9334-4479-aa7e-c607afaee462",
  "org_alias": "2d88eda5-bcfa-4565-af90-db111ea04aa3",
  "service": "virtual-assistant",
  "timestamp": "2026-07-27T00:07:09.565Z",
  "version": "unknown"
}

1h ago DEBUG virtual-assistant 🏢 Processing organization 2d88eda5-bcfa-4565-af90-db111ea04aa3 (1 work orders)

time2026-07-27 00:07:09 UTC

severityDEBUG

servicevirtual-assistant

sourcejobs/daily_work_order_cron.py:362

clustermaster

org2d88eda5-bcfa-4565-af90-db111ea04aa3

versionunknown

trace99ba360a8893d1fcd789d11f35936e01

span2b7f93492c031438

podvirtual-assistant-65b975d66b-l8wpw

message

🏢 Processing organization 2d88eda5-bcfa-4565-af90-db111ea04aa3 (1 work orders)

raw payload (8 fields)

{
  "caller": "jobs/daily_work_order_cron.py:362",
  "cluster": "master",
  "environment": "DEVELOPMENT",
  "message": "\ud83c\udfe2 Processing organization 2d88eda5-bcfa-4565-af90-db111ea04aa3 (1 work orders)",
  "org_alias": "2d88eda5-bcfa-4565-af90-db111ea04aa3",
  "service": "virtual-assistant",
  "timestamp": "2026-07-27T00:07:09.565Z",
  "version": "unknown"
}

Every log line sharing this trace id, across all services — fetched live from Cloud Logging (tilda-trial-automation), never stored. Open the Logs Explorer for the same view with field filters.