Rule Tracing & Debugging

Overview

The Rule Tracing and Debugging feature provides a high-fidelity view of how data flows through your eKuiper rules. Unlike static logs, tracing allows you to inspect individual messages as they transition through different stages (nodes) of your stream processing logic, providing visibility into transformations, filter decisions, and sink deliveries.

Core Concepts

Trace Spans and Hierarchy

Every rule execution generates a trace composed of spans. A span represents a discrete operation within the rule—such as a source read, a SQL projection, or a sink write.

• Node-level granularity: Each operator in your SQL (e.g., WHERE, GROUP BY) corresponds to a span.
• Temporal sequence: View the exact timing of when a message entered a source and when the resulting action was triggered.

Message Attribute Inspection

Within each trace span, you can inspect Attributes. This is where the actual data resides.

• Input/Output Payloads: View the raw JSON or binary data before and after it was processed by a specific node.
• Metadata: Access system-level attributes like timestamps, offsets, and sequence numbers.
• Metric Extraction: The manager automatically identifies keys like active_power, temperature, or status within stringified JSON attributes.

Using the Trace Capture

The Manager provides a specialized interface to capture real-time data for specific rules.

Capturing Logs

To capture traces for debugging:

1. Select the rule(s) you wish to debug.
2. Set the Capture Duration (Level 0–4).
3. The system will programmatically start the trace, wait for the specified window, and collect the resulting span data.

Supported Capture Durations: | Level | Duration | Use Case | | :--- | :--- | :--- | | 0 | 5 Seconds | Quick health check for high-frequency streams. | | 1 | 10 Seconds | Default debugging window. | | 4 | 60 Seconds | Deep analysis for low-frequency or intermittent data. |

API Implementation Example

The trace capture logic is managed via the EKuiperClient. Below is the logical flow used to programmatically capture debugging data:

// Example: Capturing a rule trace programmatically
const client = new EKuiperClient(SERVER_URL);

// 1. Start the trace
await client.startRuleTrace(ruleId, "always");

// 2. Wait for data to accumulate (e.g., 10 seconds)
await new Promise(resolve => setTimeout(resolve, 10000));

// 3. Stop the trace and fetch IDs
await client.stopRuleTrace(ruleId);
const traceIds = await client.getRuleTraceIds(ruleId);

// 4. Fetch detailed span data for the most recent traces
const traceDetails = await Promise.all(
    traceIds.slice(-10).map(id => client.getTraceDetail(id))
);

AI-Powered Debugging & Analysis

The eKuiper Manager integrates AI to help interpret complex trace data. Instead of manually parsing JSON spans, you can use the built-in AI assistants:

Rule Analysis Assistant

Found within the Rule detail view, this assistant ingests the captured trace data to:

• Explain "Dropped" Messages: Identify why a message failed to pass a WHERE clause.
• Performance Troubleshooting: Highlight nodes causing latency or high CPU usage.
• Metric Summarization: Provide a plain-English summary of the values flowing through the rule (e.g., "The average temperature is holding steady at 72°F").

Master AI Chat

The system-wide assistant can correlate traces across multiple rules. If a shared source (like an MQTT broker) is failing, the Master AI can identify that multiple rules are experiencing "Source Timeouts" simultaneously based on the trace attributes.

Best Practices

• Selective Tracing: Tracing consumes eKuiper server resources. Only enable tracing for the duration of your debugging session.
• Check Exception Metrics: Before deep-diving into traces, check the Rule Metrics. If the exceptions count is rising, traces will likely contain specific error messages in the sink or operator spans.
• Filter for Logic Errors: Use Level 0 (5s) captures to verify if your SQL logic is correct (e.g., checking if a calculation node is outputting the expected value).