Schema Management

Schema Management

Schema Management in eKuiper Manager allows you to define and enforce data contracts for your edge streams. By using schemas, you ensure that incoming data conforms to expected formats, which improves data integrity and enables high-performance serialization formats like Protobuf.

Overview

eKuiper Manager provides a centralized interface to manage schemas stored on the eKuiper server. These schemas are primarily used by Streams and Tables to:

• Validate incoming message structures.
• Map binary data (e.g., Protobuf) to readable SQL fields.
• Provide metadata for the AI Assistant to generate more accurate rules.

Protobuf Schema Management

Protocol Buffers (Protobuf) are widely used in IoT for efficient, low-bandwidth communication. eKuiper Manager supports the full lifecycle of Protobuf schemas.

Registering a Protobuf Schema

To use Protobuf, you must first upload the .proto file to the server.

1. Navigate to Schemas > Protobuf.
2. Click Create Schema.
3. Define a Schema ID (this will be used in your Stream definition).
4. Paste your .proto definition or upload the file.

Example Protobuf Definition:

syntax = "proto3";
package sensors;

message TemperatureReading {
  string sensorId = 1;
  float value = 2;
  int64 timestamp = 3;
}

Using Schemas in Streams

Once a schema is registered, you can associate it with a stream during the creation process using the SCHEMAID and FORMAT properties.

SQL Usage Example

When creating a stream via the SQL Editor, reference your schema as follows:

CREATE STREAM sensor_stream () 
WITH (
  FORMAT = "protobuf", 
  SCHEMAID = "sensors.TemperatureReading",
  TYPE = "mqtt",
  DATASOURCE = "factory/sensor/+"
);

Custom JSON Schemas

For JSON data, you can define schemas to enforce type safety (e.g., ensuring a temperature field is always a float rather than a string).

| Feature | Description | | :--- | :--- | | Strict Validation | Toggle whether eKuiper should drop messages that do not match the schema. | | Type Inference | Automatically generate stream fields based on an uploaded JSON schema. |

AI-Assisted Schema Generation

The eKuiper Manager includes an AI Stream Generator (located in the Stream Management section) that can help you draft schemas based on natural language descriptions.

How it works:

1. You provide a prompt: "I have an MQTT topic for a vibration sensor that sends 'axis_x', 'axis_y', and 'timestamp' as integers."
2. The AI generates the corresponding Stream JSON and suggests a schema structure.
3. You can then refine and save this definition directly to the server.

Schema API Reference

The Manager interacts with the eKuiper backend via the following standard endpoints (available for testing in the Swagger API Playground):

| Endpoint | Method | Description | | :--- | :--- | :--- | | /schemas/protobuf | GET | List all registered Protobuf schemas. | | /schemas/protobuf/{id} | GET | Fetch details of a specific schema. | | /schemas/protobuf | POST | Register a new .proto schema. | | /schemas/protobuf/{id} | DELETE | Remove a schema from the server. |

Note: Deleting a schema that is currently in use by an active Stream will cause that Stream to fail. eKuiper Manager will warn you if a schema is currently associated with active resources.