Schema · Samples · Integration · API · Compliance

Metadata Schema

ENDATA datasets adopt a unified metadata Schema, compatible with mainstream formats such as RLDS, LeRobot v3, and WebDataset, with ENDATA-specific extension fields. Each dataset package ships with a metadata.json file describing structure, licensing, and quality metrics.

Standard Fields

Example metadata.json

{
  "dataset_id": "EN-202603-0042",
  "modality": ["video", "text"],
  "domain": "film_tv",
  "task_family": ["pretrain", "sft"],
  "license_type": "commercial",
  "language": ["zh", "en"],
  "size_bytes": 2576980377,
  "record_count": 142800,
  "quality_score": 0.94,
  "compliance_audit": {
    "audit_id": "CA-2026-0178",
    "copyright_chains": 3,
    "verified_by": "ENDATA Legal Team"
  }
}

VLA-specific Extensions

Video datasets include additional action-sequence fields, compatible with RLDS and LeRobot v3:

{
  "vla_schema": "rlds_v1.2",
  "fps": 30,
  "resolution": "1080p",
  "action_dim": 7,
  "observation_keys": ["image", "depth", "proprio"],
  "task_annotations": true,
  "language_instruction": true
}

Sample Datasets

ENDATA provides free small-scale sample packages (100–5,000 records) for technical evaluation and pipeline integration testing. Sample data has identical structure to the full datasets, including complete metadata, annotations, and licensing information.

Available Samples

Request Process

① Email cs@endata.com.cn with company name, dataset name, and intended use
② Our commercial team replies within 1 business day with NDA signing link
③ Sample download link delivered within 24 hours of NDA signing (valid 7 days)

Integration Methods

ENDATA datasets support three integration methods to meet different security levels and infrastructure requirements:

① API Integration (Recommended)

POST https://api.endata.com.cn/v1/datasets/query
Authorization: Bearer <YOUR_API_KEY>
Content-Type: application/json

{
  "dataset_id": "EN-202603-0042",
  "filters": { "task_family": "sft", "language": "zh" },
  "limit": 1000,
  "format": "jsonl"
}

② Object Storage Push (OSS / S3)

Datasets can be pushed directly to your AWS S3, Aliyun OSS, or Tencent Cloud COS bucket. Encryption keys are held by the customer.

# Configuration example (Aliyun OSS)
{
  "delivery_mode": "oss_push",
  "endpoint": "oss-cn-shanghai.aliyuncs.com",
  "bucket": "your-private-bucket",
  "prefix": "endata/datasets/",
  "encryption": "SSE-KMS",
  "kms_key_id": "your-kms-key-id"
}

③ Private Cloud Deployment

For enterprises that prohibit data egress. ENDATA engineers provide:

· AES-256 encrypted offline media delivery
· Optional Docker images for private annotation platform
· Pipeline adapters (PyTorch / JAX / TF)

REST API v1

Base URL

https://api.endata.com.cn/v1

Authentication

All requests require an API Key in the Authorization header:

Authorization: Bearer YOUR_API_KEY

Main Endpoints

Rate Limits

Default: 100 req/min (standard subscription). Enterprise subscriptions support custom limits. Returns HTTP 429 when exceeded.

pip install endata-sdk

from endata import Client
client = Client(api_key="YOUR_KEY")

# Get dataset
ds = client.datasets.get("EN-202603-0042")

# Stream training data
for batch in ds.stream(batch_size=256):
    train(batch)

Copyright Authorization & Data Security

Three-Layer Authorization System

One of ENDATA's core advantages is a complete, traceable copyright authorization chain. Each dataset undergoes three layers of compliance review:

Data Security Certifications

Customer Data Isolation

Each customer's datasets are stored in dedicated encrypted namespaces with RBAC-controlled access. Customer data is never used for ENDATA's internal model training or third-party sharing.

Compliance Document Package

Verified customers can request:

· Complete copyright authorization chain documentation
· ISO 27001 / 27701 certification copies
· Data Security Management Certificate
· Data Processing Agreement (DPA) template
· Compliance due diligence support materials

Frequently Asked Questions