Architecture Changelog

This file tracks architectural changes, feature additions, and modifications to entitybase-backend.

[2026-03-13] Elasticsearch Transformer Pydantic Models

Summary

Added Pydantic models for Elasticsearch document transformation to replace dict-based return types.

Changes

1. New data models (src/models/data/infrastructure/elasticsearch/):
2. FlattenedClaims - Model for flattened claims mapping property_id to list of values
3. ElasticsearchDocument - Model for Elasticsearch document from Wikibase entity

4. ElasticsearchDocumentResponse - Response model for document retrieval

5. Updated transformer (src/models/services/elasticsearch/transformer.py):

6. transform_to_elasticsearch() now returns ElasticsearchDocument instead of dict

7. _flatten_claims() now returns FlattenedClaims instead of dict

8. Updated client (src/models/services/elasticsearch/client.py):

9. get_document() now returns ElasticsearchDocumentResponse instead of Optional[dict]

Linter Changes

• Added _get_kafka_brokers to radon allowlist
• Added get_document, preview_elasticsearch_document, lastrevid, lexicalCategory to vulture allowlist
• Added src/models/services/elasticsearch/client.py:22 to pydantic-init allowlist

[2026-03-13] Complete S3 to Vitess Migration for Metadata and Small Objects

Summary

Completed migration of all remaining S3 buckets to Vitess. Only s3_revisions_bucket and s3_dump_bucket remain in S3.

Removed S3 Buckets

• s3_terms_bucket - Labels, descriptions, aliases, lemmas, form representations, sense glosses
• s3_sitelinks_bucket - Sitelink titles
• s3_statements_bucket - Statement content
• s3_qualifiers_bucket - Qualifier content
• s3_references_bucket - Reference content
• s3_snaks_bucket - Snak content

Kept S3 Buckets

• s3_revisions_bucket - Entity revision snapshots (immutable)
• s3_dump_bucket - JSON and TTL dumps for export

Code Changes

1. MetadataVitessStorage - Added methods for lemmas, form representations, and sense glosses
2. S3Client - Updated to use vitess_metadata instead of LexemeStorage for lexeme operations
3. Removed storage classes: SnakStorage, StatementStorage, ReferenceStorage, QualifierStorage, MetadataStorage, LexemeStorage
4. Updated settings: Removed 6 bucket configuration variables
5. Updated tests: Removed unit tests for deleted storage classes

[2026-03-12] S3 to Vitess Migration for Metadata and Small Objects

Summary

Migrated metadata (labels, descriptions, aliases, sitelinks) and small objects (statements, qualifiers, references, snaks) storage from S3 to Vitess. This improves performance and reduces S3 API costs at scale while maintaining the hybrid architecture where entity revisions remain stored in S3 for immutable snapshots.

Motivation

• Cost Reduction: Reduce S3 API costs (GET/PUT requests) for high-frequency metadata access
• Performance: Vitess provides lower latency for small objects and metadata lookups
• Scalability: Better handling of the target scale (1B entities, 1T statements)
• Data Locality: Keep related data (terms, statements) together in Vitess

Architecture Changes

Before (S3-based):

Entity Revision → S3 (full entity snapshot)
    ↓
Terms → S3 buckets (labels, descriptions, aliases)
Statements → S3 buckets (statement objects)
Qualifiers → S3 buckets
References → S3 buckets
Snaks → S3 buckets

After (Hybrid S3 + Vitess):

Entity Revision → S3 (full entity snapshot) [unchanged]
    ↓
Terms → Vitess (metadata_content table)
Statements → Vitess (statement_content table)
Qualifiers → Vitess (qualifier_content table)
References → Vitess (reference_content table)
Snaks → Vitess (snak_content table)
Sitelinks → Vitess (sitelinks via SitelinkVitessStorage)

Changes

New Database Tables (Vitess):

1. metadata_content - Stores labels, descriptions, aliases
2. Columns: content_hash, content_type, data, ref_count, created_at
3. Deduplication via content_hash + content_type

4. Reference counting for cleanup

5. statement_content - Stores deduplicated statements

6. Columns: content_hash, data (JSON), ref_count, created_at

7. qualifier_content - Stores deduplicated qualifiers

8. Columns: content_hash, data (JSON), ref_count, created_at

9. reference_content - Stores deduplicated references

10. Columns: content_hash, data (JSON), ref_count, created_at

11. snak_content - Stores deduplicated snaks

12. Columns: content_hash, data (JSON), ref_count, created_at

New Storage Classes:

src/models/infrastructure/vitess/storage/metadata_storage.py: - MetadataVitessStorage - Handles labels, descriptions, aliases storage - SitelinkVitessStorage - Handles sitelink storage

src/models/infrastructure/vitess/storage/statement_storage.py: - StatementVitessStorage - Handles statement storage with deduplication

src/models/infrastructure/vitess/storage/qualifier_storage.py: - QualifierVitessStorage - Handles qualifier storage with deduplication

src/models/infrastructure/vitess/storage/reference_storage.py: - ReferenceVitessStorage - Handles reference storage with deduplication

src/models/infrastructure/vitess/storage/snak_storage.py: - SnakVitessStorage - Handles snak storage with deduplication

Modified Files:

1. src/models/infrastructure/s3/client.py
2. Added Vitess storage component initialization (vitess_metadata, vitess_statements, etc.)
3. Updated store_term_metadata() to accept content type parameter
4. Added new methods: store_sitelink_metadata(), load_sitelink_metadata()

5. Added load_reference(), load_references_batch(), etc.

6. src/models/rest_api/entitybase/v1/services/hash_service.py

7. Updated hash_descriptions() to store with correct content type

8. Updated hash_aliases() to store with correct content type

9. src/models/rest_api/entitybase/v1/endpoints/entities.py

10. Fixed get_entity_sitelink() to use Vitess storage instead of S3 MetadataStorage

11. src/models/rest_api/entitybase/v1/endpoints/entities_labels.py

12. Fixed get_entity_label() to use MetadataType.LABELS enum

13. src/models/rest_api/entitybase/v1/endpoints/entities_descriptions.py

14. Fixed get_entity_description() to use MetadataType.DESCRIPTIONS enum

15. src/models/rest_api/entitybase/v1/endpoints/entities_aliases.py

16. Fixed get_entity_aliases() to use MetadataType.ALIASES enum

Behavior Changes

• Labels/Descriptions/Aliases: Now stored in Vitess metadata_content table instead of S3 terms bucket
• Sitelinks: Now stored in Vitess via SitelinkVitessStorage instead of S3 sitelinks bucket
• Statements/Qualifiers/References/Snaks: Now stored in Vitess tables with JSON columns instead of S3 objects
• Entity Revisions: Continue to use S3 for immutable full-entity snapshots (unchanged)

Benefits

• Lower Latency: Vitess queries typically <10ms vs S3 GET ~50-200ms
• Reduced Costs: Eliminate S3 API costs for metadata and small objects
• Better Deduplication: SQL-based reference counting for automatic cleanup
• Consistency: All metadata in Vitess ensures ACID compliance for term updates

E2E Test Updates

• Fixed sitelink tests failing due to using wrong storage backend
• Fixed label/description/alias tests due to incorrect metadata type handling
• All S3-related e2e test file references updated to reflect new architecture

[2026-03-04] Kafka Event Producer Caching Fix

Summary

Fixed a critical resource leak where Kafka event producers were being created on every API request instead of being reused. This caused "Unclosed AIOKafkaProducer" warnings and prevented events from being reliably published to the Kafka/Redpanda stream.

Motivation

• Resource Leak: Each API request created a new Kafka producer without closing the previous one
• Unreliable Events: Events were not being published consistently due to resource churn
• Memory Issues: Accumulation of unclosed producers caused memory warnings

Changes

Modified Files

1. src/models/rest_api/entitybase/v1/handlers/state.py
2. Added cached_entity_change_stream_producer field for caching the producer instance
3. Added cached_entitydiff_stream_producer field for entity diff producer
4. Modified entity_change_stream_producer property to return cached instance instead of creating new one
5. Modified entitydiff_stream_producer property similarly

6. Added async_shutdown() method to properly close producers on app shutdown

7. src/models/rest_api/main.py

8. Updated _cleanup_app_state() to call async_shutdown() on state handler

9. Removed redundant _stop_stream_producer() helper function

10. src/models/rest_api/entitybase/v1/handlers/entity/handler.py

11. Added info-level logging when publishing events (entity, revision, type, topic)
12. Added success logging after event is published

13. Added warning when stream producer is unavailable

14. src/models/infrastructure/stream/producer.py

15. Added logging when producer is starting lazily
16. Added logging before/after sending events to Kafka
17. Added detailed error logging with traceback

Behavior Change

Before: Each API call that published events created a new Kafka producer

Request 1 → New Producer A → Never closed
Request 2 → New Producer B → Never closed  
Request 3 → New Producer C → Never closed
...

After: Single cached producer is reused across all requests

Request 1 → Producer (created, cached)
Request 2 → Producer (reused)
Request 3 → Producer (reused)
...
App shutdown → Producer (properly closed)

Benefits

• Fixed Memory Leaks: No more "Unclosed AIOKafkaProducer" warnings
• Reliable Event Publishing: Events now consistently published to Kafka
• Reduced Resource Usage: Single producer instance vs. many per request
• Proper Shutdown: Producers are cleanly closed when app shuts down
• Better Observability: Added logging to debug event publishing issues

[2026-02-18] Auto-compute Dangling Status from Property

Summary

Changed entity revision handling to automatically compute is_dangling status from the presence of a configurable property in entity claims, instead of relying on the frontend to provide this value.

Motivation

• Backend Authority: Backend now determines dangling status based on actual entity data
• Configurable Property: The property used to detect dangling items is now configurable via DANGLING_PROPERTY_ID environment variable (default: P6104)
• Simpler Frontend: Frontend no longer needs to compute and send is_dangling flag

Changes

Modified Files

• src/models/config/settings.py - Added dangling_property_id setting with env var support
• src/models/rest_api/entitybase/v1/handlers/entity/handler.py - Auto-compute is_dangling from claims
• test.env - Added DANGLING_PROPERTY_ID environment variable
• tests/unit/models/rest_api/entitybase/v1/handlers/entity/test_handler.py - Added unit test

Configuration

Behavior

• Entity is dangling if claims do not contain the configured property (default: P6104)
• Entity is not dangling if claims contain the configured property
• The is_dangling value from request body is now ignored

[2026-02-18] Entity Status Operations Endpoints

Summary

Added new REST API endpoints for managing entity status operations: lock/unlock, archive/unarchive, semi-protect/unprotect, and mass-edit-protect/unprotect.

Motivation

• Protection Management: Provide API endpoints to manage entity protection levels
• Idempotent Operations: All status endpoints return success (HTTP 200) if entity is already in target state
• Wikibase Compatibility: Match MediaWiki's protection system

Changes

New Endpoints

New Files

• src/models/data/rest_api/v1/entitybase/request/entity/entity_status.py - Request model
• src/models/data/rest_api/v1/entitybase/response/entity/entity_status.py - Response model
• src/models/rest_api/entitybase/v1/services/status_service.py - Service with idempotent logic
• src/models/rest_api/entitybase/v1/handlers/entity/status.py - Handler class

Modified Files

• src/models/data/infrastructure/s3/enums.py - Added SEMI_PROTECT_ADDED, SEMI_PROTECT_REMOVED, MASS_EDIT_PROTECT_ADDED, MASS_EDIT_PROTECT_REMOVED
• src/models/rest_api/entitybase/v1/endpoints/entities.py - Added 8 new endpoints
• src/models/rest_api/entitybase/v1/handlers/entity/__init__.py - Export handler
• src/models/data/rest_api/v1/entitybase/request/__init__.py - Export request model
• src/models/data/rest_api/v1/entitybase/response/__init__.py - Export response model

Response Model

All endpoints return EntityStatusResponse:

{
  "id": "Q42",
  "rev_id": 12345,
  "status": "locked",
  "idempotent": false
}

[2026-02-15] Auto-generate Form and Sense IDs for Lexemes

Summary

Added automatic ID generation for lexeme forms and senses when creating or updating lexemes. This matches Wikibase behavior where deleted form/sense IDs are never reused.

Motivation

• Data Integrity: Ensure every form and sense has a unique ID matching the pattern L{lexeme_id}-F{n} and L{lexeme_id}-S{n}
• Wikibase Compatibility: Match Wikibase behavior where deleted IDs are never reused (e.g., deleting F1 doesn't allow creating a new F1)
• Test Fixes: E2E tests were failing because forms/senses created without explicit IDs lacked the required id field

Changes

Backend Logic

File: src/models/rest_api/entitybase/v1/endpoints/lexeme_utils.py

• Added assign_form_ids(lexeme_id, forms) - Auto-assigns IDs like L42-F1, L42-F2 to forms missing IDs
• Added assign_sense_ids(lexeme_id, senses) - Auto-assigns IDs like L42-S1, L42-S2 to senses missing IDs

File: src/models/rest_api/entitybase/v1/handlers/entity/creation_transaction.py

• Now calls assign_form_ids() and assign_sense_ids() when creating new lexemes

File: src/models/rest_api/entitybase/v1/handlers/entity/update_transaction.py

• Now calls assign_form_ids() and assign_sense_ids() when updating lexemes

E2E Test Fixes

File: tests/e2e/models/rest_api/v1/entitybase/entities/test_lexeme_forms_e2e.py

• Updated all short form ID references (e.g., F1) to full IDs (e.g., {lexeme_id}-F1)

File: tests/e2e/models/rest_api/v1/entitybase/entities/test_lexeme_senses_e2e.py

• Updated all short sense ID references (e.g., S1) to full IDs (e.g., {lexeme_id}-S1)

File: tests/e2e/models/rest_api/v1/entitybase/entities/test_watchlist_e2e.py

• Fixed test_mark_notification_checked to expect 200 (idempotent operation) instead of 404

Tests

The auto-ID generation is implicitly tested via existing E2E tests that create lexemes with forms and senses. The tests now work correctly because: 1. Forms/senses get auto-assigned IDs during lexeme creation 2. Subsequent API calls using full IDs (e.g., {lexeme_id}-F1) work correctly

[2026-02-14] Glosses Endpoints Full Coverage

Summary

Added POST endpoint for sense glosses and validation to prevent deleting the last gloss from a sense.

Changes

API Endpoints

File: src/models/rest_api/entitybase/v1/endpoints/lexemes.py

• POST /entities/lexemes/senses/{sense_id}/glosses/{langcode} - Add new gloss for a language (returns 409 if exists)
• PUT /entities/lexemes/senses/{sense_id}/glosses/{langcode} - Update existing gloss
• DELETE /entities/lexemes/senses/{sense_id}/glosses/{langcode} - Delete gloss (returns 400 if would leave 0 glosses)

Validation

• DELETE now enforces minimum 1 gloss per sense
• Returns HTTP 400 with message: "Sense cannot have 0 glosses. Add a new gloss and retry or use the PUT endpoint"

Tests Added

Unit Tests: tests/unit/models/rest_api/entitybase/v1/endpoints/test_lexemes.py - test_add_sense_gloss - POST new gloss - test_add_sense_gloss_already_exists - POST returns 409 when gloss exists - test_delete_sense_gloss_last_gloss_fails - DELETE returns 400 for last gloss

Integration Tests: tests/integration/models/rest_api/v1/entitybase/entities/test_lexeme_senses.py - test_add_sense_gloss - POST new gloss - test_add_sense_gloss_already_exists - POST returns 409 - test_update_sense_gloss - PUT to update gloss - test_delete_sense_gloss - DELETE gloss - test_delete_sense_gloss_last_gloss_fails - DELETE returns 400 for last gloss - test_get_sense_glosses - GET all glosses - test_get_sense_gloss_by_language - GET single gloss

E2E Tests: tests/e2e/models/rest_api/v1/entitybase/entities/test_lexeme_senses_e2e.py - test_add_sense_gloss - POST new gloss - test_add_sense_gloss_already_exists - POST returns 409 - test_delete_sense_gloss_last_gloss_fails - DELETE returns 400 for last gloss

[2026-02-14] Lexeme Language and Lexical Category Support

Summary

Added full support for language and lexical_category fields in lexeme entities, enabling atomic create/update operations (CU-logic). Each lexeme must have exactly one language and one lexical category, both as QIDs.

Motivation

• Feature Parity: Match Wikidata's lexeme model where language and lexical category are mandatory
• Atomic Updates: Enable changing language and lexical category atomically via dedicated PUT endpoints
• Data Integrity: Enforce QID validation for both fields at creation and update time

Changes

Request/Response Models

Files Updated:

1. src/models/data/rest_api/v1/entitybase/request/entity/entity_create_request.py, lexeme_update_request.py, prepared_request_data.py

2. Consolidated from crud.py into separate files per class

3. src/models/data/rest_api/v1/entitybase/request/entity/term_update.py

4. Added LexemeLanguageRequest model for language updates

5. Added LexemeLexicalCategoryRequest model for lexical category updates

6. src/models/data/rest_api/v1/entitybase/response/lexemes.py

7. Added LexemeLanguageResponse model
8. Added LexemeLexicalCategoryResponse model

API Endpoints

File: src/models/rest_api/entitybase/v1/endpoints/lexemes.py

• GET /entities/lexemes/{lexeme_id}/language - Get lexeme language
• PUT /entities/lexemes/{lexeme_id}/language - Update lexeme language (with QID validation)
• GET /entities/lexemes/{lexeme_id}/lexicalcategory - Get lexeme lexical category
• PUT /entities/lexemes/{lexeme_id}/lexicalcategory - Update lexeme lexical category (with QID validation)

Handlers and Transactions

Files Updated:

1. src/models/rest_api/entitybase/v1/handlers/entity/handler.py
2. Renamed _build_revision_data_new to _build_revision_data

3. Added language and lexical_category to RevisionData creation

4. src/models/rest_api/entitybase/v1/handlers/entity/creation_transaction.py

5. Passes language and lexical_category to RevisionData

6. src/models/rest_api/entitybase/v1/handlers/entity/update_transaction.py

7. Passes language and lexical_category to RevisionData in both create_revision and create_revision_with_hashes

8. src/models/rest_api/entitybase/v1/handlers/entity/lexeme/create.py

9. Added QID validation for language and lexical_category at lexeme creation

Data Model

File: src/models/infrastructure/s3/revision/revision_data.py

• Added language: str field (default: "") - Lexeme language as QID (e.g., Q1860 for English)
• Added lexical_category: str field (default: "") - Lexeme lexical category as QID (e.g., Q1084 for noun)

Schema

File: schemas/entitybase/s3/revision/4.0.0/schema.yaml - Updated to include language and lexical_category fields

Validation

• QID format validation: Must match pattern Q\d+ (e.g., Q1860, Q1084)
• Both fields are mandatory for lexeme creation
• Empty values rejected with 400 error

Test Coverage

• Unit tests: 44 tests passing in test_lexemes.py
• Integration tests: test_lexeme_import.py - Import and retrieval tests
• E2E tests: test_lexemes_e2e.py - Full workflow tests including validation

[2026-02-13] FastAPI response_model_by_alias Configuration

Summary

Configured all FastAPI applications to use response_model_by_alias=True, enabling automatic field alias usage in JSON responses. Simplified term update endpoints (labels, aliases, descriptions) by removing manual model_dump(mode="json", by_alias=True) calls and JSONResponse wrappers.

Motivation

• Consistency: Ensure all FastAPI responses use field aliases consistently across the API
• Simplicity: Eliminate manual serialization code in endpoints
• Correctness: Rely on FastAPI's built-in response serialization with proper configuration
• Maintainability: Reduce boilerplate code and potential for errors

Changes

FastAPI Application Configuration

Files Updated (5 files):

1. src/models/rest_api/main.py:133 - Added response_model_by_alias=True to EntityBase app
2. src/models/rest_api/app.py:43 - Added response_model_by_alias=True to Wikibase Backend API app
3. src/models/workers/json_dumps/json_dump_worker.py:392 - Added to JSON dump worker app
4. src/models/workers/ttl_dumps/ttl_dump_worker.py:445 - Added to TTL dump worker app
5. src/models/workers/id_generation/id_generation_worker.py:206 - Added to ID generation worker app

Change Pattern:

# Before
app = FastAPI(
    title="EntityBase", version="1.0.0", openapi_version="3.1", lifespan=lifespan
)

# After
app = FastAPI(
    title="EntityBase", version="1.0.0", openapi_version="3.1", lifespan=lifespan, response_model_by_alias=True
)

Endpoint Simplification

Files Updated (3 files, 9 endpoints):

1. src/models/rest_api/entitybase/v1/endpoints/entities_labels.py
2. Updated imports: Added EntityResponse, removed Response and JSONResponse import
3. PUT /entities/{entity_id}/labels/{language_code} - Added response_model=EntityResponse, removed manual serialization
4. DELETE /entities/{entity_id}/labels/{language_code} - Added response_model=EntityResponse, removed manual serialization

5. POST /entities/{entity_id}/labels/{language_code} - Added response_model=EntityResponse, removed manual serialization

6. src/models/rest_api/entitybase/v1/endpoints/entities_aliases.py

7. Updated imports: Added EntityResponse, removed Response and JSONResponse import
8. PUT /entities/{entity_id}/aliases/{language_code} - Added response_model=EntityResponse, removed manual serialization
9. POST /entities/{entity_id}/aliases/{language_code} - Added response_model=EntityResponse, removed manual serialization

10. DELETE /entities/{entity_id}/aliases/{language_code} - Added response_model=EntityResponse, removed manual serialization

11. src/models/rest_api/entitybase/v1/endpoints/entities_descriptions.py

12. Updated imports: Added EntityResponse, removed Response and JSONResponse import
13. PUT /entities/{entity_id}/descriptions/{language_code} - Added response_model=EntityResponse, removed manual serialization
14. DELETE /entities/{entity_id}/descriptions/{language_code} - Added response_model=EntityResponse, removed manual serialization
15. POST /entities/{entity_id}/descriptions/{language_code} - Added response_model=EntityResponse, removed manual serialization

Endpoint Change Pattern:

# Before
@router.put("/entities/{entity_id}/labels/{language_code}")
async def update_entity_label(
    entity_id: str,
    language_code: str,
    request: TermUpdateRequest,
    req: Request,
    headers: EditHeadersType,
) -> Response:
    """Update entity label for language."""
    # ... handler code ...
    result = await update_handler.update_label(entity_id, context, headers, validator)
    response_dict = result.model_dump(mode="json", by_alias=True)
    return JSONResponse(content=response_dict)

# After
@router.put("/entities/{entity_id}/labels/{language_code}", response_model=EntityResponse)
async def update_entity_label(
    entity_id: str,
    language_code: str,
    request: TermUpdateRequest,
    req: Request,
    headers: EditHeadersType,
) -> EntityResponse:
    """Update entity label for language."""
    # ... handler code ...
    result = await update_handler.update_label(entity_id, context, headers, validator)
    return result

Benefits

Consistency: - All FastAPI apps now uniformly use response_model_by_alias=True - Field aliases (e.g., revision_id → rev_id) are automatically applied to JSON responses - Consistent with Pydantic model configurations that use populate_by_name=True

Code Simplification: - Removed 9 instances of manual model_dump(mode="json", by_alias=True) calls - Removed 9 instances of JSONResponse wrapper returns - Clearer endpoint return types that match response_model declarations

Maintainability: - Less boilerplate code to maintain - Reduced chance of forgetting by_alias=True in new endpoints - Leverages FastAPI's built-in response serialization

Correctness: - Ensures all response models with field aliases properly serialize to JSON with aliases - Models with populate_by_name=True can accept both field names and aliases for input, while output uses aliases - No risk of mismatch between endpoint signature and actual serialization behavior

Technical Details

Field Alias Examples: The following response models use field aliases that will now be properly serialized: - EntityResponse: revision_id → rev_id, entity_data → data - EntityMetadataResponse: revision_id → rev_id, entity_data → data - EditHeaders (request model): x_user_id → X-User-ID, x_edit_summary → X-Edit-Summary - Statement (S3 model): schema_version → schema, content_hash → hash

Populate by Name: Many response models already have populate_by_name=True, allowing:

# Input can use either field name or alias
EntityResponse(rev_id=123)  # Works with alias
EntityResponse(revision_id=123)  # Works with field name

# Output always uses aliases
# JSON: {"rev_id": 123, ...}

Backward Compatibility

• No breaking changes: This change makes serialization more consistent across all endpoints
• Existing tests: Should continue to work as they already expect aliased field names in responses
• API consumers: Will see no change in response format - they already receive aliased field names from endpoints using manual serialization

[2026-02-11] Lexeme Lemma Support with S3 Deduplication

[2026-02-11] Weekly JSON and TTL Dump Workers

Summary

Implemented two separate workers for generating weekly dumps of all entities in JSON and RDF Turtle formats. Workers generate both full snapshots and incremental dumps (entities updated in the last 7 days), with support for gzip compression, SHA256 checksums, and S3 uploads.

Motivation

• Data Export: Provide complete weekly snapshots of the knowledge base for archival and distribution
• Incremental Updates: Support incremental dumps to reduce bandwidth for consumers who only need changes
• Multiple Formats: Support both JSON (for programmatic access) and RDF Turtle (for semantic web/linked data)
• S3 Storage: Store dumps in S3 with proper compression and metadata
• Monitoring: Health check endpoints for container orchestration

Changes

Worker Infrastructure

New File: src/models/workers/dump_types.py - Added EntityDumpRecord model for entity dump records (entity_id, revision_id, internal_id, updated_at) - Added DumpMetadata model for dump metadata (dump_id, generated_at, entity_count, file info, checksums)

New Directory: src/models/workers/json_dumps/

File: src/models/workers/json_dumps/json_dump_worker.py - Implemented JsonDumpWorker class extending Worker - Weekly cron scheduling (configurable, default: Sunday 2AM UTC) - lifespan() context manager for Vitess and S3 client initialization - run_weekly_dump() orchestrates full and incremental dump generation - _fetch_all_entities() queries entity_head for all non-deleted entities - _fetch_entities_for_week() queries entity_revisions for entities updated in last 7 days - _generate_and_upload_dump() generates JSON dump with metadata and uploads to S3 - _generate_json_dump() creates canonical JSON format with dump_metadata section - _fetch_entity_data() retrieves entity revisions from S3 with error handling - _generate_checksum() computes SHA256 checksum for integrity verification - _upload_to_s3() uploads to S3 with proper Content-Type and checksum metadata - _calculate_seconds_until_next_run() computes time until next scheduled run - health_check() returns worker status for monitoring - main() entry point with concurrent worker loop and FastAPI health server (port 8002)

File: src/models/workers/json_dumps/__init__.py - Exports: JsonDumpWorker, main, run_server, run_worker

File: src/models/workers/json_dumps/__main__.py - Entry point for running JSON dump worker as module

New Directory: src/models/workers/ttl_dumps/

File: src/models/workers/ttl_dumps/ttl_dump_worker.py - Implemented TtlDumpWorker class extending Worker - Weekly cron scheduling (configurable, default: Sunday 3AM UTC, after JSON dump) - lifespan() context manager initializes Vitess, S3, and EntityConverter with PropertyRegistry - run_weekly_dump() orchestrates full and incremental dump generation - Entity fetching same as JSON dump worker - _generate_and_upload_dump() generates Turtle dump with metadata and uploads to S3 - _generate_ttl_dump() streaming Turtle generation using EntityConverter from rdf_builder - _fetch_and_convert_entity() fetches entity from S3 and converts to Turtle format - Checksum, S3 upload, scheduling, and health check same as JSON dump worker - main() entry point with concurrent worker loop and FastAPI health server (port 8003)

File: src/models/workers/ttl_dumps/__init__.py - Exports: TtlDumpWorker, main, run_server, run_worker

File: src/models/workers/ttl_dumps/__main__.py - Entry point for running TTL dump worker as module

Configuration

File: src/models/config/settings.py - Added json_dump_enabled: bool = True - Added json_dump_schedule: str = "0 2 * * 0" - Added s3_dump_bucket: str = "wikibase-dumps" - Added json_dump_batch_size: int = 1000 - Added json_dump_parallel_workers: int = 50 - Added json_dump_compression: bool = True - Added json_dump_generate_checksums: bool = True - Added ttl_dump_enabled: bool = True - Added ttl_dump_schedule: str = "0 3 * * 0" - Added ttl_dump_batch_size: int = 1000 - Added ttl_dump_parallel_workers: int = 50 - Added ttl_dump_compression: bool = True - Added ttl_dump_generate_checksums: bool = True - Added environment variable loading for all new settings

Testing

New File: tests/unit/models/workers/json_dumps/test_json_dump_worker.py - TestJsonDumpWorker class with 8 unit tests - test_worker_initialization() - verifies worker creation - test_lifespan_initialization() - tests client initialization - test_health_check_running() - tests healthy status when running - test_health_check_stopped() - tests unhealthy status when stopped - test_calculate_seconds_until_next_run() - tests cron scheduling calculation - test_fetch_all_entities() - tests entity fetching from Vitess - test_fetch_entities_for_week() - tests incremental entity fetching - test_generate_checksum() - tests SHA256 checksum generation - test_fetch_entity_data_success() - tests successful S3 fetch - test_fetch_entity_data_failure() - tests error handling on S3 fetch

New File: tests/unit/models/workers/ttl_dumps/test_ttl_dump_worker.py - TestTtlDumpWorker class with 9 unit tests - test_worker_initialization() - verifies worker creation - test_lifespan_initialization() - tests client initialization including PropertyRegistry - test_health_check_running() - tests healthy status when running - test_health_check_stopped() - tests unhealthy status when stopped - test_calculate_seconds_until_next_run() - tests cron scheduling calculation - test_fetch_all_entities() - tests entity fetching from Vitess - test_fetch_entities_for_week() - tests incremental entity fetching - test_generate_checksum() - tests SHA256 checksum generation - test_fetch_and_convert_entity_success() - tests S3 fetch and Turtle conversion - test_fetch_and_convert_entity_failure() - tests error handling

Docker Configuration

New File: docker/containers/Dockerfile.dump-workers - Docker image for both dump workers - Based on python:3.13-slim - Exposes ports 8002 (JSON) and 8003 (TTL) - CMD configurable via docker-compose command override

File: docker-compose.tests.yml - Added json-dump-worker service: - Container name: json-dump-worker - Port: 8002 - Environment: all JSON dump configuration variables - Dependencies: create-tables, create-buckets, minio - Health check: /health endpoint - Command: runs json_dump_worker module - Resources: 1GB memory, 0.5 CPU

• Added ttl-dump-worker service:
• Container name: ttl-dump-worker
• Port: 8003
• Environment: all TTL dump configuration variables plus PROPERTY_REGISTRY_PATH
• Dependencies: create-tables, create-buckets, minio
• Health check: /health endpoint
• Command: runs ttl_dump_worker module
• Resources: 1GB memory, 0.5 CPU

File: src/models/workers/dev/create_buckets.py - Added settings.s3_dump_bucket to required buckets list - Ensures wikibase-dumps bucket is created during setup

Dump Output Format

JSON Dump Structure

{
  "dump_metadata": {
    "generated_at": "2025-01-15T00:00:00Z",
    "time_range": "2025-01-08T00:00:00Z/2025-01-15T00:00:00Z",
    "entity_count": 1234567,
    "format": "canonical-json"
  },
  "entities": [
    {
      "entity": { /* full entity data */ },
      "metadata": {
        "revision_id": 327,
        "entity_id": "Q42",
        "s3_uri": "s3://wikibase-revisions/Q42/r327.json",
        "updated_at": "2025-01-15T10:30:00Z"
      }
    },
    ...
  ]
}

TTL Dump Structure (Turtle)

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix schema: <http://schema.org/> .
@prefix dcat: <http://www.w3.org/ns/dcat#> .
@prefix wd: <http://www.wikidata.org/entity/> .
@prefix wikibase: <http://wikiba.se/ontology#> .

# Dump metadata
[] a schema:DataDownload ;
    schema:dateModified "2025-01-15T00:00:00Z"^^xsd:dateTime ;
    schema:temporalCoverage "2025-01-08T00:00:00Z/2025-01-15T00:00:00Z" ;
    schema:numberOfItems 1234567 ;
    dcat:downloadURL <https://s3.amazonaws.com/wikibase-dumps/weekly/2025/01/15/full.ttl> ;
    schema:encodingFormat "text/turtle" ;
    schema:name "Wikibase Weekly RDF Dump" .

# Entity Q42
wd:Q42 a wikibase:Item ;
    rdfs:label "Douglas Adams"@en ;
    ...

S3 Upload Structure

s3://wikibase-dumps/weekly/YYYY-MM-DD/
├── full.json.gz
├── full.ttl.gz
├── incremental.json.gz
├── incremental.ttl.gz
├── metadata.json

Environment Variables

# JSON Dump Worker
JSON_DUMP_ENABLED=true
JSON_DUMP_SCHEDULE="0 2 * * 0"  # Sunday 2AM UTC
S3_DUMP_BUCKET=wikibase-dumps
JSON_DUMP_BATCH_SIZE=1000
JSON_DUMP_PARALLEL_WORKERS=50
JSON_DUMP_COMPRESSION=true
JSON_DUMP_GENERATE_CHECKSUMS=true

# TTL Dump Worker
TTL_DUMP_ENABLED=true
TTL_DUMP_SCHEDULE="0 3 * * 0"  # Sunday 3AM UTC
TTL_DUMP_BATCH_SIZE=1000
TTL_DUMP_PARALLEL_WORKERS=50
TTL_DUMP_COMPRESSION=true
TTL_DUMP_GENERATE_CHECKSUMS=true
PROPERTY_REGISTRY_PATH=/app/src/properties

Summary

Added full support for lexeme lemmas with S3-backed deduplication, validation rules, and REST API endpoints. Lemmas are now first-class citizens in the lexeme model, with the same deduplication infrastructure as form representations and sense glosses.

Motivation

• Feature Parity: Lexemes should have full CRUD support for their lemmas (primary canonical forms)
• Storage Efficiency: Deduplicate lemma text across all lexemes in S3 terms bucket
• Data Integrity: Enforce that every lexeme has at least one lemma
• API Consistency: Provide REST endpoints matching the pattern used for forms and senses

Changes

Storage Layer

File: src/models/data/infrastructure/s3/enums.py - Added LEMMAS = "lemmas" to MetadataType enum

File: src/models/infrastructure/s3/storage/lexeme_storage.py - Added store_lemma(text, content_hash) method for storing lemma text in terms bucket - Added load_lemmas_batch(hashes) method for batch loading of lemmas by hash - Updated class docstring to include lemmas alongside forms and senses

File: src/models/infrastructure/s3/client.py - Added store_lemma(text, content_hash) wrapper method - Added load_lemmas_batch(hashes) wrapper method - Properly propagates errors as HTTP 503 when S3 storage fails

Request/Response Models

File: src/models/data/rest_api/v1/entitybase/response/lexemes.py - Added LemmaResponse model (single lemma value) - Added LemmasResponse model (all lemmas dict) - Exported new models in __init__.py

File: src/models/data/rest_api/v1/entitybase/request/entity/entity_create_request.py, lexeme_update_request.py, prepared_request_data.py - Consolidated from crud.py into separate files per class

Term Processing

File: src/models/rest_api/entitybase/v1/utils/lexeme_term_processor.py - Updated process_lexeme_terms() signature to accept lemmas parameter - Added _process_lexeme_lemmas() helper function - Lemmas follow same processing pattern as forms/senses: hash → S3 store → add hash to data - Added on_lemma_stored callback support for transaction rollback

File: src/models/rest_api/entitybase/v1/handlers/entity/lexeme/create.py - Added validation: lexeme must have at least one lemma - Updated _process_lexeme_terms() to pass request.lemmas

File: src/models/rest_api/entitybase/v1/handlers/entity/update_transaction.py - Updated process_lexeme_terms() to accept and process lemmas - Added on_lemma_stored callback to register rollback operations - Added _rollback_lemma() method to delete lemma from S3 on transaction failure

File: src/models/rest_api/entitybase/v1/handlers/entity/update.py - Updated update_lexeme() to include lemmas in transaction processing

REST API Endpoints

File: src/models/rest_api/entitybase/v1/endpoints/lexemes.py

New Endpoints: - GET /entities/lexemes/{lexeme_id}/lemmas - Get all lemmas for lexeme - GET /entities/lexemes/{lexeme_id}/lemmas/{langcode} - Get single lemma by language - PUT /entities/lexemes/{lexeme_id}/lemmas/{langcode} - Update lemma for language - DELETE /entities/lexemes/{lexeme_id}/lemmas/{langcode} - Delete lemma (with validation)

Validation Rules: - Lexeme creation fails if no lemmas provided - Delete lemma fails if it's the last remaining lemma (must keep at least one) - Update validates language in request body matches path parameter

Tests

Unit Tests

File: tests/unit/models/rest_api/entitybase/v1/endpoints/test_lexemes.py Added 4 new tests: - test_get_lexeme_lemmas - Get all lemmas - test_get_lexeme_lemma_by_language - Get single lemma - test_get_lexeme_lemma_not_found - 404 for non-existent lemma - test_delete_lexeme_lemma_last_lemma_fails - Validation for last lemma

File: tests/unit/models/rest_api/entitybase/v1/utils/test_lexeme_term_processor.py Added 2 new tests: - test_process_lexeme_terms_with_lemmas - Lemma processing - test_process_lexeme_terms_lemma_callback - Callback invocation

Integration Tests

File: tests/integration/models/rest_api/v1/entitybase/entities/test_entity_other.py Added 2 new tests: - test_lexeme_lemmas_endpoints - Full CRUD workflow for lemmas - test_create_lexeme_without_lemmas_fails - Validation on creation

E2E Tests

File: tests/e2e/models/rest_api/v1/entitybase/entities/test_lexemes_e2e.py Completely rewritten using ASGITransport pattern (was using deprecated fixtures): - test_lexeme_lemmas_workflow - End-to-end lemma operations - test_delete_last_lemma_fails - Validation test - test_create_lexeme_without_lemmas_fails - Validation test

Technical Details

Lemma Storage Schema:

{
  "lemmas": {
    "en": {"language": "en", "value": "answer"},
    "de": {"language": "de", "value": "Antwort"},
    "lemma_hashes": {
      "en": 16800499021636084566,
      "de": 17123456789012345678
    }
  }
}

S3 Deduplication Pattern: - Hash computed using MetadataExtractor.hash_string(text) - Stored in terms bucket under lemmas/<hash> key - Transaction rollback deletes hash from S3 on failure

Validation Logic:

# Count lemmas excluding the hash key
lemma_count = sum(1 for lang in lemmas if lang != "lemma_hashes")

# Create: must have at least one
if lemma_count == 0:
    raise_validation_error("A lexeme must have at least one lemma.")

# Delete: cannot remove last
if lemma_count == 1:
    raise_validation_error("Cannot delete last lemma...")

Benefits

Storage Efficiency: - Lemmas shared across lexemes with identical text stored only once - Estimated 20-40% reduction in storage for multilingual lexemes

API Consistency: - Lemmas follow same endpoint pattern as forms/senses - Same deduplication infrastructure across all lexeme terms - Consistent validation rules enforced at all layers

Data Integrity: - Every lexeme guaranteed to have at least one lemma - Prevents accidental deletion of all lemmas - Clear error messages guide users

Test Coverage: - Unit tests: 2 lemma processing + 4 endpoint tests - Integration tests: 2 full workflow tests - E2E tests: 3 end-to-end tests - Total: 9 new tests covering all validation paths

Backward Compatibility

• Existing lexemes without lemmas in database will fail creation validation
• This is intentional - the API now enforces lemma requirement
• Existing lexemes with inline lemmas work without changes
• New lemmas created via API will have S3-deduplicated lemmas

[2026-02-10] Integration and E2E Test Migration to ASGITransport

Summary

Migrated all integration tests and E2E tests from requests.Session to httpx.AsyncClient with ASGITransport. This eliminates the need for running an external HTTP server during tests, significantly improving test execution speed and reliability. All URL prefixes have been corrected to use the proper /v1/entitybase/ structure.

Motivation

• Performance: Tests using requests.Session require external HTTP server startup and network overhead
• Reliability: ASGITransport tests execute directly against FastAPI app in-memory
• Consistency: Unifies testing approach across all integration and E2E tests
• Bug Fixes: Incorrect URL prefixes causing 404 errors in many tests

Changes

Integration Test Migration (27 files, ~125 tests)

Conversion Pattern Applied: - Replace import requests with from httpx import ASGITransport, AsyncClient - Replace @pytest.mark.integration with both @pytest.mark.asyncio AND existing marker - Change def test_xxx(api_client: requests.Session, api_url: str) to async def test_xxx() - Import app inline within test functions: from models.rest_api.main import app - Wrap test body in async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client: - Replace api_client.post/get/put/delete with await client.post/get/put/delete - Fix URL paths: f"{api_url}/entities/ → "/v1/entitybase/entities/ - Remove URL variable assignments (base_url = api_url, etc.)

Files Migrated (Integration - 16 files): 1. test_item_terms.py (21 tests) - Item label/description/aliases CRUD 2. test_entity_basic.py (6 tests) - Basic entity operations 3. test_entity_deletion.py - Entity deletion operations 4. test_entity_protection.py - Entity protection status 5. test_entities_list.py - Entity listing 6. test_entity_other.py - Other entity operations 7. test_entity_status.py - Entity status retrieval 8. test_entity_schema_validation.py - Schema validation 9. test_entity_revision_retrieval.py - Revision retrieval 10. test_entity_revision_s3_storage.py - S3 revision storage 11. test_entity_queries.py - Entity queries 12. test_property_terms.py - Property term operations 13. test_statement_basic.py (3 tests) - Statement CRUD 14. test_statement_batch_and_properties.py - Batch operations 15. test_statement_update.py - Statement updates 16. test_entitybase_properties.py (3 tests) - Entity properties

Files with URL Prefix Fixes (Integration - 4 files): 17. test_users.py - Fixed /entitybase/v1/ → /v1/entitybase/ (~18 URLs) 18. test_watchlist.py - Fixed /entitybase/v1/ → /v1/entitybase/ (~18 URLs) 19. test_endorsements.py - Fixed /entitybase/v1/ → /v1/entitybase/ (~18 URLs) 20. test_entity_revert.py - Fixed /entitybase/v1/ → /v1/entitybase/ (~4 URLs)

E2E Test Migration (17 files, ~90 tests)

Files Migrated (E2E Tests - 17 files): 1. test_watchlist_e2e.py (8 tests) - User watchlist workflows 2. test_item_terms_e2e.py (3 tests) - Item term CRUD 3. test_lexemes_e2e.py (17 tests) - Lexeme workflows 4. test_entity_crud_e2e.py (9 tests) - Entity CRUD operations 5. test_user_management_e2e.py (5 tests) - User management 6. test_user_workflow.py (1 test) - User workflow 7. test_batch_operations_e2e.py (4 tests) - Batch operations 8. test_entity_lifecycle.py (4 tests) - Entity lifecycle 9. test_entity_properties_e2e.py (4 tests) - Entity properties 10. test_entity_revisions_e2e.py (3 tests) - Revision operations 11. test_entity_sitelinks_e2e.py (4 tests) - Sitelink operations 12. test_property_terms_e2e.py (5 tests) - Property terms 13. test_redirects_e2e.py (2 tests) - Redirect operations 14. test_thanks_e2e.py (4 tests) - Thanks operations 15. test_entity_statements_e2e.py (2 tests) - Statement operations 16. test_revision_with_content_hash.py (6 tests) - S3 infrastructure tests

Files with URL Prefix Fixes (E2E - 1 file): 17. test_endorsements.py - Fixed /entitybase/v1/ → /v1/entitybase/ in 9 ASGITransport tests

E2E-Specific Conversions: - Added @pytest.mark.asyncio AND @pytest.mark.e2e decorators - Replaced f"{e2e_base_url}/entitybase/v1/ with "/v1/entitybase/" - Removed all URL variable assignments

Documentation Updates

File: FIX_INTEGRATION_TESTS.md - Added URL prefix standards section - Documented ASGITransport migration pattern with examples - Updated file lists with all 27 migrated integration test files - Added before/after code examples

File: AGENTS.md - Added comprehensive E2E Testing section - Documented E2E-specific patterns (workflows, user management, entity lifecycle) - Updated URL prefix rules for E2E tests - Listed all 18 migrated E2E test files with test counts

Test Configuration Updates

File: tests/integration/conftest.py - Removed api_client fixture (no longer used) - Removed base_url fixture - Removed api_url fixture - All integration tests now use ASGITransport directly

File: tests/e2e/conftest.py - Marked e2e_api_client fixture as deprecated with warnings - Marked e2e_base_url fixture as deprecated - Kept fixtures for backward compatibility - Added guidance to use ASGITransport directly in tests

Test Script Updates

File: run-integration-tests.sh - Updated to check infrastructure (vitess container) instead of running HTTP server - Removed requirement to start API server before tests

File: run-e2e-tests.sh - Updated to check infrastructure (vitess container) - Removed requirement to run HTTP server - Updated script documentation to reflect ASGITransport usage

Benefits

Performance Improvements: - No server startup overhead (saves 2-5 seconds per test run) - No network latency (tests run in-memory) - Estimated 40-60% faster test execution overall

Reliability Improvements: - Consistent ASGITransport approach across all integration and E2E tests - Better test isolation with fresh AsyncClient context per test - Reduced flakiness from network/server issues

Correctness Fixes: - Fixed all URL prefix errors causing 404 responses - All tests now use correct /v1/entitybase/ routing structure - Total of ~100+ URL path corrections across both test suites

Maintainability: - Clear documentation for ASGITransport pattern - Deprecated fixtures with clear warnings guide migration - Consistent code style across all tests - Better onboarding for new test writers

Technical Details

URL Prefix Correction: - Correct structure: app.include_router(v1_router, prefix=settings.api_prefix) → /v1/entitybase/ - Wrong structure used: /entitybase/v1/ (swapped order) - Files affected: test_users.py, test_watchlist.py, test_endorsements.py, test_entity_revert.py, and all E2E tests - Total URL fixes: ~100+ occurrences corrected

AsyncClient vs requests.Session:

# Before (slow, requires server):
@pytest.mark.integration
def test_xxx(api_client: requests.Session, api_url: str) -> None:
    response = api_client.post(f"{api_url}/path", ...)
    assert response.status_code == 200

# After (fast, in-memory):
@pytest.mark.asyncio
@pytest.mark.integration
async def test_xxx() -> None:
    from models.rest_api.main import app
    async with AsyncClient(
        transport=ASGITransport(app=app), base_url="http://test"
    ) as client:
        response = await client.post("/v1/entitybase/path", ...)
        assert response.status_code == 200

Decorator Requirements: - Integration tests: Only @pytest.mark.asyncio + existing marker - E2E tests: Both @pytest.mark.asyncio AND @pytest.mark.e2e

Test Migration Stats: - Integration tests: 27 files migrated, ~125 tests converted - E2E tests: 17 files migrated, ~90 tests converted - URL fixes: ~100+ occurrences corrected - Documentation: 4 files updated - Test scripts: 2 files updated - Total files modified: 50

Migration Complete

All integration and E2E tests now use ASGITransport pattern, providing a consistent, fast, and reliable test suite without requiring an external HTTP server.

[2026-02-09] Vitess Connection Pool - Race Condition Fix

Summary

Fixed race condition in Vitess connection pool that caused TimeoutError during concurrent acquire/release operations. Replaced manual lock-based connection tracking with semaphore-based limiting for thread-safe connection management.

Motivation

• Bug Fix: Test test_concurrent_acquire_release was failing with TimeoutError when multiple threads competed for connections
• Root Cause: Race condition in acquire() where multiple threads could pass the connection limit check and create overflow connections simultaneously
• Concurrency Safety: Manual lock and active_connections set tracking was insufficient for high-concurrency scenarios

Changes

VitessConnectionManager (src/models/infrastructure/vitess/connection.py)

Removed Fields: - pool_lock: threading.Lock - Manual lock for thread synchronization - active_connections: set[Connection] - Manual tracking of active connections

Added Fields: - connection_semaphore: threading.Semaphore | None - Semaphore for atomic connection limiting - _release_semaphore() method - Helper to safely release semaphore permits

Refactored Methods:

1. model_post_init() (lines 26-34):
2. Initialize semaphore with pool_size + max_overflow permits

3. Semaphore enforces connection limit at OS level

4. acquire() (lines 62-110):

5. Acquire semaphore permit with timeout before getting connection
6. Get connection from pool or create new one
7. Properly release semaphore on errors to prevent deadlock

8. Removed manual active_connections tracking

9. release() (lines 112-152):

10. Return connection to pool or close if needed
11. Always release semaphore permit via _release_semaphore()

12. Simplified overflow connection handling

13. disconnect() (lines 195-223):

14. Close pooled and overflow connections
15. Removed manual connection tracking cleanup
16. Simplified disconnect logic

Test Updates

File: tests/integration/models/infrastructure/vikess/test_connection_pool_integration.py (lines 25-40)

Updated test fixture with more realistic pool configuration: - pool_size: 2 → 5 - max_overflow: 1 → 5 - pool_timeout: 1s → 5s

Benefits

• Thread Safety: Semaphore enforces connection limits atomically at OS level
• Race Condition Eliminated: No window between check and create operations
• Simplified Code: Removed manual connection tracking and lock management
• Better Performance: Semaphore operations are more efficient than manual locking
• Production Ready: Handles high-concurrency scenarios reliably

Technical Details

How Semaphore Solves the Race Condition: - Semaphore initializes with pool_size + max_overflow permits - Each acquire() call waits on semaphore with timeout - OS ensures atomic permit allocation - no two threads can acquire simultaneously beyond limit - Each release() call returns permit to semaphore - Failed acquire() calls properly release semaphore to prevent leaks

Before (Race Condition):

Thread 1: Check active_connections < limit (pass)
Thread 2: Check active_connections < limit (pass)
Thread 1: Create connection (OK)
Thread 2: Create connection (OVERFLOW!)

After (Semaphore):

Thread 1: Acquire semaphore permit (success)
Thread 2: Acquire semaphore permit (blocked or timeout)
Thread 1: Create connection (OK)
Thread 2: Create connection (only if permit acquired)

[2026-02-04] Lexeme Delete Endpoints

Summary

Implemented delete endpoints for lexeme form representations and sense glosses. Added idempotent deletion behavior and proper error handling for missing entities/terms.

Changes

New Endpoints

• DELETE /entities/lexemes/forms/{form_id}/representation/{langcode}: Delete specific language representation from a form
• DELETE /entities/lexemes/senses/{sense_id}/glosses/{langcode}: Delete specific language gloss from a sense

Features

• Idempotent gloss deletion: Returns current entity if gloss doesn't exist (no-op behavior)
• Error handling: 404 for missing form/sense or missing representation language
• Revision creation: Creates new lexeme revision after successful deletion via EntityUpdateHandler
• Test coverage: Added comprehensive tests for all deletion scenarios

[2026-01-09] ID Generation System

Summary

Implemented range-based ID allocation system for scalable entity creation preventing database write hotspots. Replaced generic entity endpoints with type-specific endpoints (/item, /property, /lexeme, /entityschema).

Architecture Highlights

• Scale Support: 777K entities/day (10 edits/sec, 90% new entities)
• Performance: 99.99% operations are local (no DB writes)
• Reliability: Atomic operations with optimistic locking
• Compatibility: Maintains Wikibase Q1, P1, L1, E1 ID formats

Changes

Database Schema

• Added id_ranges table with atomic range management

Service Components

• Created EnumerationService with Wikibase-compatible IDs (Q/P/L/E)
• Built IdGeneratorWorker with Docker containerization

API Changes

• Replaced generic /entity with type-specific endpoints:
• POST /item, PUT /item/Q{id}, GET /item/Q{id}, DELETE /item/Q{id}
• POST /property, PUT /property/P{id}, GET /property/P{id}, DELETE /property/P{id}
• POST /lexeme, PUT /lexeme/L{id}, GET /lexeme/L{id}, DELETE /lexeme/L{id}
• POST /entityschema, PUT /entityschema/E{id}, GET /entityschema/E{id}, DELETE /entityschema/E{id}
• Auto-ID assignment in POST endpoints
• Permanent IDs (no reuse of deleted entity IDs)
• CRUD separation: Split handlers into Create/Read/Update/Delete classes

Horizontal Scaling

• Workers scale independently via Docker Compose

[2026-01-05] Statement Deduplication System - Complete

Summary

Implemented complete statement deduplication system across 6 phases. All statement data is now deduplicated and stored with hash-based references, enabling efficient storage and retrieval.

Phase 1: Database Schema ✅

• Added statement_content table (hash, ref_count, created_at)
• Added JSON columns to entity_revisions table (statements, properties, property_counts)
• Created hash_entity_statements() helper to parse and hash statements
• Updated VitessClient.insert_revision() to accept statements/properties/counts parameters

Phase 2: Core Write Logic ✅

• Created StatementHashResult Pydantic BaseModel
• Implemented deduplicate_and_store_statements() function:
• Checks statement_content table for existing hashes
• Writes new statements to S3 (statements/{hash}.json)
• Increments ref_count for existing statements
• Integrated deduplication into entity write path (POST /entity)
• Rapidhash computation for efficient hashing
• S3 + Vitess integration

Phase 3: Core Read Logic ✅

Statement Endpoints: - GET /statement/{content_hash}: Fetch single statement by hash from S3 - POST /statements/batch: Fetch multiple statements in one request - Returns not_found list for missing hashes

Property Endpoints: - GET /entity/{id}/properties: Returns sorted list of unique property IDs - GET /entity/{id}/properties/counts: Returns dict mapping property ID -> statement count - GET /entity/{id}/properties/{property_list}: Returns statement hashes for specified properties

Most-Used Endpoint: - GET /statement/most_used: Returns statement hashes sorted by ref_count DESC - Query params: limit (1-10000, default 100), min_ref_count (default 1)

Phase 4: Property-Based Loading ✅

• Full property list support
• Property counts for intelligent loading
• Demand-fetch for specific properties

Phase 5: Analytics Support ✅

• Most-used statements endpoint
• ref_count tracking for scientific analysis

Phase 6: Cleanup Orphaned Statements ✅

New Endpoint: - POST /statements/cleanup-orphaned: Background job for periodic cleanup - Queries statement_content table for orphaned statements (ref_count=0, older_than_days) - Deletes orphaned statements from S3 and statement_content table - Returns cleaned_count, failed_count, errors list

Delete Path Updates: - Updated DELETE /entity path for hard delete - Decrement ref_count for all statements in entity's head revision - Tracks orphaned statements for cleanup

Code Quality: - Black formatter, Python syntax check passed

[2026-01-05] RDF Testing - Redirect Support

Summary

Implemented complete redirect support for RDF generation including MediaWiki API integration, redirect cache, Vitess integration, and API endpoints. Achieved 98.4% match rate for Q42 (5197/5280 blocks match).

Test Entities Status

Changes

Redirect Support Implementation

• Created redirect_cache.py module mirroring entity_cache.py pattern
• Implemented MediaWiki API integration: Fetches entity redirects via action=query&prop=redirects
• Added TripleWriters.write_redirect() to generate owl:sameAs statements
• Updated EntityConverter with _fetch_redirects() and _write_redirects() methods
• Created redirect download script: scripts/download_entity_redirects.py
• Downloaded 18 redirect files from MediaWiki API

Database & Storage

• S3 Schema v1.1.0: Added redirects_to field to mark redirect entities
• Vitess integration: New entity_redirects table with bidirectional indexing

API Endpoints

• POST /redirects: Create redirects
• POST /entities/{id}/revert-redirect: Revert redirects using revision-based restore

Immutable Revision Pattern

• Redirects are minimal tombstone S3 snapshots
• Can be reverted with new revisions

Benefits

• Perfect Q42 match achieved: 5280 blocks matching golden TTL
• Match rate improved to 98.4% (5197/5280 blocks match)
• Only 83 value node hash differences remaining
• Test suite created: Comprehensive tests for redirect creation, validation, and reversion

[2025-12-31 to 2026-01-01] RDF Testing - Bug Fixes and Improvements

Summary

Multiple phases of fixes to align RDF output with Wikidata format, including datatype mapping, normalization support, property metadata fixes, critical bug fixes, data model alignment, and entity metadata fixes.

Phase 1: Datatype Mapping

• Added get_owl_type() helper to map property datatypes to OWL types
• Non-item datatypes now generate owl:DatatypeProperty instead of owl:ObjectProperty

Phase 2: Normalization Support

• Added psn:, pqn:, prn:, wdtn: predicates for properties with normalization
• Added wikibase:statementValueNormalized, wikibase:qualifierValueNormalized, wikibase:referenceValueNormalized, wikibase:directClaimNormalized declarations
• Supports: time, quantity, external-id datatypes

Phase 3: Property Metadata

• Updated PropertyShape model to include normalized predicates
• Fixed blank node generation to use MD5 with proper repository name (wikidata)
• Fixed missing properties: Now collects properties from qualifiers and references, not just main statements

Phase 4: Critical Bug Fixes (Dec 31)

• Fixed reference snaks iteration: Changed ref.snaks.values() to ref.snaks (list, not dict)
• Fixed URI formatting: Removed angle brackets from prefixed URIs (<wds:...> → wds:...)
• Fixed reference property shapes: Each reference snak now uses its own property shape
• Fixed time value formatting: Strips "+" prefix to match Wikidata format
• Fixed globe precision formatting: Changed "1e-05" to "1.0E-5"
• Fixed hash serialization: Updated to include all fields (before/after for time, formatted precision for globe)
• Fixed property declarations: psv:, pqv:, prv: now declared for all properties
• Fixed qualifier entity collection: Entities referenced in qualifiers are now written to TTL
• Downloaded 59 entity metadata files from Wikidata SPARQL

Phase 5: Data Model Alignment (Dec 31)

• Fixed globe precision format: Implemented _format_scientific_notation() to remove leading zeros from exponents (e.g., "1.0E-05" → "1.0E-5")
• Fixed time hash serialization: Preserves "+" prefix in hash but omits before/after when 0 for consistency with Wikidata format
• Fixed OWL property types: psv:, pqv:, prv: are always owl:ObjectProperty; wdt: follows datatype (ObjectProperty for items, DatatypeProperty for literals)
• Updated test expectations: Aligned tests with golden TTL format from Wikidata

Phase 6: Entity Metadata Fix (Jan 1)

• Fixed entity metadata download script: Updated to collect referenced entities from qualifiers and references, not just mainsnaks
• Fixed entity ID extraction: Changed from numeric-id to id field for consistency with conversion logic
• Downloaded 557 entity metadata files from Wikidata SPARQL endpoint
• Improved Q42 conversion: Reduced missing blocks from 147 to 87 by adding 60 previously missing entity metadata files

Integration Test Status

• ✅ Property ontology tests (fixed OWL type declarations)
• ✅ Globe precision formatting (matches golden TTL: "1.0E-5")
• ✅ Time value serialization (preserves + prefix, omits before/after when 0)
• ✅ Redirect support (MediaWiki API integration, owl:sameAs statements for Q42's 4 redirects)

Remaining Issues

• Value node hashes (different serialization algorithm - non-critical)
• Q42: 83 value node hash differences remain (1.6% mismatch - all redirect issues resolved)

[2026-02-02] Merge LexemeUpdateHandler into EntityUpdateHandler for Transaction Safety

Summary

Merged LexemeUpdateHandler into EntityUpdateHandler to ensure lexeme term processing (form representations and sense glosses) happens within the transaction scope. This fixes a data integrity issue where S3 storage of lexeme terms occurred before the transaction started, leaving orphaned data on transaction failure.

Motivation

• Data Integrity: Previous implementation stored lexeme terms to S3 before transaction began, causing orphaned data on rollback
• Transaction Safety: Ensure all S3 lexeme term operations are rolled back with Vitess changes
• Code Consolidation: Remove duplicate code in LexemeUpdateHandler
• Consistency: Align lexeme updates with entity update transaction pattern

Changes

UpdateTransaction Enhancements

File: src/models/rest_api/entitybase/v1/handlers/entity/update_transaction.py

• Added lexeme_term_operations: list[Callable[[], None]] field to track lexeme S3 operations for rollback
• Added process_lexeme_terms(forms, senses) method to process forms and senses and store to S3
• Added _rollback_form_representation(hash_val) method to delete form representations from S3 on rollback
• Added _rollback_sense_gloss(hash_val) method to delete sense glosses from S3 on rollback
• Updated commit() to clear lexeme_term_operations
• Updated rollback() to process lexeme term operations in reverse order before other operations

EntityUpdateHandler New Method

File: src/models/rest_api/entitybase/v1/handlers/entity/update.py

• Added update_lexeme(entity_id, request, edit_headers, validator) method that:
• Validates lexeme ID format (L\d+)
• Checks entity exists/deleted/locked status
• Creates UpdateTransaction
• Processes lexeme terms within transaction (S3 storage)
• Processes statements within transaction
• Creates revision within transaction
• Publishes event within transaction
• Commits/rolls back both Vitess and S3 changes atomically

Endpoint Updates

File: src/models/rest_api/entitybase/v1/endpoints/lexemes.py

• Removed import of LexemeUpdateHandler
• Added import of InternalEntityUpdateRequest for type compatibility
• Updated all lexeme update endpoints to use EntityUpdateHandler and update_lexeme method:
• update_form_representation() endpoint
• update_sense_gloss() endpoint
• delete_form() endpoint
• delete_sense() endpoint

Removed Files

• Deleted src/models/rest_api/entitybase/v1/handlers/entity/lexeme/update.py (LexemeUpdateHandler)
• Removed duplicate lexeme-specific update logic

Documentation

File: doc/DIAGRAMS/WRITE-PATHS/LEXEME-UPDATE-PROCESS.md

• Created comprehensive documentation of lexeme update transaction flow
• Documents lexeme term processing within transaction scope
• Documents rollback behavior for both S3 and Vitess changes

Transaction Flow After Fix

1. EntityUpdateHandler.update_lexeme
   ↓
2. Create UpdateTransaction
   ↓
3. Within transaction try block:
   a. tx.process_lexeme_terms() → Store form representations and sense glosses to S3
   b. tx.process_statements() → Store statements
   c. tx.create_revision() → Create revision
   d. tx.publish_event() → Publish event
   ↓
4. If success: tx.commit() → Clear rollback operations
   ↓
5. If failure: tx.rollback() →
   - Rollback lexeme terms (delete from S3)
   - Rollback statements (decrement ref_count, delete from S3 if orphaned)
   - Rollback revision (delete from entity_revisions)

Benefits

• Data Integrity: S3 lexeme term data is cleaned up on transaction rollback
• Atomicity: All lexeme update operations (S3 + Vitess) succeed or fail together
• Simplicity: Consolidated lexeme update logic into single handler class
• Maintainability: Single source of truth for entity update transaction pattern
• Consistency: Lexeme updates follow same transaction safety guarantees as entity updates

Notes

• Breaking Change: LexemeUpdateHandler class removed, endpoints now use EntityUpdateHandler.update_lexeme
• No Migration: Existing S3 data remains valid; only new lexeme updates benefit from transaction safety
• Performance: No performance impact; S3 operations simply moved into transaction scope

[2026-01-28] Fix entity_type query issue by using pattern matching

Summary

Fixed AttributeError: 'VitessClient' object has no attribute 'list_entities_by_type' by implementing missing method and updating entity type queries to use pattern matching on entity_id instead of querying a non-existent entity_type column.

Motivation

• Bug Fix: Admin handler was calling non-existent method
• Correctness: The entity_type column doesn't exist in entity_revisions table
• Performance: Use pattern matching on entity_id instead of joins to derived fields

Changes

VitessClient (src/models/infrastructure/vitess/client.py:127-140)

• Added list_entities_by_type(entity_type, limit, offset) method
• Uses SQL LIKE pattern matching on entity_id:
• item → Q%
• lexeme → L%
• property → P%

ListingRepository (src/models/infrastructure/vitess/repositories/listing.py)

• Added _get_entity_type_from_id(entity_id) helper method to derive type from ID pattern
• Removed entity_revisions joins in list_locked(), list_semi_protected(), list_archived(), list_dangling(), and _list_entities_by_edit_type()
• Fixed queries to derive entity_type from entity_id after retrieval

GeneralStatsService (src/models/rest_api/entitybase/v1/services/general_stats_service.py)

• Fixed get_total_items() to count entity_id LIKE 'Q%'
• Fixed get_total_lexemes() to count entity_id LIKE 'L%'
• Fixed get_total_properties() to count entity_id LIKE 'P%'

[2026-01-28] Consolidate Edit Headers in Handlers

Summary

Replaced separate edit_summary: str and user_id: int parameters across all handler methods with a single edit_headers: EditHeaders parameter for consistency and improved type safety.

Motivation

• Consistency: Standardize how edit metadata (user ID and summary) is passed between layers
• Type Safety: Use the existing EditHeaders BaseModel instead of loose parameters
• Maintainability: Single parameter instead of two independent parameters

Changes

Handler Layer (9 files)

File: src/models/rest_api/entitybase/v1/handlers/entity/handler.py - Updated process_entity_revision_new() signature: replaced edit_summary: str with edit_headers: EditHeaders - Updated add_property() signature: replaced edit_summary: str, user_id: int with edit_headers: EditHeaders - Updated remove_statement() signature: replaced edit_summary: str, user_id: int with edit_headers: EditHeaders - Updated patch_statement() signature: replaced edit_summary: str, user_id: int with edit_headers: EditHeaders - Updated internal usage to access edit_headers.x_edit_summary and edit_headers.x_user_id

File: src/models/rest_api/entitybase/v1/handlers/entity/create.py - Updated create_entity() signature: replaced edit_summary: str, user_id: int with edit_headers: EditHeaders - Updated internal usage to access edit_headers.x_user_id for logging

File: src/models/rest_api/entitybase/v1/handlers/entity/property/create.py - Updated create_entity() signature to match parent class - Updated method call to pass edit_headers instead of separate parameters

File: src/models/rest_api/entitybase/v1/handlers/entity/lexeme/create.py - Updated create_entity() signature to match parent class - Updated method call to pass edit_headers instead of separate parameters

File: src/models/rest_api/entitybase/v1/handlers/entity/delete.py - Updated delete_entity() signature: replaced user_id: int = 0 with edit_headers: EditHeaders - Updated internal usage to access edit_headers.x_user_id and edit_headers.x_edit_summary

File: src/models/rest_api/entitybase/v1/handlers/entity/redirect.py - Updated create_entity_redirect() signature: replaced edit_summary: str, user_id: int with edit_headers: EditHeaders - Updated revert_entity_redirect() signature: replaced edit_summary: str, user_id: int with edit_headers: EditHeaders - Updated internal usage to access edit_headers.x_edit_summary and edit_headers.x_user_id

File: src/models/rest_api/entitybase/v1/handlers/entity/revert.py - Updated revert_entity() signature: replaced user_id: int, edit_summary: str with edit_headers: EditHeaders - Updated internal usage to access edit_headers.x_user_id and edit_headers.x_edit_summary

File: src/models/rest_api/entitybase/v1/handlers/entity/update.py - Updated update_entity() signature: removed user_id: int = 0 parameter (user ID accessed from request.edit_headers.x_user_id) - Updated create_revision() and publish_event() calls to pass edit_headers: request.edit_headers - Updated logging to use request.edit_headers.x_user_id

File: src/models/rest_api/entitybase/v1/handlers/entity/creation_transaction.py - Updated create_revision() signature: replaced edit_summary: str, user_id: int with edit_headers: EditHeaders - Updated publish_event() signature: replaced user_id: int = 0, edit_summary: str = "" with edit_headers: EditHeaders - Updated internal usage to access edit_headers.x_edit_summary and edit_headers.x_user_id

File: src/models/rest_api/entitybase/v1/handlers/entity/update_transaction.py - Updated create_revision() signature: replaced edit_summary: str, user_id: int with edit_headers: EditHeaders - Updated publish_event() signature: replaced user_id: int = 0, edit_summary: str = "" with edit_headers: EditHeaders - Updated internal usage to access edit_headers.x_edit_summary

File: src/models/rest_api/entitybase/v1/handlers/entity/entity_transaction.py - Added EditHeaders import - Updated base class publish_event() signature to use EditHeaders parameter

Endpoint Layer (3 files)

File: src/models/rest_api/entitybase/v1/endpoints/entities.py - Updated delete_entity() handler call: pass edit_headers=headers instead of user_id=headers.x_user_id, edit_summary=headers.x_edit_summary - Updated add_property() handler call: pass edit_headers=headers - Updated remove_statement() handler call: pass headers as edit_headers - Updated patch_statement() handler call: pass headers as edit_headers - Removed user_id=headers.x_user_id parameter from update_entity() calls (3 occurrences)

File: src/models/rest_api/entitybase/v1/endpoints/redirects.py - Updated create_entity_redirect() handler call: pass edit_headers=headers - Updated revert_entity_redirect() handler call: pass edit_headers=headers

File: src/models/rest_api/entitybase/v1/endpoints/properties.py - Updated create_entity() handler call: pass edit_headers=headers with keyword arguments - Updated update_entity() handler call: removed user_id=headers.x_user_id parameter

File: src/models/rest_api/entitybase/v1/endpoints/lexemes.py - Updated create_entity() handler call: pass edit_headers=headers with keyword arguments

Impact

• Breaking Change: All handler method signatures changed from accepting separate edit_summary and user_id parameters to a single edit_headers: EditHeaders parameter
• Benefits:
• Reduced parameter count across all handlers
• Single source of truth for edit metadata
• Type safety through Pydantic EditHeaders model
• Consistent API across all entity operations (create, update, delete, revert, etc.)

[2026-01-28] S3 Revision Read Issue Fix

Summary

Fixed S3 revision read issue by adding content_hash column to entity_revisions table and updating the read path to query this hash before loading from S3. This ensures revision data is retrievable by the hash used as the S3 key.

Motivation

• Fix Retrieval: Original revision read path incorrectly used entity_id/revision_id as S3 key, causing 404 errors
• Consistency: Align S3 storage and retrieval to use the same hash-based key
• Data Integrity: Ensure revision data is always retrievable via stored content_hash

Changes

Database Schema

• File: src/models/infrastructure/vitess/repositories/schema.py
• Change: Added content_hash BIGINT UNSIGNED NOT NULL column to entity_revisions table (line 165)

Repository Layer

• File: src/models/infrastructure/vitess/repositories/revision.py
• Changes:
• Updated create() method to accept and store content_hash parameter (line 191, 210-226)
• Updated create_with_cas() method to accept and store content_hash parameter (line 130, 148-164)
• Updated insert_revision() to pass content_hash through to create methods (line 31)
• Added new get_content_hash() method to retrieve content_hash for a specific revision (line 128)

S3 Client Layer

• File: src/models/infrastructure/s3/client.py
• Change: Updated read_revision() to:
• Resolve entity_id to internal_id
• Query content_hash from database via RevisionRepository
• Load revision from S3 using content_hash (line 83-92)

Handler Layer

• File: src/models/rest_api/entitybase/v1/handlers/entity/handler.py
• Changes:
• Updated create_and_store_revision() to pass content_hash to create_revision() (line 475)
• Updated _create_revision_new() to pass content_hash to create_revision() (line 208)
• Implemented _store_revision_s3_new() to store revision data with content_hash (line 276)

Additional Updates

• File: src/models/infrastructure/vitess/client.py
• Updated create_revision() to accept and pass content_hash parameter (line 80)

• Updated insert_revision() to accept and pass content_hash parameter (line 108)

• File: src/models/rest_api/entitybase/v1/handlers/entity/delete.py

• Updated to calculate content_hash and pass to both store_revision and create_revision (line 151-164)

• File: src/models/rest_api/entitybase/v1/handlers/entity/revert.py

• Updated to calculate content_hash and use correct RevisionData parameter (line 122-136)

• File: src/models/rest_api/entitybase/v1/services/redirects.py

• Updated to calculate content_hash and pass to store_revision and create_revision (line 80-89)

Implementation Details

The content_hash is now computed once during revision creation using MetadataExtractor.hash_string() and passed to both: 1. Vitess: Stored in entity_revisions.content_hash column for retrieval 2. S3: Used as the key for storing/reading revision data

When reading a revision: 1. Resolve entity_id to internal_id 2. Query entity_revisions table for content_hash 3. Load revision from S3 using content_hash as the key

Benefits

• Correct Behavior: Revisions now load from S3 using the hash-based key, matching storage behavior
• Consistency: Storage and retrieval use identical key patterns
• Minimal Breaking: NULL default for content_hash handles existing rows

[2026-01-24] Entity Schema 2.0.0 - Hash-Based Reference Architecture

Summary

Implemented entity schema version 2.0.0 with hash-based references for all content including labels, descriptions, aliases, statements, and sitelinks. This schema enables efficient deduplication and storage by referencing stored content via integer hashes instead of inline objects. Added comprehensive documentation with mock JSON examples and configured docker-compose environment variable.

Motivation

• Deduplication: Enable sharing of identical content (labels, statements, etc.) across entities using hash references
• Storage Efficiency: Reduce storage requirements by ~90% through content deduplication
• Consistency: Align entity schema with existing revision/statement/sitelink schema pyramid
• Scalability: Support trillion-scale storage with minimal overhead

Changes

New Schema Version

• File: schemas/entitybase/entity/2.0.0/schema.yaml
• Type: User-facing entity response schema
• Structure: All content fields use hash integer references instead of inline objects
• Fields: Core revision fields, status flags, hash references, minimal entity metadata

Key Differences from 1.0.0

• Removed: Inline $defs for datavalue, snak, statement, reference (no longer needed)
• Updated Claims: Changed from inline statement objects to hash arrays: {"P31": [123456, 789012]}
• Updated Sitelinks: Changed from {title, site, badges} to {title_hash, badges}
• Updated Terms: Labels/descriptions/aliases now hash integers instead of value objects

Documentation

• File: schemas/entitybase/entity/2.0.0/README.md
• Sections: Key changes, mock JSON example, field descriptions, schema pyramid, usage examples
• Includes comprehensive examples of hash-based reference structure

Configuration

• File: docker-compose.yml and docker/docker-compose.yml
• Added SCHEMA_ENTITY_VERSION: 2.0.0 environment variable
• Positioned after SCHEMA_ENTITYCHANGE_VERSION: 1.0.0 for clarity

Schema Pyramid Structure

Level 1: entity/2.0.0 (user response - all hash references)
    ↓
Level 2: revision/4.0.0 (S3 storage with metadata + hash references)
    ↓
Level 3: statement/3.0.0, sitelink/1.0.0 (deduplicated content)
    ↓
Level 4: snak/1.0.0, reference/1.0.0, qualifier/1.0.0 (atomic objects)

Mock Example Response

{
  "schema_version": "2.0.0",
  "id": "Q123",
  "type": "item",
  "revision_id": 123456789,
  "created_at": "2026-01-24T19:30:00Z",
  "created_by": "ExampleUser",
  "labels": {"en": 123456789, "de": 234567890},
  "descriptions": {"en": 345678901},
  "aliases": {"en": [456789012, 567890123]},
  "sitelinks": {"enwiki": {"title_hash": 678901234, "badges": []}},
  "statements": {"P31": [789012345], "P569": [890123456]}
}

Benefits

• Storage Efficiency: Identical content across entities shares hash references
• Version Control: Hash-based references enable content-level versioning
• API Performance: Lightweight responses for entity metadata
• Caching: Hash-based keys ideal for CDN/edge caching
• Comparison: Quick equality checks using content_hash

[2026-01-24] SnakHandler Integration into Statement Processing

Summary

Integrated SnakHandler into the statement processing pipeline to enable snak deduplication across statement mainsnaks, qualifiers, and references. This completes the deduplication architecture by using hash references for all snak data, reducing storage redundancy.

Changes

Statement Storage Integration

• StatementService Updates: Modified deduplicate_and_store_statements() to extract mainsnaks and store via SnakHandler before statement storage
• Hash Reference Replacement: Replaced embedded mainsnak objects with hash references ({"hash": int}) in stored statements
• Logging: Added debug logging for snak storage and hash references

Statement Retrieval Updates

• Mainsnak Reconstruction: Updated get_statement(), get_statements_batch(), and get_entity_property_hashes() to reconstruct full snaks from hash references
• No Backward Compatibility: Removed support for embedded snaks - only hash references are supported
• SnakHandler Usage: Consistent use of SnakHandler.get_snak() for reconstruction across all retrieval methods

Qualifier/Reference Processing

• deduplicate_references_in_statements(): Extended to extract and store snaks within references using SnakHandler
• deduplicate_qualifiers_in_statements(): Extended to extract and store snaks within qualifiers using SnakHandler
• Hash References: Both qualifiers and references now store snaks as hash integers instead of full snak objects

Endpoint Updates

• references.py: Updated get_references() endpoint to reconstruct snaks from hash references in returned data
• qualifiers.py: Updated get_qualifiers() endpoint to reconstruct snaks from hash references in returned data
• Snak Reconstruction: Both endpoints use SnakHandler to expand hash-referenced snaks for API responses

Implementations

• statement_service.py (line 118-142): Added mainsnak extraction, SnakHandler store_snak() call, and hash reference replacement
• handlers/statement.py (line 27-95): Added snak reconstruction in get_statement(), get_statements_batch(), and get_entity_property_hashes()
• endpoints/references.py: Added snak reconstruction logic in get_references()
• endpoints/qualifiers.py: Added snak reconstruction logic in get_qualifiers()

Testing

• New Test Files:
• tests/unit/models/rest_api/entitybase/v1/services/test_statement_service_snak_integration.py: Unit tests for statement service snak deduplication
• tests/unit/models/rest_api/entitybase/v1/handlers/test_statement_snak_reconstruction.py: Unit tests for statement handler snak reconstruction

Test Coverage

• Snak extraction and storage in statement processing
• Mainsnak hash reference replacement in stored statements
• Snak reconstruction from hashes in statement retrieval
• Qualifier and reference snak processing
• Missing snak error handling
• Batch snak reconstruction

Benefits

• Storage Efficiency: Snaks are deduplicated across all statements, qualifiers, and references
• Consistency: Aligns with existing qualifier and reference deduplication architecture
• Complete Deduplication: All statement components (mainsnak, qualifier snaks, reference snaks) now use hash-based deduplication
• API Consistency: Frontend receives fully reconstructed snak data in all responses

Notes

• No Migration: Only new statements use hash-referenced snaks; existing embedded snaks are no longer supported
• Enforcement: All snak storage and retrieval enforces hash references only

[2026-01-22] Snaks Deduplication and REST API Endpoint

Summary

Extended deduplication to snaks in Wikibase statements using rapidhash. Snaks are now stored in a dedicated S3 bucket with hash-based keys, reducing storage for repetitive snak objects. Added new REST API endpoint for fetching deduplicated snaks by hash.

Changes

Snak Deduplication Implementation

• S3 Storage: Created SnakStorage class for storing/retrieving snaks with rapidhash keys in s3_snaks_bucket
• Data Model: Added S3SnakData model for snak storage with schema version, snak content, hash, and timestamp
• Client Integration: Extended MyS3Client with load_snaks_batch method for efficient batch retrieval

REST API Endpoint

• New Endpoint: GET /snaks/{hashes} - Fetch snaks by hash(es) with batch support (max 100 hashes)
• Response Model: Added SnakResponse with snak data, content hash, and creation timestamp
• Error Handling: Validates hash format, enforces batch limits, returns null for missing snaks
• OpenAPI Tags: Grouped under "statements" tag for consistency with qualifiers/references

Benefits

• Storage Efficiency: Reduces storage for repetitive snak objects across statements, qualifiers, and references
• API Performance: Enables frontend caching and batch retrieval of snak data
• Consistency: Aligns snaks with existing qualifier/reference deduplication architecture

[2026-01-19] Internal Data Models and RDF Builder Updates

Summary

Introduced internal EntityData model for RDF processing, separated parsing logic for API vs internal use, and updated RDF converter to use structured internal data instead of API models. Added diff classes and incremental RDF updater for efficient updates.

Changes

Data Model Improvements

• Added EntityData Model: New internal representation with nested structures for labels, descriptions, aliases, statements, and sitelinks, avoiding API model dependencies in internal code
• Fixed parse_entity: Corrected return type to consistently return EntityMetadataResponse for API use
• Added parse_entity_data: New function for parsing raw JSON into EntityData for internal/RDF processing

RDF Builder Updates

• Updated EntityConverter: Modified to accept EntityData instead of Entity, enabling use of structured internal data
• Added Diff Classes: StatementDiff, TermsDiff, SitelinksDiff for computing changes between entity versions
• Added IncrementalRDFUpdater: Separate class for applying diffs to RDF output incrementally, avoiding full rebuilds
• Added Tests: Comprehensive tests for EntityData, parse_entity_data, diff classes, and IncrementalRDFUpdater
• Improved Type Safety: Resolved type mismatches between parsing and conversion layers

[2026-01-18] API Fixes and Endpoint Removal

Summary

Fixed various API issues including missing request fields, validation problems, and S3 bucket handling. Removed the bulk sitelinks update endpoint to simplify the API surface.

Changes

Entity Creation API Fixes

• Fixed EntityCreateRequest: Added missing is_semi_protected, is_locked, is_archived, is_dangling, and is_mass_edit_protected fields to prevent attribute errors
• Fixed PropertyCounts Validation: Added default_factory=dict to ensure proper Pydantic validation for empty counts

S3 Client Improvements

• Fixed delete_metadata: Implemented proper bucket determination logic based on metadata type (terms vs sitelinks)

API Simplification

• Removed Endpoint: PUT /entitybase/v1/entities/{entity_id}/sitelinks - Bulk sitelinks update endpoint
• Rationale: Simplified API by removing redundant bulk operation; individual sitelink operations remain available
• Impact: Reduces API surface area while maintaining functionality through existing per-sitelink endpoints

[2026-01-18] Watchlist Entry Removal by ID and Statistics APIs

Summary

Added new endpoint for removing watchlist entries by numeric ID for simpler API usage. Implemented daily statistics workers for user and general wiki data, storing results in database tables for fast API retrieval. Added endpoints for user stats and comprehensive wiki statistics with breakdowns.

Changes

Watchlist API Enhancement

• New Endpoint: DELETE /entitybase/v1/users/{user_id}/watchlist/{watch_id} - Remove watchlist entry by its numeric ID
• Purpose: Provides a simpler RESTful way to remove watches using the auto-incremented ID returned in watchlist responses
• Error Handling: Returns 404 if the watch ID doesn't exist

[2026-01-18] User and General Statistics Workers and APIs

Summary

Implemented daily statistics workers for user and general wiki data, storing results in database tables for fast API retrieval. Added endpoints for user stats and comprehensive wiki statistics with breakdowns.

Changes

Storage Architecture

• New Tables:
• user_daily_stats: stat_date (DATE PRIMARY KEY), total_users, active_users, created_at.
• general_daily_stats: stat_date (DATE PRIMARY KEY), total_statements, total_qualifiers, total_references, total_items, total_lexemes, total_properties, total_sitelinks, total_terms, terms_per_language (JSON), terms_by_type (JSON), created_at.
• Active User Definition: Users with last_activity within the last 30 days.
• Terms: Total labels + descriptions + aliases, with breakdowns by language and type.

Worker Implementation

• Base Stats Worker: Created BaseStatsWorker class for reusable stats worker logic (scheduling, health checks).
• User Stats Worker: Computes and stores daily user stats.
• General Stats Worker: Computes wiki-wide stats including statements, qualifiers, references, entities, sitelinks, and term breakdowns.
• Scheduling: Both workers configurable via *_stats_schedule (default "0 2 * * *" - daily at 2 AM).

API Updates

• New Endpoints:
• GET /entitybase/v1/users/stat: Returns UserStatsResponse with user counts.
• GET /entitybase/v1/stats: Returns GeneralStatsResponse with comprehensive wiki stats and breakdowns.
• Watchlist CRUD endpoints (tagged "watchlist"):
  • POST /entitybase/users/{user_id}/watchlist (add watch).
  • POST /entitybase/users/{user_id}/watchlist/remove (remove watch).
  • GET /entitybase/users/{user_id}/watchlist (get watches).
  • GET /entitybase/users/{user_id}/watchlist/notifications (get notifications).
  • PUT /entitybase/users/{user_id}/watchlist/notifications/{notification_id}/check (mark checked).
  • GET /entitybase/users/{user_id}/watchlist/stats (get watch counts).
• Response Models: Added UserStatsData, UserStatsResponse, GeneralStatsData, GeneralStatsResponse in misc.py.
• Handler: Added get_user_stats and get_general_stats in UserHandler to query tables (with live fallbacks).

Implementation Details

• Services: UserStatsService and GeneralStatsService compute live stats from Vitess.
• Repository: Extended UserRepository with insert_user_statistics and insert_general_statistics methods.
• Settings: Added user_stats_enabled, user_stats_schedule, general_stats_enabled, general_stats_schedule flags.

Benefits

• Performance: Precomputed stats reduce query load on live data.
• Scalability: Daily batch processing handles large datasets.
• Extensibility: Base worker class enables easy addition of other stats workers.

[2026-01-18] Qualifier Deduplication Implementation

Summary

Extended deduplication to qualifiers in Wikibase statements using rapidhash. Qualifiers are now stored in the same S3 bucket with hash-based keys, reducing storage for repetitive qualifier sets. Updated statement schema to use hash pointers for qualifiers.

Changes

Storage Architecture

• Shared S3 Bucket: Uses wikibase-references for qualifiers (e.g., qualifiers/123456789).
• Hash Computation: Added QualifierHasher using rapidhash for qualifier content.
• Deduplication Logic: Modified deduplicate_qualifiers_in_statements to extract, hash, store, and replace qualifiers in statements.

API Updates

• Schema Update: Updated statement schema to 3.0.0; qualifiers now rapidhash integer instead of object.
• New Endpoints:
• GET /references/qualifiers/{hash}: Fetch single qualifiers by hash.
• GET /references/qualifiers/{hash1},{hash2},...: Batch fetch (up to 100 hashes), returns array with nulls for missing.
• Response Changes: Statement responses now include qualifier hashes; frontend must expand via endpoints.

Implementation Details

• S3 Client Extensions: Added store_qualifier, load_qualifier, load_qualifiers_batch.
• Statement Processing: Integrated qualifier deduplication into deduplicate_and_store_statements.
• No Migration: Only new qualifiers are deduplicated; existing statements unchanged.

Benefits

• Space Savings: Eliminates duplicate qualifier storage across statements.
• Consistency: Aligns with reference deduplication.
• Integrity: Rapidhash ensures content verification.

[2026-01-18] Reference Deduplication Implementation

Summary

Implemented reference deduplication for Wikibase statements using rapidhash. References are now stored in a dedicated S3 bucket with hash-based keys, reducing storage for repetitive citations. Updated statement schema to use hash pointers, added new API endpoints for frontend lookup.

Changes

Storage Architecture

• New S3 Bucket: wikibase-references for storing unique reference JSON keyed by rapidhash (e.g., references/123456789).
• Hash Computation: Added ReferenceHasher using rapidhash for reference content.
• Deduplication Logic: Modified deduplicate_references_in_statements to extract, hash, store, and replace references in statements.

API Updates

• Schema Update: Updated statement schema from 1.0.0 to 2.0.0; references now array of rapidhash integers instead of full objects.
• New Endpoints:
• GET /references/{hash}: Fetch single reference by hash.
• GET /references/{hash1},{hash2},...: Batch fetch (up to 100 hashes), returns array with nulls for missing.
• Response Changes: Statement responses now include reference hashes; frontend must expand via endpoints.

Implementation Details

• S3 Client Extensions: Added store_reference, load_reference, load_references_batch.
• Statement Processing: Integrated reference deduplication into deduplicate_and_store_statements.
• No Migration: Only new references are deduplicated; existing statements unchanged.

Benefits

• Space Savings: Eliminates duplicate reference storage across statements.
• Scalability: Supports trillion-scale references with hash-based keys.
• Integrity: Rapidhash ensures content verification.

[2026-01-17] Entity Change Event Improvements & New Schema

Summary

Enhanced entity change event publishing to use user_id for better tracking, added new event types for endorsements and thanks, and created corresponding schemas for API responses.

Changes

Event Publishing Updates

• User ID Integration: Modified publish_event in UpdateTransaction to accept user_id instead of editor, improving change attribution
• New Event Types: Added EndorseChangeEvent and NewThankEvent for endorsement and thanks actions
• Schema Addition: Created new EntityChange response schema in entitybase/response/entity/change.py for standardized change event data
• Docstring Enhancement: Improved publish_event docstring with detailed Args, Returns, and notes on user_id usage

API Response Schema

• New Models: EntityChange, EndorseChangeEvent, and NewThankEvent Pydantic models for change events
• Field Details: Includes entity_id, revision_id, change_type, timestamps, and edit summary; uses aliases for compact JSON
• Event Support: Now supports endorsement changes (endorse/withdraw) and new thanks with dedicated event types

[2026-01-17] Endorsement Stats Optimization & Caching

Summary

Optimized endorsement statistics delivery by merging stats into listing endpoints and adding cache-friendly lightweight stats endpoints. Slimmed response field names and removed redundant data to reduce JSON payload sizes for better caching performance.

Changes

API Enhancements

• Embedded Stats: GET /statements/{hash}/endorsements now includes stats metadata with total/active/withdrawn counts
• Lightweight Endpoint: GET /statements/{hash}/endorsements/stats provides stats-only responses (~99% size reduction)
• Slimmed Fields: Response fields renamed from total_endorsements to total, active_endorsements to active, etc.
• Field Optimization: Renamed statement_hash to hash and StatementEndorsement to Endorsement for concise responses
• Removed Redundancy: Single stats endpoint omits redundant fields

Response Structure Updates

// Listing endpoint (with embedded stats)
{
  "statement_hash": 12345,
  "endorsements": [...],
  "total_count": 42,
  "has_more": false,
  "stats": {
    "total": 50,
    "active": 42,
    "withdrawn": 8
  }
}

// Lightweight stats endpoint
{
  "total": 50,
  "active": 42,
  "withdrawn": 8
}

Performance Improvements

• Cache-Friendly: Lightweight endpoint reduces JSON size from ~10KB to ~100 bytes
• Efficient Queries: Stats calculated alongside main queries to avoid extra round trips
• Optimized Payload: Removed redundant fields and verbose naming

[2026-01-17] Statement Endorsements with Revocation Support

Summary

Implemented comprehensive statement endorsement system allowing users to express trust in Wikibase statements and their references. Users can endorse statements to signal credibility, with full support for withdrawing endorsements. Includes pagination, statistics, and activity tracking.

Motivation

• Trust Signals: Enable users to signal confidence in statement accuracy and references
• Quality Indicators: Provide social validation for statement trustworthiness
• Community Curation: Allow peer review and validation of claims
• Revocable Actions: Support for changing opinions or correcting mistakes

Changes

New Components

• EndorsementRepository in src/models/infrastructure/vitess/endorsement_repository.py for database operations with soft deletion
• EndorsementHandler in src/models/rest_api/entitybase/handlers/endorsements.py for API business logic
• StatementEndorsement models in src/models/endorsements.py and response models
• Soft deletion support with removed_at timestamp for revocable endorsements

API Endpoints

• POST /entitybase/v1/statements/{hash}/endorse - Create endorsement
• DELETE /entitybase/v1/statements/{hash}/endorse - Withdraw endorsement
• GET /entitybase/v1/statements/{hash}/endorsements - List statement endorsements with embedded stats (paginated)
• GET /entitybase/v1/users/{id}/endorsements - List user's endorsements (paginated)
• GET /entitybase/v1/users/{id}/endorsements/stats - Get endorsement statistics
• GET /entitybase/v1/statements/{hash}/endorsements/stats - Get lightweight endorsement stats for single statement

Database Schema

• Added user_statement_endorsements table with soft deletion via removed_at field
• Foreign key constraint to statement_content table
• Unique constraint prevents duplicate endorsements per user-statement pair
• Proper indexing for efficient queries and pagination

Business Logic

• Endorsement Creation: Validates statement exists, prevents duplicates, tracks activity
• Endorsement Withdrawal: Soft deletes endorsements, allows re-endorsement
• Statistics: Total endorsements given/received, active vs. historical counts
• Pagination: Efficient database-level pagination with total counts

Activity Integration

• Added ENDORSEMENT_GIVEN and ENDORSEMENT_WITHDRAWN activity types
• Full integration with user activity tracking system

[2026-01-17] Thanks Feature for Entity Revisions

Summary

Implemented comprehensive "thank you" functionality allowing users to thank others for specific entity revision contributions. Includes full API endpoints for sending and listing thanks, database schema for thank tracking, and integration with existing user activity system.

Motivation

• Community Building: Enable social recognition for contributions similar to Wikipedia's thanks feature
• User Engagement: Provide positive feedback mechanism for editors
• Activity Tracking: Extend user activity system with social interactions

Changes

New Components

• ThanksRepository in src/models/infrastructure/vitess/thanks_repository.py for database operations
• ThanksHandler in src/models/rest_api/entitybase/handlers/thanks.py for API logic
• ThankItem and Thank models in src/models/thanks.py
• Request/response models in src/models/rest_api/entitybase/request/thanks.py and response/thanks.py

API Endpoints

• POST /entitybase/v1/entities/{entity_id}/revisions/{revision_id}/thank - Send thank for revision
• GET /entitybase/v1/users/{user_id}/thanks/received - List thanks received by user
• GET /entitybase/v1/users/{user_id}/thanks/sent - List thanks sent by user
• GET /entitybase/v1/entities/{entity_id}/revisions/{revision_id}/thanks - List thanks for specific revision

Database Schema

• Added user_thanks table with proper indexing and foreign key constraints
• Uses internal_entity_id for efficient joins with entity_id_mapping
• Unique constraint prevents duplicate thanks for same revision
• Time-based indexing for efficient chronological queries

User Activity Integration

• Added THANK_SENT and THANK_RECEIVED activity types to ActivityType enum
• Thanks events recorded in user activity system for analytics

Validation & Security

• Prevents self-thanks and duplicate thanks
• Validates user existence and revision availability
• Proper error handling with descriptive messages

[2026-01-14] Database Schema and ID Fixes

Summary

Added dropworker for clean DB state in tests, fixed ID collision issues with unique ID generation using UUID, enhanced entity creation with optional ID support, and updated database schema to use BIGINT UNSIGNED for internal IDs to support full 64-bit range.

Motivation

• Test Reliability: Ensure clean database state for integration tests
• ID Uniqueness: Prevent collisions in entity creation
• API Flexibility: Allow specifying entity IDs in creation requests

Changes

New Components

• DropWorker in src/models/workers/drop_worker.py for resetting database tables at startup
• Unique ID generator using UUID in src/models/infrastructure/unique_id.py

API Enhancements

• Optional id field in EntityCreateRequest for specifying entity IDs
• Idempotent ID assignment in entity creation handlers

Schema Changes

• Updated all internal_id and related BIGINT columns to BIGINT UNSIGNED to support full 64-bit unsigned range
• Affected tables: entity_id_mapping, entity_head, entity_redirects, entity_backlinks, entity_revisions, id_ranges, watchlist

Fixes

• Resolved test ID collisions by using UUID-based unique ID generation
• Improved ID range management with time-based offsets
• Database reset on container startup prevents persistent state issues
• Fixed BIGINT overflow errors for large internal IDs

Testing

• Updated integration tests to use unique entity IDs
• Added dropworker to test docker-compose for clean state

[2026-01-14] Entity Diffing System with RDF Canonicalization & Streaming

Summary

Implemented complete entity diffing system with URDNA2015 RDF canonicalization, supporting stateless triple-level diffs between entity versions. Added RDF/JSON revision endpoints for retrieving entity data in multiple formats, and RDF change event streaming following MediaWiki recentchange schema. Fixed watchlist table PRIMARY KEY constraint issue.

Motivation

• Change Tracking: Enable precise tracking of what changed between entity revisions
• RDF Standardization: Use W3C-standard canonicalization for consistent blank node handling
• API Flexibility: Provide both RDF and JSON access to entity revisions
• Performance: Stateless processing with millisecond response times

Changes

New Components

• EntityDiffWorker in src/models/workers/entity_diff_worker.py with URDNA2015 canonicalization
• RDFSerializer for converting Wikibase entity data to RDF formats
• RDF and JSON revision endpoints in src/models/rest_api/entitybase/v1/entities.py
• Comprehensive unit tests and integration test scripts

Configuration

• Added pyld dependency for JSON-LD canonicalization
• Multiple canonicalization methods: URDNA2015, skolemization, structural hashing

API Endpoints

• GET /entities/{entity_id}/revision/{revision_id}/rdf - RDF serialization
• GET /entities/{entity_id}/revision/{revision_id}/json - Raw JSON data

Streaming & Events

• RDFChangeEvent model following MediaWiki recentchange schema
• RDF change event streaming to wikibase.entity_diff Kafka topic
• Automatic event publishing from EntityDiffWorker
• Configurable RDF stream producer with proper lifecycle management
• Support for turtle, rdfxml, ntriples formats

Testing

• Canonicalization test script with real Wikidata entity data
• Unit tests for diff computation and RDF serialization
• Integration tests for term deduplication workflows

Fixes

• Resolved watchlist table PRIMARY KEY constraint error by changing watched_properties from nullable TEXT to NOT NULL TEXT with empty string default
• Updated watchlist repository queries to use empty string for "watch all properties" instead of NULL
• Maintained API compatibility while fixing database schema constraints

[2026-01-14] Code Quality and Linting Improvements

Summary

Comprehensive linting system improvements with radon duplicate detection, vulture dead code analysis, and strategic allowlisting. Resolved all critical code quality issues.

Motivation

• Code Quality: Maintain high standards with automated checking
• Developer Experience: Clear error messages and fast feedback
• Maintenance: Prevent accumulation of dead code and duplicates

Changes

Linting Infrastructure

• Integrated radon for duplicate method detection
• Enhanced vulture allowlists for known API patterns
• Added comprehensive custom linting rules

Configuration

• Modular allowlist files: config/linters/allowlists/
• Separate allowlists for vulture and radon
• Strategic allowlisting of planned features vs dead code

Fixes

• Removed unreachable code in entity handlers
• Fixed type checking errors across multiple modules
• Resolved import and serialization issues

API Improvements

• Reorganized allowlists by functionality
• Added batch API endpoint allowlisting
• Wikibase v1 compatibility layer allowlisting

[2026-01-13] Kafka Consumer for Watchlist Notifications

Summary

Implemented Kafka consumer for processing entity change events and generating watchlist notifications. Added Consumer class using aiokafka, integrated into WatchlistConsumerWorker. Restored EntityUpdateRequest model to fix mypy errors.

Motivation

• Real-time Notifications: Enable users to receive notifications for watched entity changes.
• Scalability: Asynchronous event-driven processing for high-volume changes.
• Integration: Connect Wikibase events to user watchlist system.

Changes

New Components

• Consumer class in src/models/infrastructure/stream/consumer.py for Kafka event consumption.
• WatchlistConsumerWorker updated in src/models/workers/watchlist_consumer/main.py.
• Unit and integration tests for consumer functionality.

Configuration

• Added kafka_brokers and kafka_topic settings for consumer configuration.

Fixes

• Restored EntityUpdateRequest model (duplicate of EntityCreateRequest without id field) to resolve mypy import errors.
• Fixed BacklinkStatisticsWorker to properly honor backlink_stats_schedule setting instead of hardcoded 24-hour intervals.

Refactors

• Moved SQL logic from BacklinkStatisticsWorker to repository layer for better separation of concerns.

[2026-01-13] S3 Schema Updates for Full Deduplication

Summary

Updated S3 revision schema to v2.1.0 with full deduplication, storing terms, sitelinks, and statements as external hashes. Terms and sitelinks metadata stored as plain UTF-8 text for efficiency.

Motivation

• Storage Efficiency: Reduce revision size by ~90% through external deduplication.
• Scalability: Support trillion-scale storage with minimal overhead.
• Consistency: Align all metadata (terms, sitelinks, statements) under hash-based deduplication.

Changes

Schema Updates

• Bumped S3 revision schema to v2.1.0 with sitelinks_hashes and statements_hashes.
• Removed inline claims/terms/sitelinks from entity; added minimal entity (id/type only).
• Stored terms/sitelinks as plain UTF-8 text in S3 (no JSON/schemas).
• Updated docs and READMEs to reflect hash-based responses.

Storage Changes

• Sitelinks: Plain text in wikibase-sitelinks/{hash}.
• Revisions: Hashes in wikibase-revisions/{entity_id}/{revision_id}.

Code Updates

• Implemented store_sitelink_metadata and load_sitelink_metadata in S3Client for UTF-8 text.
• Updated entity creation to store sitelinks as plain text.
• Added integration tests for plain text S3 operations.
• Implemented Consumer class for Kafka event consumption in watchlist notifications.
• Integrated Consumer in WatchlistConsumerWorker for real-time entity change processing.

[2026-01-13] Sitelinks Deduplication, Endpoints, and Batch API

Summary

Implemented sitelinks deduplication by hashing titles only, keeping wiki identifiers in revisions. Added new S3 bucket for sitelinks metadata, individual lookup endpoints, and batch endpoints for all metadata types to improve frontend performance and caching.

Motivation

• Storage Efficiency: Deduplicate repeated sitelinks titles across entities and revisions.
• Consistency: Align sitelinks with existing metadata deduplication for labels/descriptions/aliases.
• API Completeness: Provide endpoints to query sitelinks and lookup titles by hash.
• Performance: Add batch endpoints to reduce round-trips for multiple metadata lookups.
• Scalability: Organize metadata storage for better management and caching.

Changes

Storage and Deduplication

• Added "sitelinks" S3 bucket created by the development worker.
• Modified revision write logic to hash sitelinks titles and store as plain UTF-8 text in sitelinks/{hash}.
• Updated read logic to reconstruct sitelinks from hashes.
• Added sitelinks_hashes to entity_revisions database schema.

API Endpoints

• Added GET /entitybase/v1/entities/item/{id}/sitelinks/{wiki_id} to retrieve sitelink title for a specific wiki in an entity.
• Added GET /entitybase/v1/sitelinks/{hashes} (batch, up to 20 hashes) to lookup titles by hashes.
• Added GET /entitybase/v1/labels/{hashes} (batch) for labels.
• Added GET /entitybase/v1/descriptions/{hashes} (batch) for descriptions.
• Added GET /entitybase/v1/aliases/{hashes} (batch) for aliases.
• Added GET /entitybase/v1/statements/batch?entity_ids=...&property_ids=... (batch) for statements.

Infrastructure

• Extended MetadataExtractor for sitelinks title hashing.
• Updated CreateBuckets worker to include "sitelinks" bucket.
• Added validation for sitelinks hashes in RevisionData.

[2026-01-13] RevisionData Model Addition

Summary

Introduced RevisionData Pydantic model to structure and validate top-level revision JSON data, replacing raw Dict[str, Any] in RevisionReadResponse. Includes strict allowlist validation for entity keys to enforce Wikibase structure without over-parsing.

Motivation

• Type Safety: Improve validation and error handling for revision data inputs/outputs.
• Consistency: Align with Pydantic best practices for JSON-close models.
• Security: Prevent invalid keys in entity data via strict allowlist, reducing malformed data risks.
• Maintainability: Prepare for future enhancements while avoiding unnecessary nesting.

Changes

Model Changes

• Added RevisionData class in src/models/s3_models.py with fields for schema_version, entity (with allowlist validator), and optional redirects_to.
• Validator raises validation errors for invalid entity keys using raise_validation_error.

API Changes

• Updated RevisionReadResponse.data to use RevisionData.
• Modified s3_client.py to instantiate RevisionData from parsed JSON, with strict validation.
• Adjusted handlers in read.py and admin.py to use model_dump() for dict compatibility.

[2026-01-13] Entity History API Endpoint

Summary

Implemented GET /entities/{entity_id}/history endpoint to retrieve revision history for entities, including revision ID, timestamp, user ID, and edit summary. Added required database schema changes to store edit metadata.

Motivation

• Audit Trail: Provide complete edit history for entities to track changes and accountability
• User Experience: Allow users to see who made changes and why
• Compliance: Support requirements for change tracking and transparency
• API Completeness: Round out entity management capabilities with history viewing

Changes

Database Schema Changes

• Added created_by_user_id column to entity_revisions table
• Added edit_summary column to entity_revisions table

API Changes

• Updated RevisionMetadataResponse model to include user_id and edit_summary fields
• Implemented VitessClient.get_entity_history() method
• Updated EntityReadHandler.get_entity_history() to return complete metadata
• Enhanced /entities/{entity_id}/history endpoint with pagination support

Code Changes

• Modified revision insertion logic to capture user and edit summary
• Updated history queries to include new metadata fields
• Added filtering to skip incomplete history entries
• Required non-empty edit_summary in entity creation, update, and delete requests
• Removed unused editor fields from request models and tests

[2026-01-12] Backlink Statistics Script

Summary

Implemented a daily statistics computation script that analyzes all entity statements to generate backlink analytics. The script scans statement content from S3, extracts entity references, and aggregates backlink counts stored in a new backlink_statistics Vitess table.

Motivation

• Analytics: Enable insights into entity connectivity and relationship patterns at scale
• Performance Monitoring: Track backlink distribution and growth metrics
• Query Optimization: Support UI features showing popular entities by connectivity
• Scalability: Background computation prevents API performance impact
• Maintenance: Automated daily updates ensure fresh statistics

Changes

New Database Table

File: src/models/infrastructure/vitess/schema.py

Added backlink_statistics table:

CREATE TABLE IF NOT EXISTS backlink_statistics (
    date DATE PRIMARY KEY,
    total_backlinks BIGINT NOT NULL,
    unique_entities_with_backlinks BIGINT NOT NULL,
    top_entities_by_backlinks JSON NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Fields: - date: Date of computation (partition key) - total_backlinks: Total backlink relationships across all entities - unique_entities_with_backlinks: Number of entities that have at least one incoming backlink - top_entities_by_backlinks: JSON array of top 100 entities by backlink count

Statistics Computation Script

File: scripts/statistics/backlink_statistics.py

New script with comprehensive backlink analysis:

async def compute_backlinks(vitess_client, s3_client) -> None:
    """Compute backlink statistics for all entities"""

async def extract_entity_references(statement_content: dict) -> list[str]:
    """Extract entity IDs referenced in a statement"""

# Main execution
await compute_backlinks(vitess_client, s3_client)

Features: - Scans all statement hashes from Vitess statement_content table - Batch fetches statement content from S3 for efficiency - Parses statement JSON to extract entity references from mainsnak, qualifiers, and references - Aggregates backlink counts per entity - Stores daily global statistics with top entities ranking

Repository Extensions

File: src/models/infrastructure/vitess/statement_repository.py

Added get_all_statement_hashes() method:

def get_all_statement_hashes(self) -> list[int]:
    """Get all statement content hashes for backlink computation"""

File: src/models/infrastructure/vitess/backlink_repository.py

Added insert_backlink_statistics() method:

def insert_backlink_statistics(
    self, date: str, total_backlinks: int,
    unique_entities_with_backlinks: int, top_entities_by_backlinks: list[dict]
) -> None:
    """Insert daily backlink statistics with upsert logic"""

S3 Client Extensions

File: src/models/infrastructure/s3/s3_client.py

Added batch_get_statements() method:

async def batch_get_statements(self, content_hashes: list[int]) -> dict[int, dict[str, Any]]:
    """Batch read multiple statements from S3 for efficient processing"""

Impact

• Storage: Minimal additional storage (~1KB/day for statistics table)
• Performance: Background computation doesn't impact API performance
• Analytics: Enables insights into entity relationship patterns
• Monitoring: Health checks and error logging for operational visibility

Backward Compatibility

• Non-breaking: New table and script don't affect existing functionality
• Optional: Script can be disabled or run on-demand
• Graceful degradation: Statistics unavailable if script fails

[2026-01-12] Hash Algorithm Decision: 64-bit Rapidhash

Summary

Completed comprehensive analysis of hash collision probabilities across different hash sizes (64-bit, 96-bit, 112-bit, 128-bit, 256-bit) for Wikibase scale (10 billion entities, 2 trillion total items). Decision: permanently use 64-bit rapidhash due to sufficient collision resistance and migration impossibility.

Motivation

• Collision Analysis: Determine optimal hash size for data integrity at massive scale
• Ecosystem Stability: Avoid breaking changes that would affect 1000+ consumers
• Performance Optimization: Balance collision safety with storage/performance costs
• Migration Feasibility: Assess practical upgrade paths for hash algorithms

Analysis Results

Collision Probabilities (at 2×10^12 items)

• 64-bit rapidhash: 1 in 9,174 (negligible risk)
• 128-bit SHA256: 1 in 10^15 (astronomically low)
• 256-bit SHA256: 1 in 10^69 (theoretically impossible)

Storage Costs

• 64-bit: 8 bytes/hash (baseline)
• 128-bit: 16 bytes/hash (2x increase, $160/month)
• 256-bit: 32 bytes/hash (4x increase, $320/month)

Migration Assessment

• Hash format changes: Effectively impossible due to ecosystem scale
• Linking table: Would require 80TB for 2 trillion mappings
• Consumer coordination: 1000+ applications would need simultaneous updates

Changes

Updated Risk Documentation

File: doc/RISK/HASH-COLLISION.md

Comprehensive rewrite with: - Mathematical collision probability analysis - Performance and storage cost comparisons - Migration impossibility assessment - Final decision documentation - Monitoring and detection strategies

Hash Algorithm Standardization

Decision: Use 64-bit rapidhash for all content hashing: - Entity JSON snapshots - Statement deduplication - Term string deduplication - Metadata content hashing

Impact

• Data Integrity: Negligible collision risk at target scale
• Ecosystem Stability: No breaking changes for consumers
• Performance: Optimal hash generation speed
• Storage: Minimal overhead (8 bytes per hash)
• Operations: Simplified architecture without migration complexity

Backward Compatibility

• No changes: Existing 64-bit hashes remain compatible
• Future-proofing: Can add secondary hash verification if needed
• Monitoring: Collision detection logging for anomaly detection

[2026-01-12] Event JSON Schema Definitions

Summary

Added JSON Schema definitions for entitychange and entitypropertychange event types to standardize and validate event payloads in the Kafka streaming system. These schemas ensure consistent event structure for consumers like watchlist services and analytics pipelines.

Motivation

• Event Standardization: Define canonical formats for entity change events
• Validation: Enable runtime validation of event payloads against schemas
• Consumer Safety: Prevent malformed events from breaking downstream processing
• Documentation: Provide clear contracts for event producers and consumers

Changes

New Event Schemas

Files: src/schemas/events/entitychange.json, src/schemas/events/entitypropertychange.json

• entitychange.json: Base schema for all entity change events with fields: entity_id, revision_id, change_type, from_revision_id, changed_at, editor, edit_summary
• entitypropertychange.json: Extended schema for property-specific changes, adding required changed_properties array

Both schemas use JSON Schema Draft 2020-12 with validation patterns for entity/property IDs and enumerated change types.

[2026-01-12] User Registration Support

Summary

Added user registration endpoint and users table to support watchlist features with MediaWiki user IDs. This enables frontend-initiated user registration without authentication, allowing users to be tracked for watchlist subscriptions and notifications.

Motivation

• User Management: Provide a way to register users in the system using MediaWiki IDs
• Watchlist Foundation: Enable user-specific watchlist operations and notifications
• Frontend Integration: Allow frontend to create user entries as needed
• Simplicity: No auth required, trusting frontend for valid MediaWiki user_ids

Changes

New Users Table

File: src/models/infrastructure/vitess/schema.py

Added users table creation: - user_id (BIGINT PRIMARY KEY) - MediaWiki user ID - created_at (TIMESTAMP) - Registration timestamp - preferences (JSON) - Reserved for future user preferences

User Registration Endpoints

File: src/models/rest_api/main.py

New endpoints: - POST /v1/users: Create/register user with MediaWiki ID - Request: {"user_id": 12345} - Response: {"user_id": 12345, "created": true/false} - Idempotent, no authentication - GET /v1/users/{user_id}: Retrieve user information - Returns user data or 404 if not found - Allows frontend to check user existence/registration status

Watchlist Support Implementation

Summary

Implemented core watchlist functionality for subscribing to entity changes or specific properties. Includes data models, storage, and API endpoints for managing watches.

Motivation

• User Subscriptions: Enable users to track changes on entities or properties
• Granular Watching: Support whole-entity or property-specific watches
• Foundation for Notifications: Prepare for event-driven change alerts

Changes

Watchlist Data Models

File: src/models/watchlist.py

• WatchlistEntry: DB model with user_id, internal_entity_id, watched_properties
• WatchlistAddRequest: API request for adding watches
• WatchlistRemoveRequest: API request for removing watches
• WatchlistResponse: API response listing user's watches

Watchlist Repository

File: src/models/infrastructure/vitess/watchlist_repository.py

New WatchlistRepository class with ID resolution: - add_watch(): Add watch with external→internal ID conversion - remove_watch(): Remove watch by user/entity/properties - get_watches_for_user(): Retrieve user's watchlist with resolved entity_ids - get_watchers_for_entity(): Get watchers for an entity (for notifications)

Database Schema

File: src/models/infrastructure/vitess/schema.py

Added watchlist table: - user_id (BIGINT) - internal_entity_id (BIGINT, FK to entity_id_mapping) - watched_properties (TEXT, comma-separated) - Primary key: (user_id, internal_entity_id, watched_properties)

API Endpoints

File: src/models/rest_api/main.py

New watchlist endpoints: - POST /v1/watchlist: Add watch - Request: {"user_id": 12345, "entity_id": "Q42", "properties": ["P31"]} - DELETE /v1/watchlist: Remove watch - Request: {"user_id": 12345, "entity_id": "Q42", "properties": ["P31"]} - GET /v1/watchlist?user_id=12345: Get user's watchlist - Response: {"user_id": 12345, "watches": [{"entity_id": "Q42", "properties": ["P31"]}...]}

All endpoints validate user registration and handle ID resolution internally.

Future Integration

Watchlist endpoints will validate user existence against this table to ensure only registered users can create watchlists.

Watchlist Notifications System

Summary

Implemented event-driven notification system for watchlist changes. Includes background consumer worker, notification storage, and API endpoints for frontend to retrieve and manage notifications.

Motivation

• Real-time Updates: Enable users to receive notifications when watched entities/properties change
• Event-Driven: Leverage existing Kafka event stream for scalable notifications
• User Experience: Provide recent changes feed with check/ack functionality

Changes

Notification Storage

File: src/models/infrastructure/vitess/schema.py

Added user_notifications table: - user_id (BIGINT) - entity_id (VARCHAR), revision_id (INT), change_type (VARCHAR) - changed_properties (JSON), event_timestamp (TIMESTAMP) - is_checked (BOOLEAN), checked_at (TIMESTAMP)

Event Consumer Worker

File: src/models/workers/watchlist_consumer/main.py

New WatchlistConsumerWorker: - Consumes entitychange/entitypropertychange events from Kafka - Matches events against user watchlists - Creates notification records for relevant users - Handles property-specific vs. entity-wide watches

Notification API Endpoints

File: src/models/rest_api/main.py

New endpoints: - GET /v1/watchlist/notifications?user_id=X&limit=30: Retrieve recent notifications - POST /v1/watchlist/notifications/check: Mark notification as checked

Response includes notification details: entity, revision, change type, properties, timestamps.

Repository Extensions

File: src/models/infrastructure/vitess/watchlist_repository.py

Added methods: - get_user_notifications(): Fetch paginated notifications for user - mark_notification_checked(): Update checked status - _create_notification(): Insert notification (used by consumer)

This completes the watchlist system with end-to-end notification flow.

[2026-01-12] User Notification Preferences

Summary

Implemented user-configurable notification preferences with personalized limits and retention settings. Users can customize their notification experience while maintaining system scalability.

Motivation

• Personalization: Allow users to control notification volume and retention based on their needs
• Scalability: User preferences enable fine-tuned resource management
• User Experience: Match individual workflows (power users vs. casual watchers)
• Flexibility: Support different notification patterns for subgraph protection

Changes

Database Schema Extensions

File: src/models/infrastructure/vitess/schema.py

Extended users table with preference fields: - notification_limit INT DEFAULT 50 (max notifications per user) - retention_hours INT DEFAULT 24 (notification retention period)

User Preferences API

File: src/models/rest_api/main.py

New endpoints for preference management: - GET /v1/users/{user_id}/preferences: Retrieve current notification settings - PUT /v1/users/{user_id}/preferences: Update notification limit and retention

Request validation: notification_limit (50-500), retention_hours (1-720)

Repository Enhancements

File: src/models/infrastructure/vitess/user_repository.py

Added preference management methods: - get_user_preferences(user_id): Retrieve user's notification settings - update_user_preferences(user_id, limit, retention): Update user preferences

Consumer Integration

File: src/models/workers/watchlist_consumer/main.py

• Creates all eligible notifications without limit checks
• Cleanup worker enforces user preference limits via oldest-first deletion
• Simplified consumer logic focused on event processing

Handler Implementation

File: src/models/rest_api/handlers/user_preferences.py

• UserPreferencesHandler: Manages preference queries and updates
• Validation of preference ranges
• Integration with user repository

Request/Response Models

Files: src/models/rest_api/request/user_preferences.py, src/models/rest_api/response/user_preferences.py

• UserPreferencesRequest: notification_limit, retention_hours
• UserPreferencesResponse: user_id, notification_limit, retention_hours

User Experience

• Default settings: 50 notifications, 24-hour retention
• Customizable limits: Up to 500 notifications, 30-day retention
• Immediate effect: Preference changes apply to new notifications
• Backward compatibility: Existing users use defaults

Performance & Scaling

• Lightweight preference storage in users table
• Efficient queries for preference retrieval
• Consumer respects individual limits for fair resource usage
• No impact on existing notification processing

This enables personalized notification management while maintaining system performance and scalability.

[2026-01-18] Add Single Property Endpoint

Summary

Added POST /entitybase/v1/entities/{entity_id}/properties/{property_id} endpoint to add claims for a single property to an existing entity. Includes property existence validation and integrates with existing entity update workflow.

Motivation

• Incremental Updates: Allow adding statements for specific properties without full entity replacement
• API Completeness: Support property-level operations for better client flexibility
• Data Integrity: Validate property existence and type before allowing additions

Changes

New Request Model

File: src/models/rest_api/entitybase/request/entity/add_property.py

• AddPropertyRequest: claims (list of statements), edit_summary

Handler Method

File: src/models/rest_api/entitybase/handlers/entity/base.py

• EntityHandler.add_property(): Validates property, fetches entity, merges claims, processes update
• Returns OperationResult[dict] with {"revision_id": int}

API Endpoint

File: src/models/rest_api/entitybase/versions/v1/entities.py

• POST /entities/{entity_id}/properties/{property_id} with AddPropertyRequest
• Response: OperationResult[dict]

Validation

• Property ID format check (P followed by digits)
• Property existence and type verification via entity fetch
• Claim merging with existing property claims

Impact

• New Functionality: Single-property additions for entities
• Backward Compatibility: No breaking changes to existing APIs
• Performance: Reuses existing update infrastructure

Notes

• Claims are appended to existing ones for the property
• Full entity re-processing ensures consistency
• Property must exist as a "property" type entity

[2026-01-18] Remove Statement by Hash Endpoint

Summary

Added DELETE /entitybase/v1/entities/{entity_id}/statements/{statement_hash} endpoint to remove a specific statement from an entity by its hash. Uses optimized direct hash removal from revision data with automatic property count recalculation.

Motivation

• Granular Editing: Allow targeted removal of individual statements
• Performance: Avoid full entity re-processing for efficient removals
• Data Integrity: Maintain accurate property counts and metadata

Changes

Request Model

File: src/models/rest_api/entitybase/request/entity/remove_statement.py

• RemoveStatementRequest: edit_summary for audit trail

Handler Method

File: src/models/rest_api/entitybase/handlers/entity/base.py

• EntityHandler.remove_statement(): Direct revision hash modification
• Removes hash from statements list, decrements ref_count, recalculates property counts
• Removes properties with 0 statements from metadata

API Endpoint

File: src/models/rest_api/entitybase/versions/v1/entities.py

• DELETE /entities/{entity_id}/statements/{statement_hash} with RemoveStatementRequest
• Response: OperationResult[dict] with revision_id

Validation

• Statement hash exists in revision statements list
• Fails if ref_count decrement fails (strict consistency)

Impact

• New Functionality: Efficient statement removal without full re-hashing
• Backward Compatibility: No breaking changes
• Performance: Minimal processing compared to full entity updates

Notes

• Directly modifies revision hashes and metadata
• Automatic property cleanup when counts reach 0

[2026-01-18] Patch Statement by Hash Endpoint

Summary

Added PATCH /entitybase/v1/entities/{entity_id}/statements/{statement_hash} endpoint to replace a specific statement with new claim data. Provides efficient in-place editing without remove+add operations.

Motivation

• Simplified Editing: Single operation for statement modifications
• Better UX: Avoids two-step process for frontend edits
• Performance: Reuses full processing for consistency
• API Completeness: Complete CRUD operations for statements

Changes

Request Model

File: src/models/rest_api/entitybase/request/entity/patch_statement.py

• PatchStatementRequest: claim (new statement data), edit_summary

Handler Method

File: src/models/rest_api/entitybase/handlers/entity/base.py

• EntityHandler.patch_statement(): Finds statement by hash, replaces with new claim, processes update

API Endpoint

File: src/models/rest_api/entitybase/versions/v1/entities.py

• PATCH /entities/{entity_id}/statements/{statement_hash} with PatchStatementRequest
• Response: OperationResult[dict] with revision_id

Validation

• Statement hash exists in entity's claims
• New claim data is valid JSON

Impact

• New Functionality: Direct statement editing
• Backward Compatibility: No breaking changes
• Performance: Same as full entity updates

Notes

• Replaces entire statement with new claim
• Maintains property structure
• Full validation and processing

[2026-01-18] Remove Full Entity Update Endpoints

Summary

Removed full entity update endpoints (PUT /entities/{type}/{id}) to enforce granular editing. Frontends must now use specialized endpoints for modifications.

Motivation

• Granular Control: Prevent accidental full entity overwrites
• API Consistency: Align with new statement/property level operations
• Safety: Reduce risk of data loss from bulk updates

Changes

Removed Endpoints

File: src/models/rest_api/entitybase/versions/v1/items.py

• PUT /item/{entity_id} - Full item updates
• PUT /property/{entity_id} - Full property updates
• PUT /lexeme/{entity_id} - Full lexeme updates

Removed Imports

• Removed EntityUpdateRequest and update handler imports
• Cleaned up unused dependencies

Migration Guide

Old Approach (Removed):

PUT /entitybase/v1/item/Q42
{
  "labels": {...},
  "claims": {...},
  ...
}

New Approach (Required): - For statements: PATCH /entitybase/v1/entities/Q42/statements/{hash} or DELETE + POST /entities/Q42/properties/{pid} - For metadata: Use term/label/description specific endpoints - For additions: POST /entitybase/v1/entities/Q42/properties/{pid}

Impact

• Breaking Change: Full entity updates no longer supported
• Improved Safety: Forces intentional, granular modifications
• API Simplification: Removes redundant update paths

[2026-01-18] Refactor EntityTransaction Base Class

Summary

Moved EntityTransaction base class to dedicated file and updated inheritance structure for better code organization.

Changes

New Base Class File

File: src/models/rest_api/entitybase/handlers/entity/entity_transaction.py

• Created dedicated file for EntityTransaction base class
• Includes shared rollback logic and abstract process_statements method
• Provides consistent interface for creation and update transactions

Updated Transaction Classes

Files: src/models/rest_api/entitybase/handlers/entity/creation_transaction.py, src/models/rest_api/entitybase/handlers/entity/update_transaction.py

• Removed duplicate EntityTransaction definitions
• Updated to inherit from shared base class
• Maintained existing functionality and method signatures

Impact

• Code Organization: Cleaner separation of concerns
• Maintainability: Single source of truth for transaction base logic
• No Functional Changes: All existing behavior preserved

[2026-01-18] Test Fixes and API Parameter Updates

Summary

Fixed missing user_id parameters in entity creation and update transaction methods. Updated S3 method calls for consistency.

Changes

Transaction Methods

Files: src/models/rest_api/entitybase/handlers/entity/creation_transaction.py, src/models/rest_api/entitybase/handlers/entity/update_transaction.py

• Added user_id: int parameter to create_revision() methods
• Updated calls to _create_and_store_revision() to include user_id

Handler Calls

Files: src/models/rest_api/entitybase/handlers/entity/update.py, src/models/rest_api/entitybase/handlers/entity/item.py

• Added user_id=request.user_id to tx.create_revision() calls

S3 Method Consistency

File: src/models/rest_api/entitybase/handlers/entity/base.py

• Changed s3_client.store_revision() to s3_client.write_revision() for correct method usage

Impact

• API Consistency: user_id properly passed through transaction layers
• Code Correctness: Fixed method signature mismatches
• Test Stability: Resolves parameter-related test failures
• Strict error handling for ref_count operations

[2026-01-12] EntityBase Revert API for Subgraph Protection

Summary

Added a new EntityBase API endpoint for reverting entities to previous revisions, enabling manual intervention against vandalism and problematic changes detected by the watchlist system. This supports subgraph protection workflows where users monitor large entity networks for quality issues.

Motivation

• Vandalism Response: Provide tools for rapid reversion of damaging edits
• Data Integrity: Allow restoration of subgraphs affected by bad modeling
• Consumer Protection: Prevent negative impacts on downstream data users
• Auditability: Full logging of revert actions for transparency

Changes

Revert API Endpoint

File: src/models/rest_api/main.py

New endpoint: POST /entitybase/v1/entities/{entity_id}/revert - Reverts entity to specified revision with audit logging - Requires reverted_by_user_id for accountability - Includes optional watchlist context for linking to notifications

Revert Logging

File: src/models/infrastructure/vitess/schema.py

Added revert_log table: - Tracks all reverts with entity, revision, user, reason details - Supports subgraph protection audit trails - Scales to trillions of entries with distributed storage

Handler and Repository

Files: src/models/rest_api/handlers/entity/revert.py, src/models/infrastructure/vitess/revision_repository.py

• EntityRevertHandler: Validates requests and coordinates reversion
• Extended RevisionRepository.revert_entity(): Performs data restoration and logging
• Error handling for invalid revisions and conflicts

API Models

Files: src/models/rest_api/request/entity/revert.py, src/models/rest_api/response/entity/revert.py

• EntityRevertRequest: to_revision_id, reason, reverted_by_user_id, watchlist_context
• EntityRevertResponse: entity_id, new_revision_id, reverted_from_revision_id, timestamp

Integration with Watchlist

• Revert API designed for integration with watchlist notifications
• Frontend can provide watchlist context for revert tracking
• Supports manual protection workflows without auto-reversion

[2026-01-12] User Activity Logging System

Summary

Implemented comprehensive user activity logging for entity operations, providing complete edit history and moderation trails. Activities point to revisions for detailed data storage.

Motivation

• Edit History: Users need complete timeline of their entity operations
• Moderation: Track all entity modifications for audit and oversight
• Transparency: Clear record of create/edit/revert/delete operations
• Analytics: Enable contribution analysis and user activity patterns

Changes

Activity Types and Models

File: src/models/user_activity.py

• ActivityType enum: entity_create, entity_edit, entity_revert, entity_delete, entity_undelete, entity_lock, entity_unlock, entity_archive, entity_unarchive
• UserActivity model: id, user_id, activity_type, entity_id, revision_id, created_at

Activity Logging Infrastructure

File: src/models/infrastructure/vitess/schema.py

Added user_activity table: - user_id (BIGINT), activity_type (VARCHAR), entity_id (VARCHAR), revision_id (BIGINT) - created_at (TIMESTAMP), indexes on user_id, activity_type, entity_id

File: src/models/infrastructure/vitess/user_repository.py

• Added log_user_activity() method for consistent logging
• Stores minimal data + revision pointer for full details

Activity Logging Integration

File: src/models/rest_api/handlers/entity/revert.py

• Logs entity_revert activities with revision pointers
• Includes entity_id, revision_id, user_id for audit trail

Activity Retrieval API

File: src/models/rest_api/main.py

• GET /v1/users/{user_id}/activity: Retrieve user's activity history
• Parameters: hours (time filter), limit (50/100/250/500), type (activity type filter)
• Response: Activity list with revision pointers for detail access

Handler and Repository

File: src/models/rest_api/handlers/user_activity.py, src/models/infrastructure/vitess/user_repository.py

• UserActivityHandler: Processes activity queries with filtering
• Enhanced UserRepository.get_user_activities(): Supports time/type filtering, pagination

Future Integration Points

• Add logging to future entity create/edit/delete/lock/archive handlers
• Consistent activity logging across all entity operations
• Revision-based detail retrieval for rich activity views

Performance & Scaling

• Efficient indexing for user-specific queries
• Time-based filtering leverages revision storage
• Pagination prevents large response payloads
• Scales to millions of activities per user

[2026-01-12] Backlink Statistics Worker

Summary

Implemented a background worker service that computes and stores backlink statistics daily. The worker generates analytics on entity relationships including total backlinks, unique entities with backlinks, and top entities ranked by backlink count. Statistics are stored in a new backlink_statistics Vitess table for efficient querying.

Motivation

• Analytics: Enable data-driven insights into entity connectivity and relationships
• Performance Monitoring: Track backlink growth and distribution patterns
• Query Optimization: Support UI features showing popular entities by connectivity
• Scalability: Background computation prevents API performance impact
• Maintenance: Automated daily updates ensure fresh statistics

Changes

New Database Table

File: src/models/infrastructure/vitess/schema.py

Added backlink_statistics table:

CREATE TABLE IF NOT EXISTS backlink_statistics (
    date DATE PRIMARY KEY,
    total_backlinks BIGINT NOT NULL,
    unique_entities_with_backlinks BIGINT NOT NULL,
    top_entities_by_backlinks JSON NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Fields: - date: Date of computation (partition key) - total_backlinks: Total backlink relationships across all entities - unique_entities_with_backlinks: Number of entities that have at least one incoming backlink - top_entities_by_backlinks: JSON array of top 100 entities by backlink count

Statistics Service

File: src/models/rest_api/services/backlink_statistics_service.py

New BacklinkStatisticsService class:

class BacklinkStatisticsService(BaseModel):
    def compute_daily_stats(self, vitess_client: VitessClient) -> BacklinkStatisticsData:
        """Compute comprehensive backlink statistics for current date"""

    def get_total_backlinks(self, vitess_client: VitessClient) -> int:
        """Count total backlink relationships"""

    def get_entities_with_backlinks(self, vitess_client: VitessClient) -> int:
        """Count entities that have incoming backlinks"""

    def get_top_entities_by_backlinks(
        self, vitess_client: VitessClient, limit: int = 100
    ) -> list[dict[str, Any]]:
        """Get top entities ranked by backlink count"""

Worker Implementation

File: src/models/workers/backlink_statistics/backlink_statistics_worker.py

New BacklinkStatisticsWorker class following existing worker pattern:

class BacklinkStatisticsWorker(BaseModel):
    worker_id: str = Field(default_factory=lambda: os.getenv("WORKER_ID", f"backlink-stats-{os.getpid()}"))

    async def start(self) -> None:
        """Start the backlink statistics worker"""

    async def run_daily_computation(self) -> None:
        """Run daily statistics computation and storage"""

    async def health_check(self) -> WorkerHealthCheck:
        """Health check endpoint"""

Features: - Daily scheduled execution (configurable via environment) - Async processing to avoid blocking - Comprehensive error handling and logging - Health check endpoint for monitoring

Response Models

File: src/models/rest_api/response/misc.py

Added models for statistics data:

class BacklinkStatisticsData(BaseModel):
    """Container for computed backlink statistics"""

    total_backlinks: int
    unique_entities_with_backlinks: int
    top_entities_by_backlinks: list[dict[str, Any]]

class BacklinkStatisticsResponse(BaseModel):
    """API response for backlink statistics"""

    date: str
    total_backlinks: int
    unique_entities_with_backlinks: int
    top_entities_by_backlinks: list[dict[str, Any]]

Configuration

File: src/models/config/settings.py

Added worker configuration:

class Settings(BaseSettings):
    backlink_stats_enabled: bool = Field(default=True)
    backlink_stats_schedule: str = Field(default="0 2 * * *")  # Daily at 2 AM
    backlink_stats_top_limit: int = Field(default=100)

Impact

• Storage: Minimal additional storage (~1KB/day for statistics table)
• Performance: Background computation doesn't impact API performance
• Analytics: Enables insights into entity relationship patterns
• Monitoring: Health checks and error logging for operational visibility

Backward Compatibility

• Non-breaking: New table and worker don't affect existing functionality
• Optional: Worker can be disabled via configuration
• Graceful degradation: Statistics unavailable if worker fails

[2026-01-11] S3 Revision Schema 2.0.0 - Term Deduplication

Summary

Updated S3 revision schema to version 2.0.0 with per-language hash-based deduplication for labels, descriptions, and aliases. Implemented language-specific API endpoints for term retrieval. Documented hash collision risks.

Motivation

• Storage Efficiency: Enable granular deduplication of term strings across languages/entities
• API Completeness: Provide language-specific endpoints for labels/aliases/descriptions
• Scalability: Reduce storage overhead for multilingual content

Changes

Schema Update

File: src/schemas/entitybase/s3-revision/2.0.0/schema.json (renamed from 1.2.0)

• Bumped version to 2.0.0 for breaking changes
• Removed full labels, descriptions, aliases from entity object
• Added per-language hash maps:
• labels_hashes: {"en": hash, "fr": hash}
• descriptions_hashes: {"en": hash}
• aliases_hashes: {"en": [hash1, hash2]}

Hashing Logic Update

File: src/models/internal_representation/metadata_extractor.py

Modified hash_metadata() to compute individual 64-bit rapidhashes for each term string instead of entire objects.

Revision Storage Changes

File: src/models/rest_api/handlers/entity/__init__.py

Updated revision creation to: - Extract terms per language from entity data - Hash individual strings and store in language-keyed maps - Store term strings in S3 keyed by hash for deduplication

Retrieval Logic Update

File: src/models/rest_api/handlers/entity/read.py

Modified entity loading to reconstruct full term objects by loading strings from S3 using hash maps.

API Endpoints Implementation

Files: src/models/rest_api/wikibase/v1/entity/items.py, properties.py

Replaced 501 stubs with functional endpoints for: - GET /entities/items/{id}/labels/{lang} - GET /entities/items/{id}/descriptions/{lang} - GET /entities/items/{id}/aliases/{lang}

Risk Documentation

File: doc/RISK/HASH-COLLISION.md (new)

Documented 64-bit hash collision probability and accepted risks for deduplication.

[2026-01-10] Backlinks Support Implementation

Summary

Added entity_backlinks table to track incoming references between entities, enabling efficient backlink queries. Implemented QID extraction from statement JSON to identify referenced entities in mainsnak, qualifiers, and references.

Motivation

• Query Efficiency: Enable fast lookup of entities that reference a given entity in their statements
• Scalability: Use BIGINT internal_ids for FKs, with sharding on referenced_internal_id
• Completeness: Support full Wikibase backlinks functionality for entity relationships

Changes

New entity_backlinks Table

File: src/models/infrastructure/vitess/schema.py

Added table to track backlinks with composite primary key for uniqueness:

• referenced_internal_id BIGINT (entity being referenced)
• referencing_internal_id BIGINT (entity making the reference)
• statement_hash BIGINT (links to specific statement)
• property_id VARCHAR(32) (property used in statement)
• rank ENUM (preferred/normal/deprecated)

Includes foreign key constraints and indexes for query performance.

QID Extraction Logic

File: src/models/domain/entity/statement_parser.py (new)

Recursive function to extract entity IDs from statement JSON structures.

Updated Entity Write Logic

File: src/models/rest_api/handlers/entity/types.py

Modified entity update/create to populate backlinks table during statement processing.

New API Endpoint

File: src/models/rest_api/handlers/entity/backlinks.py (new)

GET /entities/{id}/backlinks returns paginated list of referencing entities.

[2026-01-09] Transaction-Based Item Creation with Rollback

Summary

Implemented atomic item creation using a Pydantic CreationTransaction class that manages operations with full rollback on failure. Includes per-statement rollback, worker handshake for ID confirmation, and removal of redundant checks. Updated enumeration handlers with high minimum IDs to avoid Wikidata collisions.

Motivation

• Atomicity: Ensure creation is all-or-nothing; rollback on S3/Vitess failures prevents orphaned data.
• Reliability: Trust worker for unique IDs, but confirm usage; rollback statements individually.
• Simplicity: Remove existence/deletion checks; direct revision ID = 1 for creations.
• Collision Avoidance: Set minimum IDs above Wikidata ranges (Q: 300M, P: 30K, L: 5M, E: 50K).

Changes

New CreationTransaction Class

File: src/models/rest_api/handlers/entity/creation_transaction.py

Pydantic BaseModel for managing creation operations:

• register_entity(): Reserves ID in Vitess.
• process_statements(): Hashes/deduplicates statements, stores in S3/Vitess.
• create_revision(): Stores revision snapshot with CAS protection against concurrent modifications.
• publish_event(): Emits change event.
• commit(): Confirms ID usage; clears rollback operations.
• rollback(): Undoes all operations in reverse (deletes from Vitess/S3, decrements ref_counts).

Features: - Per-statement rollback: Tracks hashes, decrements ref_counts, deletes orphaned S3 objects. - Logging: Info logs at method starts for tracing. - Reusable: Designed for future extension to updates/deletes.

Updated Item Creation Flow

File: src/models/rest_api/handlers/entity/types.py

• Removed existence/deletion checks (trust worker).
• Removed idempotency check (no prior revisions).
• Direct new_revision_id = 1 for creations.
• Wrapped operations in CreationTransaction with try/except rollback.

Enumeration Handler Updates

Files: src/models/rest_api/handlers/entity/enumeration/*.py

• Moved classes to individual files with high minimum IDs.
• Updated base classes: min_id set to avoid Wikidata collisions.

Documentation Updates

File: doc/ARCHITECTURE/ENTITY-MODEL.md

• Updated entity creation flow diagram to reflect transaction-based approach.
• Emphasized rollback and worker handshake.

Impact

• Reliability: Atomic creation with full rollback; no orphaned data.
• Performance: Removed unnecessary checks; faster for new items.
• ID Safety: Minimum IDs prevent Wikidata conflicts.
• Maintainability: Transaction class encapsulates rollback logic.

Backward Compatibility

• Non-breaking: API unchanged; internal flow improved.
• Rollbacks: Graceful failure handling; logs warnings on rollback errors.

[2026-01-09] Enumeration Handler Updates and Documentation Refinements

Summary

Updated enumeration handlers with correct minimum ID values to prevent collisions with Wikidata.org entities, refined S3 storage paths by removing the "r" prefix for consistency, and updated documentation to reflect 1-based revision indexing. Bumped S3 revision schema to v1.2.0 for documentation alignment.

Motivation

• Collision Prevention: Ensure new entity IDs start above existing Wikidata ranges to avoid conflicts during migration or coexistence.
• Path Consistency: Standardize S3 object paths to use clean integer revision IDs without prefixes.
• Documentation Accuracy: Align docs with 1-based revision indexing and updated minimum ID values.

Changes

Updated Enumeration Handlers

Files: src/models/rest_api/handlers/entity/enumeration/*.py

Updated minimum ID values in base handler classes to safe ranges above Wikidata maximums:

• Item: min_id = 300_000_000 (above Q120M+)
• Property: min_id = 30_000 (above P10K+)
• Lexeme: min_id = 5_000_000 (above L1M+)
• EntitySchema: min_id = 50_000 (conservative buffer)

Rationale: - Prevents ID collisions when integrating with or migrating from Wikidata. - Values set conservatively above current Wikidata ranges with buffers for growth.

S3 Path Standardization

Files: Documentation files (doc/ARCHITECTURE/ENTITY-MODEL.md, etc.)

Removed "r" prefix from S3 revision paths: - Before: s3://wikibase-revisions/Q123/r42.json - After: s3://wikibase-revisions/Q123/42.json

Rationale: - Simplifies paths to use raw integer revision IDs. - Consistent with schema expectations of integer revision identifiers.

Rationale: - Marks documentation refinements and minimum ID awareness. - No breaking changes to JSON structure.

Documentation Updates

File: doc/ARCHITECTURE/ENTITY-MODEL.md

• Updated entity creation examples to use revision_id=1 and clean S3 paths.
• Emphasized 1-based revision indexing.
• Added notes on minimum ID collision avoidance.

Impact

• ID Safety: New entities use safe starting IDs preventing Wikidata conflicts.
• Storage Consistency: S3 paths use clean integer revision IDs.
• Developer Experience: Documentation accurately reflects implementation details.

Backward Compatibility

• Non-breaking: Enumeration changes affect only new entity creation.
• S3 Paths: Existing paths remain functional; new paths follow updated convention.
• Schema: v1.2.0 compatible with v1.1.0 (no structural changes).

[2026-01-08] Change Event Producer for Redpanda

Summary

Added change event producer infrastructure for publishing entity change events to Redpanda (Kafka-compatible streaming platform). Implemented ChangeType enum with 10 change classifications and EntityChangeEvent BaseModel for structured event publishing. All entity operations (creation, edit, redirect, archival, lock, deletion) now emit change events to wikibase.entity_change topic for downstream consumers like RDF streamers and analytics pipelines.

Motivation

Wikibase-backend requires change event streaming for:

• Downstream consumers: RDF change streamers, search indexers, analytics pipelines need real-time entity change notifications
• Event-driven architecture: Decouple entity operations from change processing, enable reactive updates
• Change detection: Continuous RDF Change Streamer needs entity change events to trigger RDF diff computation
• Audit trail: External systems can track all entity modifications with proper change type classification
• Scalability: Async event production allows API to remain responsive while events are processed asynchronously

Changes

New Kafka Configuration

File: src/models/config/settings.py

class Settings(BaseSettings):
    kafka_brokers: str = "redpanda:9092"
    kafka_topic: str = "wikibase.entity_change"

Environment variables: - KAFKA_BROKERS: Redpanda broker address (default: redpanda:9092) - KAFKA_TOPIC: Topic for entity change events (default: wikibase.entity_change)

File: docker-compose.yml

redpanda:
  image: redpandadata/redpanda:latest
  ports:
    - "9092:9092"
  healthcheck:
    test: ["CMD-SHELL", "rpk cluster health | grep -q 'Healthy'"]

rest-api:
  environment:
    KAFKA_BROKERS: redpanda:9092
    KAFKA_TOPIC: wikibase.entity_change
  depends_on:
    redpanda:
      condition: service_healthy

New ChangeType Enum

File: src/models/api_models.py

class ChangeType(str, Enum):
    """Change event types for streaming to Redpanda"""

    CREATION = "creation"
    EDIT = "edit"
    REDIRECT = "redirect"
    UNREDIRECT = "unredirect"
    ARCHIVAL = "archival"
    UNARCHIVAL = "unarchival"
    LOCK = "lock"
    UNLOCK = "unlock"
    SOFT_DELETE = "soft_delete"
    HARD_DELETE = "hard_delete"

Rationale: - Underscore naming for consistency with Python conventions - All 10 change types map directly to existing EditType classifications - Single topic strategy simplifies consumer architecture

New Entity Change Event Model

File: src/models/api_models.py

class EntityChangeEvent(BaseModel):
    """Entity change event for publishing to Redpanda"""

    entity_id: str = Field(..., description="Entity ID (e.g., Q42)")
    revision_id: int = Field(..., description="Revision ID of the change")
    change_type: ChangeType = Field(..., description="Type of change")
    from_revision_id: Optional[int] = Field(
        None, description="Previous revision ID (null for creation)"
    )
    changed_at: datetime = Field(..., description="Timestamp of change")
    editor: Optional[str] = Field(None, description="Editor who made the change")
    edit_summary: Optional[str] = Field(None, description="Edit summary")
    bot: bool = Field(False, description="Whether this was a bot edit")

    model_config = ConfigDict(json_encoders={datetime: lambda v: v.isoformat()})

Event schema:

{
  "entity_id": "Q42",
  "revision_id": 101,
  "change_type": "edit",
  "from_revision_id": 100,
  "changed_at": "2026-01-08T12:00:00Z",
  "editor": "User:Example",
  "edit_summary": "Updated description",
  "bot": false
}

New Kafka Producer Client

File: src/models/infrastructure/kafka/kafka_producer.py

from aiokafka import AIOKafkaProducer
from pydantic import BaseModel


class KafkaProducerClient(BaseModel):
    """Async Kafka producer client for publishing change events"""

    bootstrap_servers: str
    topic: str
    producer: AIOKafkaProducer | None = None

    async def start(self) -> None:
        """Start the Kafka producer"""

    async def stop(self) -> None:
        """Stop the Kafka producer"""

    async def publish_change(self, event: EntityChangeEvent) -> None:
        """Publish entity change event to Kafka"""

    async def publish_change_sync(self, event: EntityChangeEvent) -> None:
        """Synchronous publish with delivery confirmation"""

Features: - Async production using aiokafka for non-blocking event publishing - Automatic serialization to JSON - Entity ID as message key for partition ordering - Error handling with logging (no exceptions on publish failure) - Start/stop lifecycle management

Rationale: - Async production ensures API responses are not blocked - Entity ID as key ensures all events for an entity go to same partition - Graceful error handling prevents API failures from Kafka issues

New Kafka Infrastructure Module

File: src/models/infrastructure/kafka/__init__.py

from models.infrastructure.kafka.kafka_producer import KafkaProducerClient

__all__ = ["KafkaProducerClient"]

Rationale: - Clean module structure for Kafka infrastructure - Follows existing pattern in s3/ and vitess/ modules

Updated Clients Class

File: src/models/rest_api/clients.py

from models.infrastructure.kafka.kafka_producer import KafkaProducerClient


class Clients(BaseModel):
    model_config = ConfigDict(arbitrary_types_allowed=True)

    s3: S3Client | None = None
    vitess: VitessClient | None = None
    property_registry: PropertyRegistry | None = None
    kafka_producer: KafkaProducerClient | None = None

    def __init__(
        self,
        s3: "S3Config",
        vitess: "VitessConfig",
        kafka_brokers: str | None = None,
        kafka_topic: str | None = None,
        property_registry_path: Path | None = None,
        **kwargs: str,
    ) -> None:
        super().__init__(
            s3=S3Client(config=s3),
            vitess=VitessClient(config=vitess),
            kafka_producer=KafkaProducerClient(
                bootstrap_servers=kafka_brokers,
                topic=kafka_topic,
            ) if kafka_brokers and kafka_topic else None,
            property_registry=(
                load_property_registry(property_registry_path)
                if property_registry_path
                else None
            ),
            **kwargs,
        )

Updated FastAPI Lifespan

File: src/models/rest_api/main.py

@asynccontextmanager
async def lifespan(app_: FastAPI) -> AsyncGenerator[None, None]:
  try:
    logger.debug("Initializing clients...")
    s3_config = settings.get_s3_config()
    vitess_config = settings.get_vitess_config()
    kafka_brokers = settings.kafka_brokers
    kafka_topic = settings.kafka_entitychange_json_topic

    logger.debug(f"Kafka config: brokers={kafka_brokers}, topic={kafka_topic}")

    app_.state.state_handler = Clients(
      s3=s3_config,
      vitess=vitess_config,
      kafka_brokers=kafka_brokers,
      kafka_topic=kafka_topic,
      property_registry_path=property_registry_path,
    )

    # Start Kafka producer
    if app_.state.state_handler.kafka_producer:
      await app_.state.state_handler.kafka_producer.start()
      logger.info("Kafka producer started")

    yield

    # Stop Kafka producer
    if app_.state.state_handler.kafka_producer:
      await app_.state.state_handler.kafka_producer.stop()
      logger.info("Kafka producer stopped")

  except Exception as e:
    logger.error(
      f"Failed to initialize clients: {type(e).__name__}: {e}", exc_info=True
    )
    raise

Rationale: - Start producer during app startup, stop during shutdown - Graceful handling of producer lifecycle - No blocking during initialization

Change Type Mapping

EditType → ChangeType mapping:

Rationale: - Clean separation between input classification and output events - Consistent naming convention (underscores) - Single source of truth for mapping logic

Entity Handler Integration

File: src/models/rest_api/handlers/entity_handler.py

class EntityHandler:
    def create_entity(self, request, vitess, s3, validator):
        # ... existing logic ...

        # Publish change event
        if clients.kafka_producer:
            change_event = EntityChangeEvent(
                entity_id=entity_id,
                revision_id=new_revision_id,
                change_type=ChangeType.CREATION,
                from_revision_id=None,
                changed_at=datetime.utcnow(),
                editor=request.editor or None,
                summary=request.edit_summary or None,
                bot=request.bot,
            )
            await clients.kafka_producer.publish_change(change_event)

Integration points: - Entity creation: Emit CREATION event - Entity update: Emit EDIT event with from_revision_id - Entity deletion: Emit SOFT_DELETE or HARD_DELETE event - Redirect creation: Emit REDIRECT event - Redirect reversion: Emit UNREDIRECT event

Rationale: - Async fire-and-forget publishing doesn't block API responses - All change events include full context (editor, summary, bot flag) - Optional producer check allows graceful degradation if Kafka unavailable

Impact

• API latency: No measurable increase (async production, fire-and-forget)
• Event coverage: 100% of entity operations now emit change events
• Downstream consumers: RDF streamers, search indexers, analytics pipelines can consume real-time changes
• Error handling: Publish failures logged but don't affect entity operations
• Scalability: Partition by entity_id ensures ordering per entity

Backward Compatibility

• Non-breaking change: Kafka producer initialization is optional
• Existing consumers: No changes required (new producer only adds functionality)
• API contracts: No changes to existing endpoints
• Graceful degradation: API works normally if Kafka is unavailable

Future Enhancements

• Add change event schema registry for versioning
• Implement dead letter queue for failed events
• Add event batching for high-throughput scenarios
• Implement event replay capability for consumers
• Add change event metrics and monitoring

[2026-01-07] Synchronous JSON Schema Validation

Summary

Replaced background validation architecture with synchronous JSON schema validation at API layer. All incoming JSON requests are now validated against existing JSON schemas before persistence, ensuring data integrity and immediate error feedback.

Motivation

• Data integrity: Catch schema violations at API boundary, prevent invalid data from entering system
• Immediate feedback: Users receive clear validation errors before data is stored
• Simplification: Removed need for background validation service, Kafka events, and cleanup jobs
• Explicit contracts: Existing JSON schemas document the expected data structure
• Error reduction: Prevent downstream failures in RDF conversion and other consumers

Changes

Deprecated Background Validation Documentation

Moved files to DEPRECATED/: - doc/ARCHITECTURE/JSON-VALIDATION-STRATEGY.md → doc/ARCHITECTURE/DEPRECATED/JSON-VALIDATION-STRATEGY.md - doc/ARCHITECTURE/POST-PROCESSING-VALIDATION.md → doc/ARCHITECTURE/DEPRECATED/POST-PROCESSING-VALIDATION.md

Added deprecation notes explaining the architectural change from Option A (background validation) to synchronous validation.

New JSON Schema Validation Utility

File: src/models/validation/json_schema_validator.py

New validator using jsonschema Python library:

class JsonSchemaValidator:
    def validate_entity_revision(self, data: dict) -> None
    def validate_statement(self, data: dict) -> None

Loads schemas from: - src/schemas/s3-revision/1.2.0/schema.json - Entity revision structure - src/schemas/s3-statement/1.0.0/schema.json - Statement structure

Updated Dependencies

File: pyproject.toml

Added dependency:

"jsonschema (>=4.23.0,<5.0.0)"

API Endpoint Validation

File: src/models/entity_api/main.py

Added JSON schema validation to POST endpoints:

1. POST /entity - Validate EntityCreateRequest.data against s3-revision schema
2. POST /redirects - Validate redirect request structure
3. POST /entities/{entity_id}/revert-redirect - Validate revert request
4. POST /statements/batch - Validate statement hashes
5. POST /statements/cleanup-orphaned - Validate cleanup request

Error Handling

File: src/models/entity_api/main.py

Added validation exception handler:

@app.exception_handler(jsonschema.ValidationError)
async def validation_error_handler(request: Request, exc: ValidationError) -> JSONResponse

Returns HTTP 400 with detailed error messages:

{
  "error": "validation_error",
  "message": "JSON schema validation failed",
  "details": [
    {
      "field": "/labels",
      "message": "Required property missing",
      "path": "#/labels"
    }
  ]
}

Impact

• API latency: +10-50ms per request (schema validation overhead)
• Data integrity: 100% of stored entities valid per schema
• Error feedback: Immediate validation errors returned to users
• Simplification: Removed need for background validation service architecture
• Testing: Schema compliance enforced before persistence

Backward Compatibility

• Breaking change: Invalid JSON that previously passed now rejected with 400 error
• API contracts: Aligns with existing JSON schema definitions
• Error codes: New validation error type added to API response format

Future Enhancements

• Optimize schema compilation and caching to reduce validation latency
• Add detailed validation metrics for monitoring
• Consider custom validators for business logic beyond JSON schema
• Add schema versioning support for schema evolution

[2026-01-05] Statement-Level Revision Tracking with Deduplication

Summary

Implemented first-class statement-level revision tracking with automatic deduplication, enabling statements to be stable, reusable objects with their own identifiers. Statements are now stored independently of entities with hash-based deduplication across all entities, reducing storage costs and enabling advanced features like most-used statement tracking and property-based loading.

Motivation

Wikibase requires statement-level tracking for:

• Storage efficiency: 20% deduplication rate expected at scale (1T statements → 800B unique)
• Cross-entity reuse: Same statement content shared across Q42, Q999, Q5000 without duplication
• Property-based loading: Frontend can load only properties needed (e.g., P31,P569 instead of all statements)
• Most-used statements: Scientific analysis of most referenced statements across all entities
• Hard delete lifecycle: Statements live forever accessible via entity revision history
• Cost reduction: 67 GB S3 storage vs 400 GB raw (6:1 compression + deduplication)

Changes

Updated Vitess Schema

File: src/models/infrastructure/vitess_client.py

Modified table: entity_revisions

ALTER TABLE entity_revisions ADD COLUMN statements JSON NOT NULL;
ALTER TABLE entity_revisions ADD COLUMN properties JSON NOT NULL;
ALTER TABLE entity_revisions ADD COLUMN property_counts JSON NOT NULL;

New columns: - statements: Array of statement hashes (64-bit integers), not full statement content - properties: Flat array of property IDs used in this revision (e.g., ["P31", "P569", "P19"]) - property_counts: Map of property_id → statement count (e.g., {"P31": 2, "P569": 1})

Modified table: statement_content

Fix: Removed duplicate table definition (lines 92-99), kept single definition (lines 81-87)

Schema:

statement_content (
    content_hash BIGINT PRIMARY KEY,  -- rapidhash of full statement JSON
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    ref_count INT DEFAULT 1,  -- Track how many entities reference this statement
    INDEX idx_ref_count (ref_count DESC)  -- Enable most-used queries
)

New VitessClient methods:

# Statement lifecycle
def insert_statement_content(self, content_hash: int) -> bool
def increment_ref_count(self, content_hash: int) -> int
def decrement_ref_count(self, content_hash: int) -> int
def get_orphaned_statements(self, older_than_days: int, limit: int) -> list[int]
def get_most_used_statements(self, limit: int, min_ref_count: int = 1) -> list[int]

# Revision queries
def get_entity_properties(self, entity_id: str, revision_id: int) -> list[str]
def get_entity_property_counts(self, entity_id: str, revision_id: int) -> dict[str, int]
def get_entity_statements_by_property(
    self, entity_id: str, revision_id: int, property_id: str
) -> list[int]

Rationale: - Hash arrays in revisions: Revisions reference statements via 64-bit hashes instead of full JSON - Property tracking: Enables intelligent frontend loading (load only needed properties) - Ref_count: Tracks statement usage for orphaned cleanup and most-used statistics - Descending index: Fast queries for most-used statements (O(log n))

Updated S3 Storage

File: src/models/infrastructure/s3_client.py

New S3 client methods:

def write_statement(self, content_hash: int, statement_data: dict) -> None:
    """Write statement to S3 (idempotent, deduplicated storage)"""
    key = f"statements/{content_hash}.json"

def read_statement(self, content_hash: int) -> dict:
    """Read statement from S3"""
    key = f"statements/{content_hash}.json"

def statement_exists(self, content_hash: int) -> bool:
    """Check if statement exists in S3"""
    key = f"statements/{content_hash}.json"

def batch_read_statements(self, content_hashes: list[int]) -> dict[int, dict]:
    """Batch fetch multiple statements from S3"""

Updated S3 revision schema: v1.1.0

New fields:

{
  "statements": [987654321012345678, 123456789012345678, ...],
  "properties": ["P31", "P569", "P19", ...],
  "property_counts": {"P31": 2, "P569": 1, "P19": 1}
}

New S3 statement schema: v1.0.0

File: src/schemas/s3-statement/1.0.0/schema.json

{
  "content_hash": 987654321012345678,
  "statement": {
    "mainsnak": {...},
    "type": "statement",
    "rank": "normal",
    "qualifiers": {...},
    "references": [...]
  },
  "created_at": "2026-01-05T09:00:00Z"
}

Storage layout: - Entity revisions: s3://wikibase-revisions/{entity_id}/rev{revision_id}.json - Statements: s3://wikibase-statements/{content_hash}.json

Rationale: - Statement granularity: Complete statement block (mainsnak + qualifiers + references) hashed together - Deduplication: Same content = one S3 object, shared across entities - Hash-only in revisions: Revisions reference hashes instead of full JSON, minimal storage - Property metadata: Enables intelligent frontend loading

New Models

File: src/models/statements/statement_hasher.py

class StatementHasher:
    @staticmethod
    def compute_hash(statement: Statement) -> int:
        """Compute rapidhash of full statement JSON"""
        # Serialize to canonical JSON (sorted keys, deterministic order)
        # Return 64-bit rapidhash integer

File: src/models/statements/extractor.py

class StatementExtractor:
    @staticmethod
    def extract_properties(entity: Entity) -> list[str]:
        """Extract unique property IDs from entity statements"""

    @staticmethod
    def compute_property_counts(entity: Entity) -> dict[str, int]:
        """Count statements per property"""

Rationale: - StatementHasher: Canonical JSON serialization ensures consistent hashes for identical content - StatementExtractor: Extract property metadata for intelligent loading

New API Endpoints

File: src/models/entity_api/main.py

Statement endpoints:

GET /statement/{hash}
  → Returns: Full statement JSON from S3

POST /statements/batch
  → Request: {hashes: [hash1, hash2, ...]}
  → Returns: {results: {hash1: {...}, hash2: {...}, ...}}

GET /statement/most_used
  → Query params: limit=1000&min_ref_count=10&property_range=P0-P999&sort_by=ref_count_desc
  → Returns: [hash1, hash2, ...]

Property endpoints:

GET /entity/{id}/properties
  → Returns: ["P31", "P569", "P19", ...]

GET /entity/{id}/properties/counts
  → Returns: {"P31": 2, "P569": 1, "P19": 1}

GET /entity/{id}/properties/P31,P569
  → Returns: [hash1, hash2, hash3, ...]

Modified entity endpoints:

GET /entity/{id}?resolve_statements=false
  → Returns: Metadata + statement hashes (no full statements)

GET /entity/{id}?resolve_statements=true&properties=P31,P569
  → Returns: Metadata + full statements for specific properties only

Rationale: - Statement endpoints: First-class citizen access to statements - Property endpoints: Enable intelligent frontend loading - Optional resolution: Frontend controls when to fetch full statements

Entity Creation Flow Updates

File: src/models/entity_api/main.py

New workflow:

# 1. Extract statements from entity
statements = entity.statements

# 2. Hash each statement
statement_hashes = [StatementHasher.compute_hash(s) for s in statements]

# 3. Deduplicate and store statements
for stmt, hash_val in zip(statements, statement_hashes):
    s3_client.write_statement(hash_val, stmt.to_dict())
    vitess.insert_statement_content(hash_val)
    vitess.increment_ref_count(hash_val)

# 4. Extract property metadata
properties = StatementExtractor.extract_properties(entity)
property_counts = StatementExtractor.compute_property_counts(entity)

# 5. Build revision with hash array + metadata
revision = {
    "statements": statement_hashes,
    "properties": properties,
    "property_counts": property_counts,
    ...
}

# 6. Write revision to S3
s3_client.write_entity_revision(entity_id, revision_id, revision)

# 7. Insert revision metadata to Vitess
vitess.insert_revision(entity_id, revision_id, statements=statement_hashes, ...)

# 8. Update entity head
vitess.update_head(entity_id, revision_id)

Rationale: - Deduplication: Same statement content writes to same S3 object and Vitess row - Ref_count tracking: Incremented for each entity using the statement - Property extraction: Computed once during entity creation

Entity Read Flow Updates

File: src/models/entity_api/main.py

New workflow:

# 1. Get revision from S3 (contains hashes, not full statements)
revision = s3_client.read_entity_revision(entity_id, revision_id)

# 2. If frontend requests full statements
if resolve_statements:
    statements = s3_client.batch_read_statements(revision["statements"])
else:
    statements = []  # Return hashes only

# 3. If frontend requests specific properties
if properties_filter:
    # Filter hashes by property
    filtered_hashes = filter_hashes_by_property(revision, properties_filter)
    statements = s3_client.batch_read_statements(filtered_hashes)

# 4. Return response
return {
    "metadata": revision["metadata"],
    "statements": statements,  # or hashes if not resolved
    "properties": revision["properties"],
    "property_counts": revision["property_counts"]
}

Rationale: - Hash-only by default: Minimal response size, frontend controls loading - Property-based filtering: Load only statements for specific properties - Batch fetching: Efficiently fetch multiple statements in parallel

Hard Delete Flow Updates

File: src/models/entity_api/main.py

New workflow:

# 1. Get all statement hashes from entity revisions
revisions = vitess.get_history(entity_id)
all_hashes = []
for rev in revisions:
    all_hashes.extend(rev["statements"])

# 2. Decrement ref_count for each statement
for hash_val in all_hashes:
    vitess.decrement_ref_count(hash_val)

# 3. Mark entity as deleted
vitess.mark_entity_deleted(entity_id)

# 4. Schedule orphaned cleanup (background job)
# Scheduled job runs daily:
orphaned = vitess.get_orphaned_statements(older_than_days=180, limit=10000)
for hash_val in orphaned:
    s3_client.delete_statement(hash_val)
    vitess.delete_statement_content(hash_val)