Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Troubleshooting

Common issues and diagnostic procedures for the Content Manager plugin.

Common errors

“Cannot reach the Wazuh Engine: its API socket is not available”

The Wazuh Engine is not running or the Unix socket is not accessible.

Resolution:

  1. Check the socket file exists:

    ls -la /usr/share/wazuh-indexer/engine/sockets/engine-api-http.sock

  2. Ensure the Wazuh Indexer process has permission to access the socket file.

Subscription content is not synchronized

No CTI access token has been registered, so the instance synchronizes only the public content. GET /_plugins/_content_manager/subscription reports "is_registered": false in this state.

Resolution

Register credentials by posting the CTI access token:

curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X POST \
  "https://127.0.0.1:9200/_plugins/_content_manager/subscription" \
  -H 'Content-Type: application/json' \
  -d '{
    "access_token": "<your-cti-access-token>"
  }'

A successful registration returns {"message":"Access token received successfully.","status":201}. The token is persisted in .wazuh-internal-state and loaded into memory immediately.

Sync not running

Content is not being updated despite having a valid subscription.

Diagnosis

  1. Check consumer state and offsets:

    curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X GET \
  "https://127.0.0.1:9200/.wazuh-cti-consumers/_search?pretty"

    If local_offset equals remote_offset, the content is already up-to-date.

  2. Check the sync job is registered and enabled:

    curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X GET \
  "https://127.0.0.1:9200/.wazuh-content-manager-jobs/_search?pretty"

    Verify the wazuh-catalog-sync-job document has "enabled": true and that its schedule interval matches your configuration. The plugin keeps this document in step with the settings automatically, so a mismatch means reconciliation has not run — check the log for a Catalog Sync Job reconciled entry rather than editing the document by hand.

  3. Check the effective value of the scheduled sync setting. It can be set in opensearch.yml:

    plugins.content_manager.catalog.update_on_schedule: true

    It is also dynamic, and a value set through the Cluster Settings API overrides opensearch.yml, so check there too:

    curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X GET \
  "https://127.0.0.1:9200/_cluster/settings?pretty"

    When the setting is disabled, every scheduled run is refused and logged:

    INFO   ... Scheduled catalog synchronization (ID: wazuh-catalog-sync-job) skipped: plugins.content_manager.catalog.update_on_schedule is false.

  4. Trigger a manual sync to test:

    curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X POST \
  "https://127.0.0.1:9200/_plugins/_content_manager/update"

Engine validation rejects a temporary field

Symptoms: Creating or updating a decoder (or any resource with a check/detection expression) fails with an Engine validation error naming a field that looks correct, for example:

{"message":"Engine validation failed. Failed to validate resource of type 'decoder': Validation failed for 'decoder': Failed to build operation 'tmp_json.event.action: string_equal(\"netflow_flow\")': Field 'tmp_json.event.action' is not defined in WCS schema and is not a temporary field"}

Cause: The Engine validates every field referenced in a check or detection expression against the Wazuh Common Schema (WCS). A field that is intentionally temporary — used only during decoding and not part of the final normalized event — is not in WCS by definition, so the Engine only accepts it if it’s prefixed with an underscore.

Resolution

Prefix temporary fields with _ in both the check expression and anywhere else the field is referenced within the same resource:

{
  "check": [
    {
      "_tmp_json.event.action": "string_equal(\"netflow_flow\")"
    }
  ]
}

Fields that are part of WCS (e.g., event.action, source.ip) never need the underscore prefix — only add it for genuinely temporary, decoder-internal fields.

Socket file not found

The Unix socket used for Engine communication does not exist.

Expected path: /usr/share/wazuh-indexer/engine/sockets/engine-api-http.sock

Resolution

  1. Verify the Wazuh Engine is installed and running.
  2. Check the Engine configuration for the socket path.
  3. Ensure the engine/sockets/ directory exists under the Wazuh Indexer installation path.

Sync fails with a request timeout during a CTI rate-limit

A synchronization pass fails and the log shows a timeout on a CTI request, e.g.:

Error during content update: Timeout deadline: 10000 MILLISECONDS, actual: 10000 MILLISECONDS

This means the CTI API rate-limited the request (HTTP 429). The client now honors the response Retry-After header and retries automatically (see CTI rate-limit retries), so a transient rate-limit self-recovers within the pass.

Resolution

  1. If the failure is transient, no action is needed — the request is retried and the next scheduled sync resumes from the last checkpoint (no data is lost or duplicated).
  2. If 429s persist across passes, the instance is being rate-limited by CTI for longer than the retry budget. Increase plugins.content_manager.client.max_retries and/or plugins.content_manager.client.retry_backoff_base_seconds.
  3. Look for the warning CTI API rate-limited request [...] (HTTP 429); retrying in {}s in the logs to confirm the retry path is engaging.

Resource creation fails with a missing indices:admin/create permission

A POST to /rules, /decoders, /integrations, /kvdbs or /filters returns HTTP 429 with Too many concurrent requests creating this resource. Please retry. even though there is no concurrency, and the indexer log shows the security plugin denying indices:admin/create on the lock index for the requesting user:

[INFO ][o.o.s.p.PrivilegesEvaluatorImpl] No index-level perm match for User [name=<caller>, ...]
  Resolved [... allIndices=[.wazuh-content-manager-resource-locks] ...]:
  Insufficient permissions for the referenced index [Action [indices:admin/create]]
[WARN ][c.w.c.t.AbstractTransportCreateAction] Failed to acquire resource-creation lock for [integration]:
  no permissions for [indices:admin/create] and User [name=<caller>, ...]

The .wazuh-content-manager-resource-locks index is created at node startup and every operation on it runs as the plugin, so this can only happen on a node that never completed that startup step (check the log for Failed to create .wazuh-content-manager-resource-locks index, due to: ...).

Note that the index is hidden and has no alias, so its absence cannot be confirmed with _cat/aliases or a plain _cat/indices.

Resolution

  1. Confirm whether the index exists:

    curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X GET "https://127.0.0.1:9200/_cat/indices/.wazuh-content-manager-resource-locks?expand_wildcards=all&v"

  2. If it is missing, restart the node — the plugin recreates it during initialization — and check the startup log for the reason it failed the first time (a red cluster or a node still joining are the usual causes).

  3. If a restart is not an option, create it manually with an account that holds indices:admin/create (for example admin, as below). Only the acquired_at field is needed:

    curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X PUT "https://127.0.0.1:9200/.wazuh-content-manager-resource-locks" \
  -H 'Content-Type: application/json' -d '{
    "settings": { "index": { "number_of_replicas": 0, "auto_expand_replicas": "0-1", "hidden": true, "refresh_interval": "-1" } },
    "mappings": { "dynamic": "strict", "properties": { "acquired_at": { "type": "long" } } }
  }'

  4. Do not add indices:admin/create to the user’s role to work around this. No role needs index-level privileges on the lock index — see Access control — Plugin-internal indices.

Diagnostic commands

Check consumer state

View synchronization state for all content contexts:

curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X GET \
  "https://127.0.0.1:9200/.wazuh-cti-consumers/_search?pretty"

Example output:

{
  "hits": {
    "hits": [
      {
        "_id": "cti:catalog:consumer:ruleset",
        "_source": {
          "name": "public-ruleset-5",
          "context": "beta-2-ruleset-5",
          "type": "cti:catalog:consumer:ruleset",
          "resource": "https://api.pre.cloud.wazuh.com/api/v1/catalog/contexts/beta-2-ruleset-5/consumers/public-ruleset-5",
          "is_public": true,
          "status": "ready",
          "local_offset": 3932,
          "remote_offset": 3932,
          "pending_sync_phases": []
        }
      }
    ]
  }
}
  • status == ready: Sync is complete; content is safe to read.
  • status == running: Sync is in progress. If this persists after a sync should have finished, the node process may have been interrupted (e.g. killed) mid-cycle without a chance to record failed.
  • status == failed: The previous sync cycle was interrupted by an unexpected exception. Check the Content Manager logs around the time this consumer was last synced; the job retries automatically on its next scheduled run.
  • local_offset == remote_offset: Content is up-to-date.
  • local_offset < remote_offset: Content needs updating.
  • local_offset == 0: Content has never been synced (snapshot required).
  • pending_sync_phases: for the ruleset consumer, lists which of integrations, rules, and detectors failed to sync to Ruleset Management and are retried on the next scheduled sync; empty means all sub-phases are in sync.

Check sync job

View the periodic sync job configuration:

curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X GET \
  "https://127.0.0.1:9200/.wazuh-content-manager-jobs/_search?pretty"

Count content documents

Check how many rules, decoders, etc. have been indexed:

# Rules
curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X GET "https://127.0.0.1:9200/wazuh-threatintel-rules/_count?pretty"

# Decoders
curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X GET "https://127.0.0.1:9200/wazuh-threatintel-decoders/_count?pretty"

# Integrations
curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X GET "https://127.0.0.1:9200/wazuh-threatintel-integrations/_count?pretty"

# KVDBs
curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X GET "https://127.0.0.1:9200/wazuh-threatintel-kvdbs/_count?pretty"

# IoCs
curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X GET "https://127.0.0.1:9200/wazuh-threatintel-enrichments/_count?pretty"

Job scheduling on startup

During node startup, the catalog sync job and the update check job both require the .wazuh-content-manager-jobs index to reach yellow status with at least one active shard before they can register their job documents. On a freshly initialized or resource-constrained cluster this can time out, producing entries like:

INFO   ... Failed to schedule Telemetry Ping Job: Index .wazuh-content-manager-jobs not ready
INFO   ... Retrying Telemetry Ping Job (attempt 1/3) in 15s.

The plugin automatically retries each registration up to plugins.content_manager.job_schedule.max_retries times (default 3) with a linear backoff of plugins.content_manager.job_schedule.retry_backoff_seconds × attempt (default 15 s, so 15 s, 30 s, 45 s). Each attempt logs the failure reason and the scheduled retry delay at INFO — these are expected during startup and do not require action.

If all retries fail, the plugin logs ERROR ... Giving up scheduling <job> after N attempts. and the job will only be retried on the next node start. Raising plugins.content_manager.job_schedule.max_retries gives a slow-starting cluster more room. A persistent failure usually indicates the cluster cannot allocate shards — check cluster health with GET _cluster/health and verify index allocation settings.

Log monitoring

Content Manager logs are part of the Wazuh Indexer logs. Use the following patterns to filter relevant entries:

# General Content Manager activity
grep -i "content.manager\|ContentManager\|CatalogSync" \
  /var/log/wazuh-indexer/wazuh-indexer.log

# Sync job execution
grep -i "CatalogSyncJob\|consumer-sync" \
  /var/log/wazuh-indexer/wazuh-indexer.log

# CTI API communication
grep -i "cti\|CTIClient" \
  /var/log/wazuh-indexer/wazuh-indexer.log

# Engine socket communication
grep -i "engine.*socket\|EngineClient" \
  /var/log/wazuh-indexer/wazuh-indexer.log

# Errors only
grep -i "ERROR.*content.manager" \
  /var/log/wazuh-indexer/wazuh-indexer.log

Resetting content

To force a full re-sync from snapshot, delete the consumer state documents and restart the indexer:

# Delete consumer state (forces snapshot on next sync)
curl -sk -u admin:$WAZUH_INDEXER_ADMIN_PASSWORD -X POST \
  "https://127.0.0.1:9200/.wazuh-cti-consumers/_delete_by_query?refresh=true" \
  -H 'Content-Type: application/json' \
  -d '{"query": {"match_all": {}}}'

# Restart indexer to trigger sync
systemctl restart wazuh-indexer

Warning: This will re-download and re-index all content from scratch. Use only when troubleshooting persistent sync issues.