amazon web services - Opensearch Ingestion Pipeline: reference a dynamic field - Stack Overflow

admin管理员组

文章数量:1435859

Opensearch Ingestion Pipeline reads from S3 bucket with a file:

{"key1": "value1" , "value1": "54", "key3": "value10"}
{"key1": "value3" , "value3": "65", "key3": "value9"}

Pipeline configuration

version: "2"
log-pipeline:
  source:
    s3:
      codec:
        newline: null
      compression: "none"
      aws:
        region: "eu-west-2"
        sts_role_arn: "arn:aws:iam::{account_no}:role/s3-os-ingestion-pipeline-role"
      acknowledgments: true
      scan:
        buckets:
          - bucket:
              name: "tempesdata"
      delete_s3_objects_on_read: false
  processor:
    - parse_json:
    - delete_entries:
        with_keys: ["s3", "message"]
  sink:
    - opensearch:
        hosts: [ "{host}" ]
        aws:
          sts_role_arn: "arn:aws:iam::{account_no}:role/s3-os-ingestion-pipeline-role"
          region: "eu-west-2"
          serverless: false
        index: "test"
        document_id: ${/key3} # UPDATE THIS

Problem is i want document id to be concat of key1's value's value + ${/key3}, Something like $/{${/key1}}-${/key3}. But this does not work. In the given example, I would want documents to be indexed with 54-value10, 65-value9

本文标签： amazon web servicesOpensearch Ingestion Pipeline reference a dynamic fieldStack Overflow