Variable in delivery_stream_name

[rolandjitsu]

It looks like this plugin does not like variable names in the delivery_stream_name:

2024-02-28 10:43:05 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2024-02-28 10:43:07 +0000 chunk="6126ece798cd9e545512cad26b6ecca8" error_class=Aws::Firehose::Errors::ValidationException error="1 validation error detected: Value at 'deliveryStreamName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+"
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.191.3/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/plugins/response_target.rb:24:in `call'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/aws-sdk-core-3.191.3/lib/seahorse/client/request.rb:72:in `send_request'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/aws-sdk-firehose-1.65.0/lib/aws-sdk-firehose/client.rb:1972:in `put_record_batch'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-kinesis-3.4.2/lib/fluent/plugin/out_kinesis_firehose.rb:52:in `block in write'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-kinesis-3.4.2/lib/fluent/plugin/kinesis_helper/api.rb:94:in `batch_request_with_retry'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-kinesis-3.4.2/lib/fluent/plugin/kinesis.rb:157:in `block in write_records_batch'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-kinesis-3.4.2/lib/fluent/plugin/kinesis_helper/api.rb:82:in `block in split_to_batches'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/event.rb:315:in `each'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/event.rb:315:in `block in each'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin/buffer/memory_chunk.rb:81:in `open'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin/buffer/memory_chunk.rb:81:in `open'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/event.rb:314:in `each'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-kinesis-3.4.2/lib/fluent/plugin/kinesis_helper/api.rb:79:in `each'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-kinesis-3.4.2/lib/fluent/plugin/kinesis_helper/api.rb:79:in `split_to_batches'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-kinesis-3.4.2/lib/fluent/plugin/kinesis.rb:155:in `write_records_batch'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-kinesis-3.4.2/lib/fluent/plugin/out_kinesis_firehose.rb:48:in `write'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin/output.rb:1180:in `try_flush'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin/output.rb:1501:in `flush_thread_run'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'
  2024-02-28 10:43:05 +0000 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

And the config is:

<match {events,metrics}>
  @type kinesis_firehose
  delivery_stream_name ${tag}-stream
  region ap-southeast-1
  aws_key_id keyid
  aws_sec_key secret

  <buffer>
    flush_interval 5
    chunk_limit_size 1m
    flush_thread_interval 1
    flush_thread_burst_interval 0.01
    flush_thread_count 15
  </buffer>
</match>

Any idea how to get it working?

[rolandjitsu]

Hmm, maybe syntax is implemented differently as I can see something about placeholders in https://github.com/awslabs/aws-fluent-plugin-kinesis?tab=readme-ov-file#delivery_stream_name.

So would:

delivery_stream_name "${$.tag}-stream"

work?

[simukappu]

The plugin supports this delivery_stream_name ${tag}-stream format since it is tested here:

aws-fluent-plugin-kinesis/test/plugin/test_out_kinesis_firehose.rb

Line 215 in fac107f

"delivery_stream_name" => "stream-placeholder-${tag}",

Can you confirm whether your tag variable violated Kinesis Firehose delivery stream name regulations? regular expression pattern: [a-zA-Z0-9_.-]+

[rolandjitsu]

Yes, the stream name is based on:

<match {events,metrics}>
  @type copy
  <store>
    @type kinesis_firehose
    delivery_stream_name ${tag}-stream
    region ap-southeast-1
    aws_key_id keyid
    aws_sec_key secret

    <buffer>
      flush_interval 5
      chunk_limit_size 1m
      flush_thread_interval 1
      flush_thread_burst_interval 0.01
      flush_thread_count 15
    </buffer>
  </store>
</match>

So I'd expect it to be: events-stream and metrics-stream; which is compliant with the regex.

I have something similar setup with https://github.com/uken/fluent-plugin-elasticsearch and that works (impl. is probably different, but the tag matches the ones in the match statement).

I run td-agent, installed on Ubuntu 18.04.6 LTS via apt, with the following version:

td-agent 4.4.1 fluentd 1.15.2 (c32842297ed2c306f1b841a8f6e55bdd0f1cb27f)

[simukappu]

When you specify delivery_stream_name configuration with built-in placeholder, you should specify the corresponding attributes in buffer section. Can you try this?
https://docs.fluentd.org/configuration/buffer-section#placeholders

<match {events,metrics}>
  @type copy
  <store>
    @type kinesis_firehose
    delivery_stream_name ${tag}-stream
    region ap-southeast-1
    aws_key_id keyid
    aws_sec_key secret

    <buffer tag>
      flush_interval 5
      chunk_limit_size 1m
      flush_thread_interval 1
      flush_thread_burst_interval 0.01
      flush_thread_count 15
    </buffer>
  </store>
</match>