log groups streams are not been collected by fluentd

[haiakab]

Problem

Hi, I'm using Fluentd agent to scrape streams from CloudWatch log groups. The issue I'm facing is that data is received only for specific log groups (those with names that start with /aws/lambda). These kinds of log groups have a specific template for log stream names, such as:
2024/07/04/[$LATEST]80cd86893da74d0783015c4173ce6eda

**We have permissions to fetch all log groups other than /aws/lambda using ROLE_ARN from AWS CLI

Attached is the Fluentd configuration:

system:
log_level: error
log_file: /var/log/fluent/fluentd.log

config:

• !include config.d/*.yaml
• match:
  $type: forward
  server:
  name: HOST
  host: HOST
  port: 3000
• source:
  $type: cloudwatch_logs
  tag: cloudwatch.input1
  log_group_name: LOG_GROUP_NAME
  use_todays_log_stream: true
  include_metadata: true
  region: REGION
  aws_use_sts: true
  aws_sts_role_arn: ROLE_ARN
  parse:
  $type: none

** no errors in fluentd logs

Using Fluentd and CloudWatchLogs plugin versions

fluentd 1.16.3, within docker
OS:
NAME="Amazon Linux"
VERSION="2"

• Dependent gem versions
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock:

aws-eventstream (1.3.0)
aws-partitions (1.907.0, 1.867.0, 1.866.0)
aws-sdk-cloudwatch (1.84.0)
aws-sdk-cloudwatchlogs (1.76.0, 1.71.0)
aws-sdk-core (3.191.6, 3.190.0)
aws-sdk-kms (1.74.0)
aws-sdk-s3 (1.141.0)
aws-sigv4 (1.8.0)

[haiakab]

currently I'm getting this error in fluentd logs:

2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:293:in block in get_events' 2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:331:in throttling_handler'
2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:279:in get_events' 2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:209:in block (2 levels) in run'
2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:207:in each' 2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:207:in block in run'
2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:201:in each' 2024-07-07 13:49:20 +0000 [error]: #0 /etc/fluent/plugin/in_cloudwatch_logs.rb:201:in run'
2024-07-07 13:49:20 +0000 [error]: #0 /opt/fluent/lib/ruby/gems/3.2.0/gems/fluentd-1.16.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2024-07-07 13:49:20 +0000 [error]: Worker 0 exited unexpectedly with status 1

** I'm not sure if it's affecting the case I described, as I'm not receiving it consistently