Issue at Graylog

[gow1998]

Hi team,
I am facing some issue at graylog installation in sunbird setup. I am done that Graylog setup But I can't able to check the logs in UI, can you please help anyone to sort out this issue.
Also, I include the ES logs and Graylog logs at below,


ES Logs:
[2023-02-21T13:29:37,789][WARN ][i.n.c.DefaultChannelPipeline] [10.0.2.7] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) ~[?:?]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:110) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.internal.SocketUtils$5.run(SocketUtils.java:107) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121]
at io.netty.util.internal.SocketUtils.accept(SocketUtils.java:107) ~[netty-common-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:143) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:75) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

Graylog Logs
2023-02-21T13:31:02.438Z WARN [MessagesAdapterES6] Failed to index message: index=<graylog_5> id= error=<{"type":"unavailable_shards_exception","reason":"[graylog_5][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_5][0]] containing [149] requests]"}>
2023-02-21T13:31:02.438Z WARN [MessagesAdapterES6] Failed to index message: index=<graylog_5> id= error=<{"type":"unavailable_shards_exception","reason":"[graylog_5][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_5][0]] containing [149] requests]"}>
2023-02-21T13:31:02.438Z WARN [MessagesAdapterES6] Failed to index message: index=<graylog_5> id= error=<{"type":"unavailable_shards_exception","reason":"[graylog_5][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_5][0]] containing [149] requests]"}>
2023-02-21T13:31:02.438Z WARN [MessagesAdapterES6] Failed to index message: index=<graylog_5> id= error=<{"type":"unavailable_shards_exception","reason":"[graylog_5][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_5][0]] containing [149] requests]"}>
2023-02-21T13:31:02.438Z WARN [MessagesAdapterES6] Failed to index message: index=<graylog_5> id= error=<{"type":"unavailable_shards_exception","reason":"[graylog_5][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_5][0]] containing [149] requests]"}>
2023-02-21T13:31:02.438Z WARN [MessagesAdapterES6] Failed to index message: index=<graylog_5> id= error=<{"type":"unavailable_shards_exception","reason":"[graylog_5][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_5][0]] containing [149] requests]"}>
2023-02-21T13:31:02.438Z WARN [MessagesAdapterES6] Failed to index message: index=<graylog_5> id= error=<{"type":"unavailable_shards_exception","reason":"[graylog_5][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_5][0]] containing [149] requests]"}>
2023-02-21T13:31:02.438Z WARN [MessagesAdapterES6] Failed to index message: index=<graylog_5> id= error=<{"type":"unavailable_shards_exception","reason":"[graylog_5][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_5][0]] containing [149] requests]"}>
2023-02-21T13:31:02.438Z WARN [MessagesAdapterES6] Failed to index message: index=<graylog_5> id= error=<{"type":"unavailable_shards_exception","reason":"[graylog_5][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_5][0]] containing [149] requests]"}>
2023-02-21T13:31:02.438Z WARN [MessagesAdapterES6] Failed to index message: index=<graylog_5> id= error=<{"type":"unavailable_shards_exception","reason":"[graylog_5][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_5][0]] containing [149] requests]"}>
2023-02-21T13:31:02.438Z WARN [MessagesAdapterES6] Failed to index message: index=<graylog_5> id= error=<{"type":"unavailable_shards_exception","reason":"[graylog_5][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[graylog_5][0]] containing [149] requests]"}>

Mongodb Logs
{"t":{"$date":"2023-02-21T13:30:02.221+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"WTCheckpointThread","msg":"WiredTiger message","attr":{"message":"[1676986202:221702][23161:0x7feac2fbd700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 209673, snapshot max: 209673 snapshot count: 0, oldest timestamp: (1676986197, 3) , meta checkpoint timestamp: (1676986202, 3) base write gen: 579661"}}
{"t":{"$date":"2023-02-21T13:31:02.273+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"WTCheckpointThread","msg":"WiredTiger message","attr":{"message":"[1676986262:273205][23161:0x7feac2fbd700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 209928, snapshot max: 209928 snapshot count: 0, oldest timestamp: (1676986257, 1) , meta checkpoint timestamp: (1676986262, 1) base write gen: 579661"}}
{"t":{"$date":"2023-02-21T13:32:02.318+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"WTCheckpointThread","msg":"WiredTiger message","attr":{"message":"[1676986322:318209][23161:0x7feac2fbd700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 210228, snapshot max: 210228 snapshot count: 0, oldest timestamp: (1676986317, 1) , meta checkpoint timestamp: (1676986322, 1) base write gen: 579661"}}

[gandham-santhosh]

@gow1998 can you check if the elasticsearch cluster is healthy? Looks like there is an issue there

@saiakhil46

[gow1998]

{
"cluster_name" : "dev-log",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1609,
"active_shards" : 1609,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2712,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 37.23675075214071
}
@gandham-santhosh The elasticsearch cluster is in unhealthy state

[gow1998]

@gandham-santhosh @saiakhil46
In elasticsearch, the shards as still unassigned. I am trying to update the unassigned shards, after reaching some percentage of active shards the elasticsearch status as turns failed.
{
"cluster_name" : "dev-log",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1356,
"active_shards" : 1356,
"relocating_shards" : 0,
"initializing_shards" : 4,
"unassigned_shards" : 2961,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 10,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 267972,
"active_shards_percent_as_number" : 31.381624623929643
}

[saiakhil46]

@gow1998 As this is new setup, try to delete all indices and restart graylog and elasticsearch service and then check.

[gow1998]

Its worked. Thank you @saiakhil46

[chitranshu-keshav]

@gow1998 can you please mark this thread as answered