Skip to content

Commit

Permalink
Protected the automated adding of event properties as Loki labels beh…
Browse files Browse the repository at this point in the history
…ind a feature flag eventPropertiesAsLabels and documented it. Warned users when using the feature flag.
  • Loading branch information
corentinaltepe committed Aug 10, 2023
1 parent 2cf2377 commit 16f35a6
Show file tree
Hide file tree
Showing 3 changed files with 17 additions and 25 deletions.
21 changes: 4 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ Under .NET Core, [remember to register](https://github.com/nlog/nlog/wiki/Regist
orderWrites="true"
compressionLevel="noCompression"
layout="${level}|${message}${onexception:|${exception:format=type,message,method:maxInnerExceptionLevel=5:innerFormat=shortType,message,method}}|source=${logger}"
eventPropertiesAsLabels="false"
sendLastFormatParameter="false"
proxyUrl="http://proxy:8888"
proxyUser="username"
Expand Down Expand Up @@ -96,7 +97,9 @@ Under .NET Core, [remember to register](https://github.com/nlog/nlog/wiki/Regist
`layout` - While it is possible to define a simple layout structure in the attributes of the target configuration,
prefer using a JsonLayout to structure your logs. This will allow better parsing in Grafana Loki.

`sendLastFormatParameter`: enables the last parameter of a log message format to be sent to Loki as separate fields per property. Example:
`eventPropertiesAsLabels`: creates one Grafana Loki's label per event property. Beware, this goes against [Grafana Loki's best practices](https://grafana.com/docs/loki/latest/best-practices/) since _Too many label value combinations leads to too many streams._ In order to structure your logs, you are advised to keep away from this feature and to use the `JsonLayout` provided in the example.

`sendLastFormatParameter`: enables the last parameter of a log message format to be sent to Grafana Loki as separate fields per property. Feature `eventPropertiesAsLabels` must be `true` as well. Example:

```csharp
// using simple anonymous type object to create custom fields for a log entry
Expand Down Expand Up @@ -163,22 +166,6 @@ These raw messages would look like the following in Grafana Loki :
See [Log Queries and Parser Expressions in Loki](https://grafana.com/docs/loki/latest/logql/log_queries/#parser-expression)
for more details.

### Additional Properties

Like Gelf target, NLog.Loki reads custom fields from the log event properties. It significantly streamlines the migration process from Graylog to Loki. Example:

```csharp
// using additional properties to create custom fields for a log entry
var eventInfo = new LogEventInfo
{
Message = "Testing additional properties inside LogEventInfo",
Level = LogLevel.Info,
};
eventInfo.Properties.Add("publisher", "ACME Publisher");
eventInfo.Properties.Add("releaseDate", DateTime.Now);
log.Log(eventInfo);
```

### Benchmark

See [NLog.Loki.Benchmarks](https://github.com/corentinaltepe/nlog.loki.benchmark) for benchmark between HTTP and gRPC clients for NLog targets for Grafana Loki.
18 changes: 11 additions & 7 deletions src/NLog.Loki/LokiTarget.cs
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ public class LokiTarget : AsyncTaskTarget
public Layout Password { get; set; }

public bool SendLastFormatParameter { get; set; }
public bool EventPropertiesAsLabels { get; set; }

/// <summary>
/// Orders the logs by timestamp before sending them to Loki. False by default.
Expand Down Expand Up @@ -82,14 +83,15 @@ private IEnumerable<LokiEvent> GetLokiEvents(IEnumerable<LogEventInfo> logEvents

private LokiEvent GetLokiEvent(LogEventInfo logEvent)
{
var labels = new LokiLabels(RenderAndMapLokiLabels(Labels, logEvent, SendLastFormatParameter));
var labels = new LokiLabels(RenderAndMapLokiLabels(Labels, logEvent, SendLastFormatParameter, EventPropertiesAsLabels));
return new LokiEvent(labels, logEvent.TimeStamp, RenderLogEvent(Layout, logEvent));
}

private static ISet<LokiLabel> RenderAndMapLokiLabels(
IList<LokiTargetLabel> lokiTargetLabels,
LogEventInfo logEvent,
bool sendLastFormatParameter)
bool sendLastFormatParameter,
bool eventPropertiesAsLabels)
{
var set = new HashSet<LokiLabel>();
for(var i = 0; i < lokiTargetLabels.Count; i++)
Expand All @@ -108,11 +110,13 @@ private static ISet<LokiLabel> RenderAndMapLokiLabels(
}
}

// programmer might also want to create fields in loki using event properties
foreach(var property in logEvent.Properties)
{
_ = set.Add(new LokiLabel(property.Key.ToString(), property.Value?.ToString() ?? ""));
}
// programmer might also want to create labels in loki using event properties
// This goes against Loki best pratices as it tends to create too many streams.
// But the feature was requested twice in a short span so it is included in the library,
// with warnings in the readme.
if(eventPropertiesAsLabels)
foreach(var property in logEvent.Properties)
_ = set.Add(new LokiLabel(property.Key.ToString(), property.Value?.ToString() ?? ""));

return set;
}
Expand Down
3 changes: 2 additions & 1 deletion src/Template/nlog.config
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@
retryCount="3"
orderWrites="true"
compressionLevel="fastest"
sendLastFormatParameter="true">
eventPropertiesAsLabels="false"
sendLastFormatParameter="false">
<label name="app" layout="template" />
<layout xsi:type="JsonLayout" includeEventProperties="true">
<attribute name="level" layout="${level} "/>
Expand Down

0 comments on commit 16f35a6

Please sign in to comment.