ASP.NET Core logging with NLog and Elasticsearch

This article shows how to Log to Elasticsearch using NLog in an ASP.NET Core application. NLog is a free open-source logging for .NET.

Code: https://github.com/damienbod/AspNetCoreNlog

2020-01-12 Updated to .NET Core 3.1, NLog.Web.AspNetCore 4.9.0
2018-04-04 Updated to NLog.Web.AspNetCore 4.5.1, ASP.NET Core 2.0
2017.02.08 Updated to NLog.Web.AspNetCore 4.3.0 and VS2017 RC3
17.12.2016 Updated to ASP.NET Core 1.1

NLog posts in this series:

ASP.NET Core logging with NLog and Microsoft SQL Server
ASP.NET Core logging with NLog and Elasticsearch
Settings the NLog database connection string in the ASP.NET Core appsettings.json
.NET Core logging to MySQL using NLog
.NET Core logging with NLog and PostgreSQL

NLog.Extensions.Logging is required to use NLog in an ASP.NET Core application. This is added to the dependencies of the project. NLog.Targets.ElasticSearch is also added to the dependencies. This project is at present NOT the NuGet package from ReactiveMarkets, but the source code from ReactiveMarkets and updated to dotnetcore. Thanks to ReactiveMarkets for this library, hopefully the NuGet package will be updated and the NuGet package can be used directly.

The NLog configuration file also needs to be added to the publishOptions in the csproj file.

<Project Sdk="Microsoft.NET.Sdk.Web">
  
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="NLog.Targets.ElasticSearch" Version="7.1.1" />
    <PackageReference Include="NLog.Web.AspNetCore" Version="4.9.0" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="3.1.0" PrivateAssets="All" />
    <PackageReference Include="System.Data.SqlClient" Version="4.8.0" />
  </ItemGroup>
  
  <ItemGroup>
    <DotNetCliToolReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Tools" Version="2.0.0" />
  </ItemGroup>
  
  <ItemGroup>
    <Content Update="nlog.config">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </Content>
  </ItemGroup>
</Project>

The NLog configuration is added to the Startup.cs class in the Configure method.

public void ConfigureServices(IServiceCollection services)
{
	services.AddSingleton<IHttpContextAccessor, HttpContextAccessor>();
	// Add framework services.
	services.AddControllers();

	services.AddScoped<LogFilter>();
}

and the program class:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using NLog;
using NLog.Web;

namespace AspNetCoreNlog
{
    public static class Program
    {
        public static void Main(string[] args)
        {
            var logger = NLog.Web.NLogBuilder.ConfigureNLog("nlog.config").GetCurrentClassLogger();
            try
            {
                logger.Debug("init main");
                CreateHostBuilder(args).Build().Run();
            }
            catch (Exception exception)
            {
                //NLog: catch setup errors
                logger.Error(exception, "Stopped program because of exception");
                throw;
            }
            finally
            {
                // Ensure to flush and stop internal timers/threads before application-exit (Avoid segmentation fault on Linux)
                NLog.LogManager.Shutdown();
            }
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
          Host.CreateDefaultBuilder(args)
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseStartup<Startup>();
            })
            .ConfigureLogging(logging =>
            {
                logging.ClearProviders();
                logging.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Trace);
            })
            .UseNLog();  // NLog: Setup NLog for Dependency injection
    }
}

The nlog.config target and rules can be configured to log to Elasticsearch. NLog.Targets.ElasticSearch is an extension and needs to be added using the extensions tag.

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true"
      internalLogLevel="Warn"
      internalLogFile="C:\git\damienbod\AspNetCoreNlog\Logs\internal-nlog.txt">
    
    <extensions>
        <add assembly="NLog.Targets.ElasticSearch"/>
    </extensions>
            
  <targets>

    <target name="ElasticSearch" xsi:type="BufferingWrapper" flushTimeout="5000">
      <target xsi:type="ElasticSearch"/>
    </target>
   
  </targets>

  <rules>
    <logger name="*" minlevel="Trace" writeTo="ElasticSearch" />
      
  </rules>
</nlog>

The NLog.Targets.ElasticSearch package Elasticsearch URL can be configured using the ElasticsearchUrl property. This can be defined in the appsettings configuration file.

{
    "Logging": {
        "IncludeScopes": false,
        "LogLevel": {
            "Default": "Debug",
            "System": "Information",
            "Microsoft": "Information"
        }
    },
    "ElasticsearchUrl": "http://localhost:9200"
}

NLog.Targets.ElasticSearch ( ReactiveMarkets )

When the application is started the logs are written to Elasticsearch. These logs can be viewed in Elasticsearch

http://localhost:9200/logstash-‘date’/_search

{
	"took": 2,
	"timed_out": false,
	"_shards": {
		"total": 5,
		"successful": 5,
		"failed": 0
	},
	"hits": {
		"total": 18,
		"max_score": 1.0,
		"hits": [{
			"_index": "logstash-2016.08.19",
			"_type": "logevent",
			"_id": "AVaiJHPycDWw4BKmTWqP",
			"_score": 1.0,
			"_source": {
				"@timestamp": "2016-08-19T09:31:44.5790894Z",
				"level": "Debug",
				"message": "2016-08-19 11:31:44.5790|DEBUG|Microsoft.AspNetCore.Hosting.Internal.WebHost|Hosting starting"
			}
		},
		{
			"_index": "logstash-2016.08.19",
			"_type": "logevent",
			"_id": "AVaiJHPycDWw4BKmTWqU",
			"_score": 1.0,
			"_source": {
				"@timestamp": "2016-08-19T09:31:45.4788003Z",
				"level": "Info",
				"message": "2016-08-19 11:31:45.4788|INFO|Microsoft.AspNetCore.Hosting.Internal.WebHost|Request starting HTTP/1.1 DEBUG http://localhost:55423/  0"
			}
		},
		{
			"_index": "logstash-2016.08.19",
			"_type": "logevent",
			"_id": "AVaiJHPycDWw4BKmTWqW",
			"_score": 1.0,
			"_source": {
				"@timestamp": "2016-08-19T09:31:45.6248512Z",
				"level": "Debug",
				"message": "2016-08-19 11:31:45.6248|DEBUG|Microsoft.AspNetCore.Server.Kestrel|Connection id \"0HKU82EHFC0S9\" completed keep alive response."
			}
		},

Links

https://github.com/NLog/NLog.Extensions.Logging

https://github.com/ReactiveMarkets/NLog.Targets.ElasticSearch

https://github.com/NLog

https://docs.asp.net/en/latest/fundamentals/logging.html

https://msdn.microsoft.com/en-us/magazine/mt694089.aspx

https://github.com/nlog/NLog/wiki/Database-target

https://www.elastic.co/products/elasticsearch

https://github.com/elastic/logstash

https://github.com/elastic/elasticsearch-net

https://www.nuget.org/packages/Elasticsearch.Net/

https://github.com/nlog/NLog/wiki/File-target#size-based-file-archival