ASP.NET Core logging with NLog and Elasticsearch

This article shows how to Log to Elasticsearch using NLog in an ASP.NET Core application. NLog is a free open-source logging for .NET.

Code: VS2017 RC3 csproj | VS2015 project.json

2017.02.08 Updated to NLog.Web.AspNetCore 4.3.0 and VS2017 RC3
17.12.2016 Updated to ASP.NET Core 1.1

NLog posts in this series:

  1. ASP.NET Core logging with NLog and Microsoft SQL Server
  2. ASP.NET Core logging with NLog and Elasticsearch
  3. Settings the NLog database connection string in the ASP.NET Core appsettings.json
  4. .NET Core logging to MySQL using NLog
  5. .NET Core logging with NLog and PostgreSQL

NLog.Extensions.Logging is required to use NLog in an ASP.NET Core application. This is added to the dependencies of the project. NLog.Targets.ElasticSearch is also added to the dependencies. This project is at present NOT the NuGet package from ReactiveMarkets, but the source code from ReactiveMarkets and updated to dotnetcore. Thanks to ReactiveMarkets for this library, hopefully the NuGet package will be updated and the NuGet package can be used directly.

The NLog configuration file also needs to be added to the publishOptions in the csproj file.

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>netcoreapp1.1</TargetFramework>
    <PreserveCompilationContext>true</PreserveCompilationContext>
    <AssemblyName>AspNetCoreNlog</AssemblyName>
    <OutputType>Exe</OutputType>
    <PackageId>AspNetCoreNlog</PackageId>
    <PackageTargetFallback>$(PackageTargetFallback);dotnet5.6;portable-net45+win8</PackageTargetFallback>
  </PropertyGroup>
  <ItemGroup>
    <Content Update="wwwroot\**\*;Views;Areas\**\Views;appsettings.json;nlog.config;web.config">
      <CopyToPublishDirectory>PreserveNewest</CopyToPublishDirectory>
    </Content>
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\NLog.Targets.ElasticSearch\NLog.Targets.ElasticSearch.csproj" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.Diagnostics" Version="1.1.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc" Version="1.1.0" />
    <PackageReference Include="Microsoft.AspNetCore.Localization" Version="1.1.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Localization" Version="1.1.0" />
    <PackageReference Include="Microsoft.AspNetCore.Routing" Version="1.1.0" />
    <PackageReference Include="Microsoft.AspNetCore.Server.IISIntegration" Version="1.1.0" />
    <PackageReference Include="Microsoft.AspNetCore.Server.Kestrel" Version="1.1.0" />
    <PackageReference Include="Microsoft.AspNetCore.StaticFiles" Version="1.1.0" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="1.0.0-msbuild3-final" />
    <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="1.1.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="1.1.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" Version="1.1.0" />
    <PackageReference Include="Microsoft.Extensions.Logging" Version="1.1.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="1.1.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="1.1.0" />
    <PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="1.1.0" />
    <PackageReference Include="NLog.Web.AspNetCore" Version="4.3.0" />
    <PackageReference Include="System.Data.SqlClient" Version="4.3.0" />
  </ItemGroup>
  <ItemGroup>
    <DotNetCliToolReference Include="Microsoft.EntityFrameworkCore.Tools.DotNet" Version="1.0.0-msbuild3-final" />
    <DotNetCliToolReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Tools" Version="1.0.0-msbuild3-final" />
  </ItemGroup>
</Project>

The NLog configuration is added to the Startup.cs class in the Configure method.

public void ConfigureServices(IServiceCollection services)
{
	services.AddSingleton<IHttpContextAccessor, HttpContextAccessor>();
	// Add framework services.
	services.AddMvc();

	services.AddScoped<LogFilter>();
}

// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
	loggerFactory.AddNLog();

	//add NLog.Web
	app.AddNLogWeb();

	LogManager.Configuration.Variables["configDir"] = "C:\\git\\damienbod\\AspNetCoreNlog\\Logs";

	app.UseMvc();
}

The nlog.config target and rules can be configured to log to Elasticsearch. NLog.Targets.ElasticSearch is an extension and needs to be added using the extensions tag.

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true"
      internalLogLevel="Warn"
      internalLogFile="C:\git\damienbod\AspNetCoreNlog\Logs\internal-nlog.txt">
    
    <extensions>
        <add assembly="NLog.Targets.ElasticSearch"/>
    </extensions>
            
  <targets>

    <target name="ElasticSearch" xsi:type="BufferingWrapper" flushTimeout="5000">
      <target xsi:type="ElasticSearch"/>
    </target>
   
  </targets>

  <rules>
    <logger name="*" minlevel="Trace" writeTo="ElasticSearch" />
      
  </rules>
</nlog>

The NLog.Targets.ElasticSearch package Elasticsearch URL can be configured using the ElasticsearchUrl property. This can be defined in the appsettings configuration file.

{
    "Logging": {
        "IncludeScopes": false,
        "LogLevel": {
            "Default": "Debug",
            "System": "Information",
            "Microsoft": "Information"
        }
    },
    "ElasticsearchUrl": "http://localhost:9200"
}

NLog.Targets.ElasticSearch ( ReactiveMarkets )

The existing NLog.Targets.ElasticSearch project from ReactiveMarkets is updated to a NETStandard Library. This class library requires Elasticsearch.Net, NLog and Newtonsoft.Json. The dependencies are added to the csproj file. The library supports both netstandard1.6 and also net451.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netstandard1.6;net451</TargetFrameworks>
    <AssemblyName>NLog.Targets.ElasticSearch</AssemblyName>
    <PackageId>NLog.Targets.ElasticSearch</PackageId>
    <PackageTargetFallback Condition=" '$(TargetFramework)' == 'netstandard1.6' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
    <GenerateAssemblyConfigurationAttribute>false</GenerateAssemblyConfigurationAttribute>
    <GenerateAssemblyCompanyAttribute>false</GenerateAssemblyCompanyAttribute>
    <GenerateAssemblyProductAttribute>false</GenerateAssemblyProductAttribute>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="NLog" Version="5.0.0-beta05" />
    <PackageReference Include="Newtonsoft.Json" Version="9.0.1" />
    <PackageReference Include="Elasticsearch.Net" Version="2.5.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration" Version="1.1.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.FileExtensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="1.1.0" />
  </ItemGroup>
  <ItemGroup Condition=" '$(TargetFramework)' == 'net451' ">
    <Reference Include="System.Runtime.Serialization" />
    <Reference Include="System.Runtime" />
    <Reference Include="System" />
    <Reference Include="Microsoft.CSharp" />
  </ItemGroup>
</Project>

The StringExtensions class is extended to make it possible to define the Elasticsearch URL in a configuration file.
( original code from ReactiveMarkets )

using System;
using System.IO;
#if NET45
#else
using Microsoft.Extensions.Configuration;
#endif

namespace NLog.Targets.ElasticSearch
{
    internal static class StringExtensions
    {
        public static object ToSystemType(this string field, Type type)
        {
            switch (type.FullName)
            {
                case "System.Boolean":
                    return Convert.ToBoolean(field);
                case "System.Double":
                    return Convert.ToDouble(field);
                case "System.DateTime":
                    return Convert.ToDateTime(field);
                case "System.Int32":
                    return Convert.ToInt32(field);
                case "System.Int64":
                    return Convert.ToInt64(field);
                default:
                    return field;
            }
        }

        public static string GetConnectionString(this string name)
        {
            var value = GetEnvironmentVariable(name);
            if (!string.IsNullOrEmpty(value))
                return value;
#if NET45
            var connectionString = ConfigurationManager.ConnectionStrings[name];
            return connectionString?.ConnectionString;
#else
            IConfigurationRoot configuration;
            var builder = new Microsoft.Extensions.Configuration.ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true);

            configuration = builder.Build();
            return configuration["ElasticsearchUrl"];
#endif

        }

        private static string GetEnvironmentVariable(this string name)
        {
            return string.IsNullOrEmpty(name) ? null : Environment.GetEnvironmentVariable(name);
        }
    }
}

When the application is started the logs are written to Elasticsearch. These logs can be viewed in Elasticsearch

http://localhost:9200/logstash-‘date’/_search

{
	"took": 2,
	"timed_out": false,
	"_shards": {
		"total": 5,
		"successful": 5,
		"failed": 0
	},
	"hits": {
		"total": 18,
		"max_score": 1.0,
		"hits": [{
			"_index": "logstash-2016.08.19",
			"_type": "logevent",
			"_id": "AVaiJHPycDWw4BKmTWqP",
			"_score": 1.0,
			"_source": {
				"@timestamp": "2016-08-19T09:31:44.5790894Z",
				"level": "Debug",
				"message": "2016-08-19 11:31:44.5790|DEBUG|Microsoft.AspNetCore.Hosting.Internal.WebHost|Hosting starting"
			}
		},
		{
			"_index": "logstash-2016.08.19",
			"_type": "logevent",
			"_id": "AVaiJHPycDWw4BKmTWqU",
			"_score": 1.0,
			"_source": {
				"@timestamp": "2016-08-19T09:31:45.4788003Z",
				"level": "Info",
				"message": "2016-08-19 11:31:45.4788|INFO|Microsoft.AspNetCore.Hosting.Internal.WebHost|Request starting HTTP/1.1 DEBUG http://localhost:55423/  0"
			}
		},
		{
			"_index": "logstash-2016.08.19",
			"_type": "logevent",
			"_id": "AVaiJHPycDWw4BKmTWqW",
			"_score": 1.0,
			"_source": {
				"@timestamp": "2016-08-19T09:31:45.6248512Z",
				"level": "Debug",
				"message": "2016-08-19 11:31:45.6248|DEBUG|Microsoft.AspNetCore.Server.Kestrel|Connection id \"0HKU82EHFC0S9\" completed keep alive response."
			}
		},

Links

https://github.com/NLog/NLog.Extensions.Logging

https://github.com/ReactiveMarkets/NLog.Targets.ElasticSearch

https://github.com/NLog

https://docs.asp.net/en/latest/fundamentals/logging.html

https://msdn.microsoft.com/en-us/magazine/mt694089.aspx

https://github.com/nlog/NLog/wiki/Database-target

https://www.elastic.co/products/elasticsearch

https://github.com/elastic/logstash

https://github.com/elastic/elasticsearch-net

https://www.nuget.org/packages/Elasticsearch.Net/

https://github.com/nlog/NLog/wiki/File-target#size-based-file-archival

http://www.danesparza.net/2014/06/things-your-dad-never-told-you-about-nlog/

Advertisements

9 comments

  1. […] Asp.net Core Logging With Nlog And Elasticsearch | Software Engineering […]

  2. […] Produire les logs avec .NET Core et Elasticsearch. […]

  3. ASP.Net core 1.0 support Log4Net ?

  4. […] ASP.NET Core logging with NLog and Elasticsearch (Damien Bod) […]

  5. Sending logs over http seems poor to me. Is there a plan to log via UDP.

  6. Please, please, please log to file system and use filebeat (or any alternative) to send the logs to elasticsearch. More information about ELK stack:
    https://github.com/dzharii/awesome-elasticsearch

    1. Hi Dmytro, thanks for the comment. I never used filebeat, I’ll have a look a this. Thanks for the link

      Greetings Damien

  7. […] Asp.Net Core Logging with Nlog and Elasticsearch by Damien Bowden […]

  8. Thank you for the post, that’s cool, Can you also share the source code as an attachment?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: