UPDATE: This NuGet package will no longer be supported as SLAB now supports an Elasticsearch sink. This blog still demonstates how to create a custom sink for SLAB. With the version 1.1 bulk inserts can also be used for custom sinks.
This post demonstrates how to create a Semantic.Logging Sink which adds documents to the Elasticsearch database or search engine. The code can be used directly or it can be downloaded from NuGet: Slab.Elasticsearch
Source Code: https://github.com/damienbod/Slab.Elasticsearch
NuGet: http://www.nuget.org/packages/Slab.Elasticsearch
Help on how to setup the Elasticsearch can be found here: Getting started with Elasticsearch and .NET
Using the NuGet Package:
Download the Slab.Elasticsearch NuGet package: Slab.Elasticsearch
Use as required. The Elasticsearch index and type can be specified for each ObservableEventListener.
Property configurations LogToElasticsearchSink method:
- Connectionstring for your elasticsearch database. Example:”Server=localhost;Index=log;Port=9200″
- Elasticsearch index. example: “slab” , use only lowercase.
- Elasticsearch type. Example: “myEventType”
private static void InProcessLogging()
{
ObservableEventListener listener = new ObservableEventListener();
listener.EnableEvents(TestEvents.Log, EventLevel.LogAlways, Keywords.All);
listener.LogToConsole();
listener.LogToElasticsearchSink("Server=localhost;Index=log;Port=9200", "slab", "SLABEvent");
TestEvents.Log.Critical("Hello world In-Process Critical");
TestEvents.Log.Error("Hello world In-Process Error");
TestEvents.Log.Informational("Hello world In-Process Informational");
}
You will required an EventSource to test:
using System.Diagnostics.Tracing;
namespace Slab.Elasticsearch.Console
{
[EventSource(Name = "TestEvents")]
public class TestEvents : EventSource
{
public static readonly TestEvents Log = new TestEvents();
[Event(1, Message = "TestEvents Critical: {0}", Level = EventLevel.Critical)]
public void Critical(string message)
{
if (IsEnabled()) WriteEvent(1, message);
}
[Event(2, Message = "TestEvents Error {0}", Level = EventLevel.Error)]
public void Error(string message)
{
if (IsEnabled()) WriteEvent(2, message);
}
[Event(3, Message = "TestEvents Informational {0}", Level = EventLevel.Informational)]
public void Informational(string message)
{
if (IsEnabled()) WriteEvent(3, message);
}
[Event(4, Message = "TestEvents LogAlways {0}", Level = EventLevel.LogAlways)]
public void LogAlways(string message)
{
if (IsEnabled()) WriteEvent(4, message);
}
[Event(5, Message = "TestEvents Verbose {0}", Level = EventLevel.Verbose)]
public void Verbose(string message)
{
if (IsEnabled()) WriteEvent(5, message);
}
[Event(6, Message = "TestEvents Warning {0}", Level = EventLevel.Warning)]
public void Warning(string message)
{
if (IsEnabled()) WriteEvent(6, message);
}
}
}
Semantic.Logging is now ready to for in-process use.
Using the Slab.Elasticsearch sink in the Out-Of-Process service
Add the following configuration to the SemanticLogging-svc.xml in your out-of-process service.
<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns="http://schemas.microsoft.com/practices/2013/entlib/semanticlogging/etw"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://schemas.microsoft.com/practices/2013/entlib/semanticlogging/etw SemanticLogging-svc.xsd">
<!-- Optional settings for fine tuning performance and Trace Event Session identification-->
<traceEventService/>
<!-- Sinks reference definitons used by this host to listen ETW events -->
<sinks>
<customSink name="MyElasticsearchSink" type ="Slab.Elasticsearch.ElasticsearchSink, Slab.Elasticsearch">
<sources>
<eventSource name="TestEvents" level="LogAlways" />
</sources>
<parameters>
<parameter name="connectionString" type="System.String" value="Server=localhost;Index=log;Port=9200" />
<parameter name="searchIndex" type="System.String" value="slab" />
<parameter name="searchType" type="System.String" value="slabOutOfProcess" />
</parameters>
</customSink>
</sinks>
</configuration>
Copy the following assemblies to the Out-Of-Process service bin:
- Slab.Elasticsearch.dll
- Newtonsoft.Json.dll
- Nest.dll
Start the Out-Of-Process service and everything runs. All Processes on the same operating system can log now.
private static void OutOfProcessLogging()
{
TestEvents.Log.Critical("Hello world Out-Of-Process Critical");
TestEvents.Log.Error("Hello world Out-Of-Process Error");
TestEvents.Log.Informational("Hello world Out-Of-Process Informational");
}
The ObservableEventListener is not required for out-of-process logging
Creating a Custom Sink
To create a custom Sink in Semantic.Logging, the sink class needs to inherit from IObserver with type EventEntry. This example uses the following NuGet packages:
<?xml version="1.0" encoding="utf-8"?> <packages> <package id="EnterpriseLibrary.SemanticLogging" version="1.0.1304.0" targetFramework="net45" /> <package id="NEST.Signed" version="0.12.0.0" targetFramework="net45" /> <package id="Newtonsoft.Json" version="4.5.11" targetFramework="net45" /> </packages>
The required configuration properties for the custom sink are added in the constructor of the class. For Elasticsearch, the database connection string, search index and search type is required. The CreateLogEvent method maps the Slab log event to the database. You can add/remove anything here as the search engine has no fixed schema.
using System;
using System.Dynamic;
using System.IO;
using Microsoft.Practices.EnterpriseLibrary.SemanticLogging;
using Microsoft.Practices.EnterpriseLibrary.SemanticLogging.Formatters;
using Nest;
namespace Slab.Elasticsearch
{
public sealed class ElasticsearchSink : IObserver<EventEntry>
{
private readonly IEventTextFormatter _formatter;
private ElasticClient _client;
private readonly string _searchIndex = "slab";
private readonly string _searchType = "SLABEvent";
private readonly string _connectionString = "Server=localhost;Index=log;Port=9200";
public ElasticsearchSink(string connectionString, string searchIndex, string searchType, IEventTextFormatter formatter)
{
if (!string.IsNullOrEmpty(connectionString)) _connectionString = connectionString;
if (!string.IsNullOrEmpty(searchIndex)) _searchIndex = searchIndex.ToLower();
if (!string.IsNullOrEmpty(searchType)) _searchType = searchType.ToLower();
_formatter = formatter ?? new EventTextFormatter();
}
public ElasticsearchSink(string connectionString, string searchIndex, string searchType)
{
if (!string.IsNullOrEmpty(connectionString)) _connectionString = connectionString;
if (!string.IsNullOrEmpty(searchIndex)) _searchIndex = searchIndex.ToLower();
if (!string.IsNullOrEmpty(searchType)) _searchType = searchType.ToLower();
_formatter = new EventTextFormatter();
}
public void OnNext(EventEntry entry)
{
if (entry != null)
{
using (var writer = new StringWriter())
{
_formatter.WriteEvent(entry, writer);
Post(entry, writer.ToString());
}
}
}
private void Post(EventEntry loggingEvent, string body)
{
if (string.IsNullOrEmpty(_connectionString))
{
return;
}
var settings = ConnectionBuilder.BuildElsticSearchConnection(_connectionString);
_client = new ElasticClient(settings);
var logEvent = CreateLogEvent(loggingEvent, body);
try
{
_client.IndexAsync(logEvent, _searchIndex, _searchType);
}
catch (InvalidOperationException)
{
}
}
private static dynamic CreateLogEvent(EventEntry loggingEvent, string formattedBody)
{
if (loggingEvent == null)
{
throw new ArgumentNullException("loggingEvent");
}
dynamic logEvent = new ExpandoObject();
logEvent.Id = new UniqueIdGenerator().GenerateUniqueId();
logEvent.EventName = loggingEvent.Schema.EventName;
logEvent.Keywords = loggingEvent.Schema.Keywords.ToString();
logEvent.Level = loggingEvent.Schema.Level;
logEvent.Opcode = loggingEvent.Schema.Opcode;
logEvent.OpcodeName = loggingEvent.Schema.OpcodeName;
logEvent.Payload = loggingEvent.Payload;
logEvent.ProviderId = loggingEvent.Schema.ProviderId;
logEvent.ProviderName = loggingEvent.Schema.ProviderName;
logEvent.Task = loggingEvent.Schema.Task;
logEvent.TaskName = loggingEvent.Schema.TaskName;
logEvent.Version = loggingEvent.Schema.Version;
logEvent.EventId = loggingEvent.EventId;
logEvent.FormattedMessage = formattedBody;
logEvent.Timestamp = loggingEvent.Timestamp;
return logEvent;
}
public void OnCompleted()
{
}
public void OnError(Exception error)
{
}
}
}
The ElasticsearchSinkExtensions class is added to make it easy to use.
using System;
using Microsoft.Practices.EnterpriseLibrary.SemanticLogging;
using Microsoft.Practices.EnterpriseLibrary.SemanticLogging.Formatters;
namespace Slab.Elasticsearch
{
public static class ElasticsearchSinkExtensions
{
public static SinkSubscription<ElasticsearchSink> LogToElasticsearchSink(
this IObservable<EventEntry> eventStream, string connectionString, string searchIndex, string searchType, IEventTextFormatter formatter = null)
{
var sink = new ElasticsearchSink(connectionString, searchIndex, searchType, formatter);
var subscription = eventStream.Subscribe(sink);
return new SinkSubscription<ElasticsearchSink>(subscription, sink);
}
}
}
The console application demonstates how to use the Sinks.
using System.Diagnostics.Tracing;
using Microsoft.Practices.EnterpriseLibrary.SemanticLogging;
namespace Slab.Elasticsearch.Console
{
class Program
{
static void Main(string[] args)
{
InProcessLogging();
System.Console.ReadLine();
}
private static void InProcessLogging()
{
ObservableEventListener listener = new ObservableEventListener();
listener.EnableEvents(TestEvents.Log, EventLevel.LogAlways, Keywords.All);
listener.LogToConsole();
listener.LogToElasticsearchSink("Server=localhost;Index=log;Port=9200", "slab", "SLABEvent");
TestEvents.Log.Critical("Hello world In-Process Critical");
TestEvents.Log.Error("Hello world In-Process Error");
TestEvents.Log.Informational("Hello world In-Process Informational");
}
}
}
The logs results can then be viewed in the Elasticsearch HQ:
TODO in next versions of the NuGet package:
- Bulk inserts or batch
- Add Event Logging when the Elasticsearch server is not available
UPDATE:
The slab team are implement this in the core.
http://slab.codeplex.com/SourceControl/network/forks/trentmswanson/esdev
When this is implemented, buffered logging will be supported to Elasticsearch.
Links:
http://msdn.microsoft.com/en-us/library/dn440729%28v=pandp.60%29.aspx
http://nest.azurewebsites.net/
https://github.com/jptoto/log4net.ElasticSearch
http://joelabrahamsson.com/elasticsearch-101/
http://exploringelasticsearch.com/book/an-overview/what-is-elasticsearch.html
Software Engineering 