This article shows how an audit trail can be implemented in ASP.NET Core which saves the audit documents to Elasticsearch using NEST.
Code: https://github.com/damienbod/AspNetCoreElasticsearchNestAuditTrail
History
2020-01-12 Updated to .NET Core 3.1
2019-02-15 Updated to .NET Core 2.2
Should I just use a logger?
Depends. If you just need to save requests, responses and application events, then a logger would be a better solution for this use case. I would use NLog as it provides everything you need, or could need, when working with ASP.NET Core.
If you only need to save business events/data of the application in the audit trail, then this solution could fit.
Using the Audit Trail
The audit trail is implemented so that it can be used easily. In the Startup class of the ASP.NET Core application, it is added to the application in the ConfigureServices method. The class library provides an extension method, AddAuditTrail, which can be configured as required. It takes 2 parameters, a bool parameter which defines if a new index is created per day or per month to save the audit trail documents, and a second int parameter which defines how many of the previous indices are included in the alias used to select the audit trail items. If this is 0, all indices are included for the search.
Because the audit trail documents are grouped into different indices per day or per month, the amount of documents can be controlled in each index. Usually the application user requires only the last n days, or last 2 months of the audit trails, and so the search does not need to search through all audit trails documents since the application began. This makes it possible to optimize the data as required, or even remove, archive old unused audit trail indices.
public void ConfigureServices(IServiceCollection services)
{
var indexPerMonth = false;
var amountOfPreviousIndicesUsedInAlias = 3;
services.AddAuditTrail<CustomAuditTrailLog>(options =>
options.UseSettings(indexPerMonth, amountOfPreviousIndicesUsedInAlias)
);
services.AddControllersWithViews()
.AddNewtonsoftJson();
}
The AddAuditTrail extension method requires a model definition which will be used to save or retrieve the documents in Elasticsearch. The model must implement the IAuditTrailLog interface. This interface just forces you to implement the property Timestamp which is required for the audit logs.
The model can then be designed, defined as required. NEST attributes can be used for each of the properties in the model. Use the keyword attribute, if the text field should not be analyzed. If you must use enums, then save the string value and NOT the integer value to the persistent layer. If integer values are saved for the enums, then it cannot be used without the knowledge of what each integer value represents, making it dependent on the code.
using AuditTrail.Model;
using Nest;
using System;
namespace AspNetCoreElasticsearchNestAuditTrail
{
public class CustomAuditTrailLog : IAuditTrailLog
{
public CustomAuditTrailLog()
{
Timestamp = DateTime.UtcNow;
}
public DateTime Timestamp { get; set; }
[Keyword]
public string Action { get; set; }
public string Log { get; set; }
public string Origin { get; set; }
public string User { get; set; }
public string Extra { get; set; }
}
}
The audit trail can then be used anywhere in the application. The IAuditTrailProvider can be added in the constructor of the class and an audit document can be created using the AddLog method.
private readonly IAuditTrailProvider<CustomAuditTrailLog> _auditTrailProvider;
public HomeController(IAuditTrailProvider<CustomAuditTrailLog> auditTrailProvider)
{
_auditTrailProvider = auditTrailProvider;
}
public IActionResult Index()
{
var auditTrailLog = new CustomAuditTrailLog()
{
User = User.ToString(),
Origin = "HomeController:Index",
Action = "Home GET",
Log = "home page called doing something important enough to be added to the audit log.",
Extra = "yep"
};
_auditTrailProvider.AddLog(auditTrailLog);
return View();
}
The audit trail documents can be viewed using QueryAuditLogs which supports paging and uses a simple query search which accepts wildcards. The AuditTrailSearch method returns a MVC view with the audit trail items in the model.
public IActionResult AuditTrailSearch(string searchString, int skip, int amount)
{
var auditTrailViewModel = new AuditTrailViewModel
{
Filter = searchString,
Skip = skip,
Size = amount
};
if (skip > 0 || amount > 0)
{
var paging = new AuditTrailPaging
{
Size = amount,
Skip = skip
};
auditTrailViewModel.AuditTrailLogs = _auditTrailProvider.QueryAuditLogs(searchString, paging).ToList();
return View(auditTrailViewModel);
}
auditTrailViewModel.AuditTrailLogs = _auditTrailProvider.QueryAuditLogs(searchString).ToList();
return View(auditTrailViewModel);
}
How is the Audit Trail implemented?
The AuditTrailExtensions class implements the extension methods used to initialize the audit trail implementations. This class accepts the options and registers the interfaces, classes with the IoC used by ASP.NET Core.
Generics are used so that any model class can be used to save the audit trail data. This changes always with each project, application. The type T must implement the interface IAuditTrailLog.
using System;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Localization;
using AuditTrail;
using AuditTrail.Model;
namespace Microsoft.Extensions.DependencyInjection
{
public static class AuditTrailExtensions
{
public static IServiceCollection AddAuditTrail<T>(this IServiceCollection services) where T : class, IAuditTrailLog
{
if (services == null)
{
throw new ArgumentNullException(nameof(services));
}
return AddAuditTrail<T>(services, setupAction: null);
}
public static IServiceCollection AddAuditTrail<T>(
this IServiceCollection services,
Action<AuditTrailOptions> setupAction) where T : class, IAuditTrailLog
{
if (services == null)
{
throw new ArgumentNullException(nameof(services));
}
services.TryAdd(new ServiceDescriptor(
typeof(IAuditTrailProvider<T>),
typeof(AuditTrailProvider<T>),
ServiceLifetime.Transient));
if (setupAction != null)
{
services.Configure(setupAction);
}
return services;
}
}
}
When a new audit trail log is added, it uses the index defined in the _indexName field.
public void AddLog(T auditTrailLog)
{
var indexRequest = new IndexRequest<T>(auditTrailLog);
var response = _elasticClient.Index(indexRequest);
if (!response.IsValid)
{
throw new ElasticsearchClientException("Add auditlog disaster!");
}
}
The _indexName field is defined using the date pattern, either days or months depending on your options.
private const string _alias = "auditlog";
private string _indexName = $"{_alias}-{DateTime.UtcNow.ToString("yyyy-MM-dd")}";
index definition per month:
if(_options.Value.IndexPerMonth)
{
_indexName = $"{_alias}-{DateTime.UtcNow.ToString("yyyy-MM")}";
}
When quering the audit trail logs, a simple query search query is used to find, select the audit trial documents required for the view. This is used so that wildcards can be used. The method accepts a query filter and paging options. If you search without any filter, all documents are returned which are defined in the alias (used indices). By using the simple query, the filter can accept options like AND, OR for the search.
public IEnumerable<T> QueryAuditLogs(string filter = "*", AuditTrailPaging auditTrailPaging = null)
{
var from = 0;
var size = 10;
EnsureAlias();
if(auditTrailPaging != null)
{
from = auditTrailPaging.Skip;
size = auditTrailPaging.Size;
if(size > 1000)
{
// max limit 1000 items
size = 1000;
}
}
var searchRequest = new SearchRequest<T>(Indices.Parse(_alias))
{
Size = size,
From = from,
Query = new QueryContainer(
new SimpleQueryStringQuery
{
Query = filter
}
),
Sort = new List<ISort>
{
new FieldSort { Field = TimestampField, Order = SortOrder.Descending }
}
};
var searchResponse = _elasticClient.Search<T>(searchRequest);
return searchResponse.Documents;
}
The alias is also updated in the search query, if required. Depending on you configuration, the alias uses all the audit trail indices or just the last n days, or n months. This check uses a static field. If the alias needs to be updated, the new alias is created, which also deletes the old one.
private void EnsureAlias()
{
if (_options.Value.IndexPerMonth)
{
if (aliasUpdated.Date < DateTime.UtcNow.AddMonths(-1).Date)
{
aliasUpdated = DateTime.UtcNow;
CreateAlias();
}
}
else
{
if (aliasUpdated.Date < DateTime.UtcNow.AddDays(-1).Date)
{
aliasUpdated = DateTime.UtcNow;
CreateAlias();
}
}
}
Here’s how the alias is created for all indices of the audit trail.
private void CreateAliasForAllIndices()
{
var response = _elasticClient.Indices.AliasExists(new AliasExistsRequest(new Names(new List<string> { _alias })));
if (response.Exists)
{
_elasticClient.Indices.DeleteAlias(new DeleteAliasRequest(Indices.Parse($"{_alias}-*"), _alias));
}
var responseCreateIndex = _elasticClient.Indices.PutAlias(new PutAliasRequest(Indices.Parse($"{_alias}-*"), _alias));
if (!responseCreateIndex.IsValid)
{
throw response.OriginalException;
}
}
The full AuditTrailProvider class which implements the audit trail.
using AuditTrail.Model;
using Elasticsearch.Net;
using Microsoft.Extensions.Options;
using Nest;
using Newtonsoft.Json.Converters;
using System;
using System.Collections.Generic;
using System.Linq;
namespace AuditTrail
{
public class AuditTrailProvider<T> : IAuditTrailProvider<T> where T : class
{
private const string _alias = "auditlog";
private string _indexName = $"{_alias}-{DateTime.UtcNow.ToString("yyyy-MM-dd")}";
private static Field TimestampField = new Field("timestamp");
private readonly IOptions<AuditTrailOptions> _options;
private ElasticClient _elasticClient { get; }
public AuditTrailProvider(
IOptions<AuditTrailOptions> auditTrailOptions)
{
_options = auditTrailOptions ?? throw new ArgumentNullException(nameof(auditTrailOptions));
if(_options.Value.IndexPerMonth)
{
_indexName = $"{_alias}-{DateTime.UtcNow.ToString("yyyy-MM")}";
}
var pool = new StaticConnectionPool(new List<Uri> { new Uri("http://localhost:9200") });
var connectionSettings = new ConnectionSettings(pool)
.DefaultMappingFor<T>(m => m
.IndexName(_indexName));
//new HttpConnection(),
//new SerializerFactory((jsonSettings, nestSettings) => jsonSettings.Converters.Add(new StringEnumConverter())))
//.DisableDirectStreaming();
_elasticClient = new ElasticClient(connectionSettings);
}
public void AddLog(T auditTrailLog)
{
var indexRequest = new IndexRequest<T>(auditTrailLog);
var response = _elasticClient.Index(indexRequest);
if (!response.IsValid)
{
throw new ElasticsearchClientException("Add auditlog disaster!");
}
}
public long Count(string filter = "*")
{
EnsureAlias();
var searchRequest = new SearchRequest<T>(Indices.Parse(_alias))
{
Size = 0,
Query = new QueryContainer(
new SimpleQueryStringQuery
{
Query = filter
}
),
Sort = new List<ISort>
{
new FieldSort { Field = TimestampField, Order = SortOrder.Descending }
}
};
var searchResponse = _elasticClient.Search<AuditTrailLog>(searchRequest);
return searchResponse.Total;
}
public IEnumerable<T> QueryAuditLogs(string filter = "*", AuditTrailPaging auditTrailPaging = null)
{
var from = 0;
var size = 10;
EnsureAlias();
if(auditTrailPaging != null)
{
from = auditTrailPaging.Skip;
size = auditTrailPaging.Size;
if(size > 1000)
{
// max limit 1000 items
size = 1000;
}
}
var searchRequest = new SearchRequest<T>(Indices.Parse(_alias))
{
Size = size,
From = from,
Query = new QueryContainer(
new SimpleQueryStringQuery
{
Query = filter
}
),
Sort = new List<ISort>
{
new FieldSort { Field = TimestampField, Order = SortOrder.Descending }
}
};
var searchResponse = _elasticClient.Search<T>(searchRequest);
return searchResponse.Documents;
}
private void CreateAliasForAllIndices()
{
var response = _elasticClient.Indices.AliasExists(new AliasExistsRequest(new Names(new List<string> { _alias })));
//if (!response.IsValid)
//{
// throw response.OriginalException;
//}
if (response.Exists)
{
_elasticClient.Indices.DeleteAlias(new DeleteAliasRequest(Indices.Parse($"{_alias}-*"), _alias));
}
var responseCreateIndex = _elasticClient.Indices.PutAlias(new PutAliasRequest(Indices.Parse($"{_alias}-*"), _alias));
if (!responseCreateIndex.IsValid)
{
throw response.OriginalException;
}
}
private void CreateAlias()
{
if (_options.Value.AmountOfPreviousIndicesUsedInAlias > 0)
{
CreateAliasForLastNIndices(_options.Value.AmountOfPreviousIndicesUsedInAlias);
}
else
{
CreateAliasForAllIndices();
}
}
private void CreateAliasForLastNIndices(int amount)
{
var responseCatIndices = _elasticClient.Cat.Indices(new CatIndicesRequest(Indices.Parse($"{_alias}-*")));
var records = responseCatIndices.Records.ToList();
List<string> indicesToAddToAlias = new List<string>();
for(int i = amount;i>0;i--)
{
if (_options.Value.IndexPerMonth)
{
var indexName = $"{_alias}-{DateTime.UtcNow.AddMonths(-i + 1).ToString("yyyy-MM")}";
if(records.Exists(t => t.Index == indexName))
{
indicesToAddToAlias.Add(indexName);
}
}
else
{
var indexName = $"{_alias}-{DateTime.UtcNow.AddDays(-i + 1).ToString("yyyy-MM-dd")}";
if (records.Exists(t => t.Index == indexName))
{
indicesToAddToAlias.Add(indexName);
}
}
}
var response = _elasticClient.Indices.AliasExists(new AliasExistsRequest(new Names(new List<string> { _alias })));
//if (!response.IsValid)
//{
// throw response.OriginalException;
//}
if (response.Exists)
{
_elasticClient.Indices.DeleteAlias(new DeleteAliasRequest(Indices.Parse($"{_alias}-*"), _alias));
}
Indices multipleIndicesFromStringArray = indicesToAddToAlias.ToArray();
var responseCreateIndex = _elasticClient.Indices.PutAlias(new PutAliasRequest(multipleIndicesFromStringArray, _alias));
if (!responseCreateIndex.IsValid)
{
throw responseCreateIndex.OriginalException;
}
}
private static DateTime aliasUpdated = DateTime.UtcNow.AddYears(-50);
private void EnsureAlias()
{
if (_options.Value.IndexPerMonth)
{
if (aliasUpdated.Date < DateTime.UtcNow.AddMonths(-1).Date)
{
aliasUpdated = DateTime.UtcNow;
CreateAlias();
}
}
else
{
if (aliasUpdated.Date < DateTime.UtcNow.AddDays(-1).Date)
{
aliasUpdated = DateTime.UtcNow;
CreateAlias();
}
}
}
}
}
Testing the audit log
The created audit trails can be checked using the following HTTP GET requests:
Counts all the audit trail entries in the alias.
http://localhost:9200/auditlog/_count
Shows all the audit trail indices. You can count all the documents from the indices used in the alias and it must match the count from the alias.
http://localhost:9200/_cat/indices/auditlog*
You can also start the application and the AuditTrail logs can be displayed in the Audit Trail logs MVC view.
This view is just a quick test, if implementing properly, you would have to localize the timestamp display and add proper paging in the view.
Notes, improvements
If lots of audit trail documents are written at once, maybe a bulk insert could be used to add the documents in batches, like most of the loggers implement this. You should also define a strategy on how the old audit trails, indices should be cleaned up, archived or whatever. The creating of the alias could be optimized depending on you audit trail data, and how you clean up old audit trail indices.
Links:
https://www.elastic.co/guide/en/elasticsearch/reference/5.2/indices-aliases.html
https://docs.microsoft.com/en-us/aspnet/core/
https://www.elastic.co/products/elasticsearch
Software Engineering 
Reblogged this on Kossi Selom Banybah.
[…] Implementing an Audit Trail using ASP.NET Core and Elasticsearch with NEST […]
[…] Implementing an Audit Trail using ASP.NET Core and Elasticsearch with NEST by Damien Bod. […]