This article shows how an audit trail can be implemented in ASP.NET Core which saves the audit documents to Elasticsearch using the Elastic.Clients.Elasticsearch Nuget package.
Code: https://github.com/damienbod/AspNetCoreElasticsearchAuditTrail
History
- 2024-09-11 Updated to .NET 8 and Elastic.Clients.Elasticsearch
- 2020-01-12 Updated to .NET Core 3.1
- 2019-02-15 Updated to .NET Core 2.2
Should I just use a logger?
Depends. If you just need to save requests, responses and application events, then a logger would be a better solution for this use case. I would use Serilog as it provides everything you need, or could need, when working with ASP.NET Core.
If you only need to save business events/data of the application in the audit trail, then this solution could fit.
Using the Audit Trail
The audit trail is implemented so that it can be used easily. In the program file of the ASP.NET Core application, it is added to the application using the builder. The class library provides an extension method, AddAuditTrail, which can be configured as required. It takes 5 parameters, a bool parameter which defines if a new index is created per day or per month to save the audit trail documents, and a second int parameter which defines how many of the previous indices are included in the alias used to select the audit trail items. If this is 0, all indices are included for the search.
Because the audit trail documents are grouped into different indices per day or per month, the amount of documents can be controlled in each index. Usually the application user requires only the last n days, or last 2 months of the audit trails, and so the search does not need to search through all audit trails documents since the application began. This makes it possible to optimize the data as required, or even remove, archive old unused audit trail indices.
var indexPerMonth = false;
var amountOfPreviousIndicesUsedInAlias = 3;
builder.Services.AddAuditTrail<CustomAuditTrailLog>(options =>
options.UseSettings(indexPerMonth,
amountOfPreviousIndicesUsedInAlias,
builder.Configuration["ElasticsearchUserName"],
builder.Configuration["ElasearchPassword"],
builder.Configuration["ElasearchUrl"])
);
builder.Services.AddControllersWithViews();
The AddAuditTrail extension method requires a model definition which will be used to save or retrieve the documents in Elasticsearch. The model must implement the IAuditTrailLog interface. This interface just forces you to implement the property Timestamp which is required for the audit logs.
The model can then be designed, defined as required. Use the keyword attribute, if the text field should not be analyzed. If you must use enums, then save the string value and NOT the integer value to the persistent layer. If integer values are saved for the enums, then it cannot be used without the knowledge of what each integer value represents, making it dependent on the code.
public class CustomAuditTrailLog : IAuditTrailLog
{
public CustomAuditTrailLog()
{
Timestamp = DateTime.UtcNow;
}
public DateTime Timestamp { get; set; }
public string Action { get; set; } = string.Empty;
public string Log { get; set; } = string.Empty;
public string Origin { get; set; } = string.Empty;
public string User { get; set; } = string.Empty;
public string Extra { get; set; } = string.Empty;
}
The audit trail can then be used anywhere in the application. The IAuditTrailProvider can be added in the constructor of the class and an audit document can be created using the AddLog method.
public class HomeController(IAuditTrailProvider<CustomAuditTrailLog> auditTrailProvider) : Controller
{
private readonly IAuditTrailProvider<CustomAuditTrailLog> _auditTrailProvider = auditTrailProvider;
public IActionResult Index()
{
var auditTrailLog = new CustomAuditTrailLog()
{
User = GetUserName(),
Origin = "HomeController:Index",
Action = "Home GET",
Log = "home page called doing something important enough to be added to the audit log.",
Extra = "yep"
};
_auditTrailProvider.AddLog(auditTrailLog);
return View();
}
private string GetUserName()
{
return User.Identity!.Name ?? "Anonymous";
}
The audit trail documents can be viewed using QueryAuditLogs which supports paging and uses a simple query search which accepts wildcards. The AuditTrailSearch method returns a MVC view with the audit trail items in the model.
public async Task<IActionResult> AuditTrailSearchAsync(string searchString, int skip, int amount)
{
var auditTrailLog = new CustomAuditTrailLog()
{
User = GetUserName(),
Origin = "HomeController:AuditTrailSearchAsync",
Action = "AuditTrailSearchAsync GET",
Log = $"user clicked the audit trail nav. {searchString}"
};
await _auditTrailProvider.AddLog(auditTrailLog);
var auditTrailViewModel = new AuditTrailViewModel
{
Filter = searchString,
Skip = skip,
Size = amount
};
if (skip > 0 || amount > 0)
{
var paging = new AuditTrailPaging
{
Size = amount,
Skip = skip
};
auditTrailViewModel.AuditTrailLogs = (await _auditTrailProvider.QueryAuditLogs(searchString, paging)).ToList();
return View(auditTrailViewModel);
}
auditTrailViewModel.AuditTrailLogs = (await _auditTrailProvider.QueryAuditLogs(searchString)).ToList();
return View(auditTrailViewModel);
}
How is the Audit Trail implemented?
The AuditTrailExtensions class implements the extension methods used to initialize the audit trail implementations. This class accepts the options and registers the interfaces, classes with the IoC used by ASP.NET Core.
Generics are used so that any model class can be used to save the audit trail data. This changes always with each project, application. The type T must implement the interface IAuditTrailLog.
using AuditTrail;
using AuditTrail.Model;
using Microsoft.Extensions.DependencyInjection.Extensions;
using System;
namespace Microsoft.Extensions.DependencyInjection;
public static class AuditTrailExtensions
{
public static IServiceCollection AddAuditTrail<T>(this IServiceCollection services) where T : class, IAuditTrailLog
{
ArgumentNullException.ThrowIfNull(services);
return AddAuditTrail<T>(services, setupAction: null);
}
public static IServiceCollection AddAuditTrail<T>(
this IServiceCollection services,
Action<AuditTrailOptions> setupAction) where T : class, IAuditTrailLog
{
ArgumentNullException.ThrowIfNull(services);
services.TryAdd(new ServiceDescriptor(
typeof(IAuditTrailProvider<T>),
typeof(AuditTrailProvider<T>),
ServiceLifetime.Transient));
if (setupAction != null)
{
services.Configure(setupAction);
}
return services;
}
}
When a new audit trail log is added, it uses the index defined in the _indexName field.
public async Task AddLog(T auditTrailLog)
{
EnsureElasticClient(_indexName, _options.Value);
var indexRequest = new IndexRequest<T>(auditTrailLog);
var response = await _elasticsearchClient.IndexAsync(indexRequest);
if (!response.IsValidResponse)
{
throw new ArgumentException("Add auditlog disaster!");
}
}
The _indexName field is defined using the date pattern, either days or months depending on your options.
private const string _alias = "auditlog";
private readonly string _indexName = $"{_alias}-{DateTime.UtcNow:yyyy-MM-dd}";
private readonly IOptions<AuditTrailOptions> _options;
private readonly static Field TimestampField = new("timestamp");
private static ElasticsearchClient _elasticsearchClient;
private static string _actualIndex = string.Empty;
index definition per month:
if (_options.Value.IndexPerMonth)
{
_indexName = $"{_alias}-{DateTime.UtcNow:yyyy-MM}";
}
When quering the audit trail logs, a simple query search query is used to find, select the audit trial documents required for the view. This is used so that wildcards can be used. The method accepts a query filter and paging options. If you search without any filter, all documents are returned which are defined in the alias (used indices). By using the simple query, the filter can accept options like AND, OR for the search.
public async Task<IEnumerable<T>> QueryAuditLogs(string filter = "*", AuditTrailPaging auditTrailPaging = null)
{
var from = 0;
var size = 10;
EnsureElasticClient(_indexName, _options.Value);
await EnsureAlias();
if (auditTrailPaging != null)
{
from = auditTrailPaging.Skip;
size = auditTrailPaging.Size;
if (size > 1000)
{
// max limit 1000 items
size = 1000;
}
}
var searchRequest = new SearchRequest<T>(Indices.Parse(_alias))
{
Size = size,
From = from,
Query = new SimpleQueryStringQuery
{
Query = filter
},
Sort = BuildSort()
};
var searchResponse = await _elasticsearchClient.SearchAsync<T>(searchRequest);
return searchResponse.Documents;
}
The alias is also updated in the search query, if required. Depending on you configuration, the alias uses all the audit trail indices or just the last n days, or n months. This check uses a static field. If the alias needs to be updated, the new alias is created, which also deletes the old one.
private async Task EnsureAlias()
{
if (_options.Value.IndexPerMonth)
{
if (aliasUpdated.Date < DateTime.UtcNow.AddMonths(-1).Date)
{
aliasUpdated = DateTime.UtcNow;
await CreateAlias();
}
}
else if (aliasUpdated.Date < DateTime.UtcNow.AddDays(-1).Date)
{
aliasUpdated = DateTime.UtcNow;
await CreateAlias();
}
}
Here’s how the alias is created for all indices of the audit trail.
private async Task CreateAliasForLastNIndicesAsync(int amount)
{
EnsureElasticClient(_indexName, _options.Value);
var responseCatIndices = await _elasticsearchClient
.Indices.GetAsync(new GetIndexRequest(Indices.Parse($"{_alias}-*")));
var records = responseCatIndices.Indices.ToList();
var indicesToAddToAlias = new List<string>();
for (int i = amount; i > 0; i--)
{
if (_options.Value.IndexPerMonth)
{
var indexName = $"{_alias}-{DateTime.UtcNow.AddMonths(-i + 1):yyyy-MM}";
if (records.Exists(t => t.Key == indexName))
{
indicesToAddToAlias.Add(indexName);
}
}
else
{
var indexName = $"{_alias}-{DateTime.UtcNow.AddDays(-i + 1):yyyy-MM-dd}";
if (records.Exists(t => t.Key == indexName))
{
indicesToAddToAlias.Add(indexName);
}
}
}
var response = await _elasticsearchClient.Indices
.ExistsAliasAsync(new ExistsAliasRequest(new Names(new List<string> { _alias })));
if (response.Exists)
{
await _elasticsearchClient.Indices
.DeleteAliasAsync(new DeleteAliasRequest(Indices.Parse($"{_alias}-*"), _alias));
}
Indices multipleIndicesFromStringArray = indicesToAddToAlias.ToArray();
var responseCreateIndex = await _elasticsearchClient.Indices
.PutAliasAsync(new PutAliasRequest(multipleIndicesFromStringArray, _alias));
if (!responseCreateIndex.IsValidResponse)
{
var res = responseCreateIndex.TryGetOriginalException(out var ex);
throw ex;
}
}
The full AuditTrailProvider class which implements the audit trail can be found in the Github repository linked above.
Testing the audit log
The created audit trails can be checked using the following HTTP GET requests:
Counts all the audit trail entries in the alias.
https://localhost:9200/auditlog/_count
Shows all the audit trail indices. You can count all the documents from the indices used in the alias and it must match the count from the alias.
https://localhost:9200/_cat/indices/auditlog*
You can also start the application and the AuditTrail logs can be displayed in the Audit Trail logs MVC view.
This view is just a quick test, if implementing properly, you would have to localize the timestamp display and add proper paging in the view.
Notes, improvements
If lots of audit trail documents are written at once, maybe a bulk insert could be used to add the documents in batches, like most of the loggers implement this. You should also define a strategy on how the old audit trails, indices should be cleaned up, archived or whatever. The creating of the alias could be optimized depending on you audit trail data, and how you clean up old audit trail indices.
Links:
https://www.elastic.co/guide/en/elasticsearch/reference/current/aliases.html
https://docs.microsoft.com/en-us/aspnet/core/
Software Engineering 
Reblogged this on Kossi Selom Banybah.
[…] Implementing an Audit Trail using ASP.NET Core and Elasticsearch with NEST […]
[…] Implementing an Audit Trail using ASP.NET Core and Elasticsearch with NEST by Damien Bod. […]