This article demonstrates how to create a Web API RESTful service and use Elasticsearch as the persistence infrastructure. Nest is used to access Elasticsearch store. Simple CRUD is supported in the controller.

code: https://github.com/damienbod/WebAPIRestWithNest

Update 2014.09.28: updated all NuGet packages and fixed NEST, breaking changes in version 1

Structure
The Application structure is based on the onion pattern. The business layer contains the domain model and all the provider interfaces. The Web API service layer uses unity to create all instances. The business layer has no dependencies. Provider implementations have dependencies to the business layer. Due to this architecture, Elasticsearch could be replaced with MS SQL server or MongoDB. The application has just dependencies to the provider interface and one reference where unity registers the Elasticsearch provider. This is a simple pattern which can also be used for implementing portable libraries.

The Business Layer

This layer is the core of the application. In a large or professional application, the layer could be split. The important thing in this assembly is that it has no dependencies. The provider interfaces, domain model and the core classes are define here.

The SearchProvider interface provides the CRUD operations required for the domain model class Animal. In a later version, the search functionality will be added.

using System.Collections.Generic;
using Damienbod.BusinessLayer.DomainModel;

namespace Damienbod.BusinessLayer.Providers
{
    public interface ISearchProvider
    {
        void CreateAnimal(Animal animal);

        void UpdateAnimal(Animal animal);

        IEnumerable<Animal> GetAnimals();
        void DeleteById(int id);
        void DeleteIndex(string index);
    }
}

The animal manager class implements the business logic if required. For example, domain validation could be defined here. As this is just a demo application, the api events are sent to the ISearchProvider without any extra logic.

using System.Collections.Generic;
using Damienbod.BusinessLayer.Attributes;
using Damienbod.BusinessLayer.DomainModel;
using Damienbod.BusinessLayer.Providers;

namespace Damienbod.BusinessLayer.Managers
{
    [TransientLifetime]
    public class AnimalManager : IAnimalManager
    {
        private readonly ILogProvider _logProvider;
        private readonly ISearchProvider _searchProvider;

        public AnimalManager(ILogProvider logProvider,
                              ISearchProvider searchProvider)
        {
            _logProvider = logProvider;
            _searchProvider = searchProvider;
            _logProvider.BusinessLayerVerbose("created animal manager instance");
        }

        public IEnumerable<Animal> GetAnimals()
        {
            return _searchProvider.GetAnimals();
        }

        public Animal GetAnimal(int id)
        {
            return new Animal { AnimalType = "Dog", Id = 1 };
        }

        public void UpdateAnimal(Animal value)
        {
            _searchProvider.UpdateAnimal(value);
        }

        public void DeleteAnimal(int id)
        {
            _searchProvider.DeleteById(id);
        }

        public void CreateAnimal(Animal value)
        {
            _searchProvider.CreateAnimal(value);
        }

        public void DeleteIndex(string index)
        {
            _searchProvider.DeleteIndex(index);
        }
    }
}

The domain model is just a simple class for this demo.

using System;

namespace Damienbod.BusinessLayer.DomainModel
{
    public class Animal
    {
        public const string SearchIndex = "animals";

        public int Id { get; set; }

        public string AnimalType { get; set; }

        public string TypeSpecificForAnimalType { get; set; }

        public string Description { get; set; }

        public string Gender { get; set; }

        public string LastLocation { get; set; }

        public DateTime DateOfBirth { get; set; }

        public DateTime CreatedTimestamp { get; set; }

        public DateTime UpdatedTimestamp { get; set; }
    }
}

Elasticsearch Provider using Nest

This provider implements the ISearchProvider from the business layer. This could be replaced with a different persistence provider, for example one which uses SOLR or MS SQL Server, all depends on your needs.

Nest can be included in the project using NuGet.

The Index method from the Nest Api creates a new document in the search store or if a document already exists with the same Id, it will be updated. Due to this, the application prevents the update by throwing an exception if the document exists. The ValidateIfIdIsAlreadyUsedForIndex method uses the Search method to check if the document exists.

public void CreateAnimal(Animal animal)
{
  ValidateIfIdIsAlreadyUsedForIndex(animal.Id.ToString(CultureInfo.InvariantCulture));
  var index = _elasticsearchClient.Index(animal, i => i
			   .Index("animals")
			   .Type("animal")
			   .Id(animal.Id)
			   .Refresh()
			   .Ttl("1m")
  );
  _logProvider.ElasticSearchProviderVerbose(string.Format("Created animal: {0}, {1}", animal.Id, animal.AnimalType));
}

private void ValidateIfIdIsAlreadyUsedForIndex(string id)
{
 var idsList = new List<string> { id};
 var result = _elasticsearchClient.Search<Animal>(s => s
        .Index("animals")
        .AllTypes()
        .Query(p => p.Ids(idsList)));
 if (result.Documents.Any()) 
   throw new ArgumentException("Id already exists in store");
}

The update method will update a document in the store if the document exists, otherwise it will create a new one.

public void UpdateAnimal(Animal animal)
{

  var index = _elasticsearchClient.Index(animal, i => i
	.Index("Animal.SearchIndex")
	.Type("animal")
	.Id(animal.Id)
	.Refresh()
	.Ttl("1m")
  );

  _logProvider.ElasticSearchProviderVerbose(string.Format("Updated animal: {0}, {1}", animal.Id, animal.AnimalType));
}

The GetAnimals method returns all records from the store using the search method. This is not very practical if a lot of records exist. It is just used to test the application for this demo.

public IEnumerable<Animal> GetAnimals()
{
 var result = _elasticsearchClient.Search<Animal>(s => s
                    .Index("animals")
                    .AllTypes()
                    .MatchAll()
                );
 return result.Documents.ToList();
}

The DeleteById method deletes an animal type from the animals index which matches the id provided as a parameter.

public void DeleteById(int id)
{
 _logProvider.ElasticSearchProviderVerbose(
    string.Format("Sending DELETE animal type from animals index with id: {0}", id));
 _elasticsearchClient.DeleteById("animals", "animal", id);
}

The DeleteIndex method is used to delete a complete index in the store. This should be used with caution, usually you do not need to delete a whole index…

public void DeleteIndex(string index)
{ 
 _logProvider.ElasticSearchProviderWarning(
    string.Format("Sending DELETE index: {0}", index));
 _elasticsearchClient.Delete<Animal>(id);
}

LogProvider using Slab.Elasticsearch

This Provider uses Slab.Elasticsearch to log events to the Elasticsearch store. The Slab.Elasticsearch package uses Semantic Logging from the Enterprise Library and also the Nest Api.

The provider uses the ILogProvider from the business layer. This provider could easily be replaced with a log4net provider or a NLog provider.

namespace Damienbod.BusinessLayer.Providers
{
    public interface ILogProvider
    {
        void ServiceCritical(string message);
        void ServiceError(string message);
        void ServiceInformational(string message);
        void ServiceLogAlways(string message);
        void ServiceVerbose(string message);
        void ServiceWarning(string message);

        void ElasticSearchProviderCritical(string message);
        void ElasticSearchProviderError(string message);
        void ElasticSearchProviderInformational(string message);
        void ElasticSearchProviderLogAlways(string message);
        void ElasticSearchProviderVerbose(string message);
        void ElasticSearchProviderWarning(string message);

        void BusinessLayerCritical(string message);
        void BusinessLayerError(string message);
        void BusinessLayerInformational(string message);
        void BusinessLayerLogAlways(string message);
        void BusinessLayerVerbose(string message);
        void BusinessLayerWarning(string message);  
    }
}

The Events for logging are defined in the ServiceLayerEvents class or one of the other event classes. The demo application uses SLAB out of processing logging. See Semantic Logging with Elasticsearch for details on how to configure the package.

using System.Diagnostics.Tracing;

namespace Damienbod.LogProvider.Events
{
    [EventSource(Name = "ServiceLayerEvents")]
    public class ServiceLayerEvents : EventSource
    {
        public static readonly ServiceLayerEvents Log = new ServiceLayerEvents();

        [Event(1, Message = "ServiceLayerEvents Critical: {0}", Level = EventLevel.Critical)]
        public void Critical(string message)
        {
            if (IsEnabled()) WriteEvent(1, message);
        }

        [Event(2, Message = "ServiceLayerEvents Error {0}", Level = EventLevel.Error)]
        public void Error(string message)
        {
            if (IsEnabled()) WriteEvent(2, message);
        }

        [Event(3, Message = "ServiceLayerEvents Informational {0}", Level = EventLevel.Informational)]
        public void Informational(string message)
        {
            if (IsEnabled()) WriteEvent(3, message);
        }

        [Event(4, Message = "ServiceLayerEvents LogAlways {0}", Level = EventLevel.LogAlways)]
        public void LogAlways(string message)
        {
            if (IsEnabled()) WriteEvent(4, message);
        }

        [Event(5, Message = "ServiceLayerEvents Verbose {0}", Level = EventLevel.Verbose)]
        public void Verbose(string message)
        {
            if (IsEnabled()) WriteEvent(5, message);
        }

        [Event(6, Message = "ServiceLayerEvents Warning {0}", Level = EventLevel.Warning)]
        public void Warning(string message)
        {
            if (IsEnabled()) WriteEvent(6, message);
        }
    }
}

OUT-OF-PROCESS configuration for slab logging.

<customSink name="MyElasticsearchSink" type ="Slab.Elasticsearch.ElasticsearchSink, Slab.Elasticsearch">
  <sources>
    <eventSource name="ServiceLayerEvents" level="LogAlways" />
	<eventSource name="BusinessLayerEvents" level="LogAlways" />
	<eventSource name="ElasticSearchProviderEvents" level="LogAlways" />
  </sources>
  <parameters>
    <parameter name="connectionString" type="System.String" value="Server=localhost;Index=log;Port=9200" />
    <parameter name="searchIndex" type="System.String" value="log" />
    <parameter name="searchType" type="System.String" value="webapiexampleservice" />
  </parameters>
</customSink>

The Service Layer

This Layer is implemented using a MVC web application with a Web API RESTful service. This could also be defined in a windows service as the Web API uses OWIN and can be hosted anywhere. The AnimalController class is used for the service. This class uses attribute routing to define the routes. The Routing configuration is added in the WebApiConfig class found in the App_Start folder. This class also adds Unity to the Web API.

using System.Web.Http;
using System.Web.Http.ExceptionHandling;
using Damienbod.BusinessLayer.Providers;
using Microsoft.Practices.Unity;
using WebAPIRestWithNest.App_Start;

namespace WebAPIRestWithNest
{
    public static class WebApiConfig
    {
        public static void Register(HttpConfiguration config)
        {
            config.MapHttpAttributeRoutes();
           // config.DependencyResolver = new UnityDependencyResolver(UnityConfig.GetConfiguredContainer());
            config.Services.Add(typeof(IExceptionLogger), new SlabLogExceptionLogger(UnityConfig.GetConfiguredContainer().Resolve<ILogProvider>()));

            WebApiUnityActionFilterProvider.RegisterFilterProviders(config);
        }
    }
}

The AnimalController class.

using System.Collections.Generic;
using System.Web.Http;
using Damienbod.BusinessLayer.DomainModel;
using Damienbod.BusinessLayer.Managers;
using Damienbod.BusinessLayer.Providers;
using WebAPIRestWithNest.Filters;

namespace WebAPIRestWithNest.Controllers
{
    [RoutePrefix("api/animals")]
    [LoggingFilter]
    [AnimalExceptionFilter]
    public class AnimalsController : ApiController
    {
        private readonly IAnimalManager _animalManager;
        private readonly ILogProvider _logProvider;

        public AnimalsController(IAnimalManager animalManager, ILogProvider logProvider)
        {
            _animalManager = animalManager;
            _logProvider = logProvider;
        }

        // GET api/animals
        [HttpGet]
        [Route("")]
        public IEnumerable<Animal> Get()
        {
            return _animalManager.GetAnimals();
        }

        [HttpGet]
        [Route("{id}")]
        public Animal Get(int id)
        {
            return _animalManager.GetAnimal(id);
           
        }

        // POST api/animals
        [HttpPost]
        [Route("")]
        public void Post([FromBody]Animal value)
        {
            _animalManager.CreateAnimal(value);
        }

        // PUT api/animals/5
        [HttpPut]
        [HttpPatch]
        [Route("")]
        public void Put( [FromBody]Animal value)
        {
            _animalManager.UpdateAnimal(value);
        }

        // DELETE api/animals/5
        [HttpDelete]
        [Route("{id}")]
        public void Delete(int id)
        {
            _animalManager.DeleteAnimal(id);
        }

        // DELETE api/animals/5
        [HttpDelete]
        [Route("deleteIndex/{index}")]
        public void DeleteIndex(string index)
        {
            _animalManager.DeleteIndex(index);
        }
    }
}

The controller uses ActionFilters to log the service HTTP GET, POST, PUT, DELETE methods. This is added using an attribute and adds a log before and after the method execution.

using System.Diagnostics;
using System.Net.Http;
using System.Web.Http.Controllers;
using System.Web.Http.Filters;
using Damienbod.BusinessLayer.Providers;
using Microsoft.Practices.Unity;

namespace WebAPIRestWithNest.Filters
{
    public class LoggingFilterAttribute : ActionFilterAttribute
    {
        [Dependency]
        internal ILogProvider LogProvider { get; set; }

        public override void OnActionExecuting(HttpActionContext actionContext)
        {
            // pre-processing
            LogProvider.ServiceVerbose(string.Format("{0}, HTTP: {1}", actionContext.Request.RequestUri, actionContext.Request.Method));
        }
 
        public override void OnActionExecuted(HttpActionExecutedContext actionExecutedContext)
        {
            if((actionExecutedContext.Response != null))
            {
                var objectContent = actionExecutedContext.Response.Content as ObjectContent;
                if (objectContent != null)
                {
                    var type = objectContent.ObjectType; //type of the returned object
                    var value = objectContent.Value; //holding the returned value
                }
                LogProvider.ServiceVerbose(string.Format("{0}, HTTP STATUS CODE: {1}",
                    actionExecutedContext.Request.RequestUri, actionExecutedContext.Response.StatusCode));
            }           
        }
    }
}

Exceptions for this controller are handled using the AnimalExceptionFilterAttribute class.

using System;
using System.Net;
using System.Net.Http;
using System.Web.Http;
using System.Web.Http.Filters;

namespace WebAPIRestWithNest.Filters
{
    public class AnimalExceptionFilterAttribute : ExceptionFilterAttribute
    {
        public override void OnException(HttpActionExecutedContext context)
        {
            if (context.Exception is ArgumentException)
            {
                var resp = new HttpResponseMessage(HttpStatusCode.BadRequest)
                {
                    Content = new StringContent(context.Exception.Message),
                    ReasonPhrase = "ArgumentException"
                };
                throw new HttpResponseException(resp);

            }
        }
    }
}

All unhandled exceptions are logged using the SlabLogExceptionLogger. This implements the IExceptionLogger interface using the ExceptionLogger class.

using System.Web.Http.ExceptionHandling;
using Damienbod.BusinessLayer.Providers;

namespace WebAPIRestWithNest
{
    public class SlabLogExceptionLogger : ExceptionLogger
    {
        private readonly ILogProvider _logProvider;

        public SlabLogExceptionLogger(ILogProvider logProvider)
        {
            _logProvider = logProvider;
        }

        public override void Log(ExceptionLoggerContext context)
        {
            _logProvider.ServiceCritical(string.Format("{0}, {1}, {2}", context.Request.Method, context.Request.RequestUri, context.Exception.Message));
        }
    }
}

The service can be tested using curl, fiddler or postman.

Create HTTP POST

HTTP POST ../api/animals

User-Agent: Fiddler
Host: localhost:53548
Content-Length: 129
Accept : application/json
Content-Type: application/json

{
“dateOfBirth”: “2010-02-27T00:00:00”,
“id”: 75,
“updatedTimestamp”: “2014-02-27T20:36:04.8431623Z”,
“typeSpecificForAnimalType”: “Jack Russel”,
“createdTimestamp”: “2012-02-23T20:36:04.8431623Z”,
“description”: “dangerous but controlled”,
“gender”: “Female”,
“lastLocation”: “Wabern”,
“animalType”: “cat”
}

If the create method is used for an existing document, the service returns a bad request.

Update HTTP PUT

HTTP PUT../api/animals

User-Agent: Fiddler
Host: localhost:53548
Content-Length: 129
Accept : application/json
Content-Type: application/json

{
“dateOfBirth”: “2010-02-27T00:00:00”,
“id”: 75,
“updatedTimestamp”: “2014-02-27T20:36:04.8431623Z”,
“typeSpecificForAnimalType”: “Jack Russel”,
“createdTimestamp”: “2012-02-23T20:36:04.8431623Z”,
“description”: “dangerous but controlled”,
“gender”: “Female”,
“lastLocation”: “Wabern”,
“animalType”: “cat”
}

Delete HTTP DELETE

HTTP DELETE../api/animals/{id}

GetAll HTTP GET

HTTP GET../api/animals

Here’s an example of Animal documents in the database:

Next steps:

Now that a basic application is setup to do CRUD operations with Elasticsearch, the service can be extended to implement some search logic.

Links:
http://nest.azurewebsites.net/

http://www.elasticsearch.org/

https://github.com/mastoj/NestDemo

http://sravi-kiran.blogspot.in/2013/12/UnitTestingAsynchronousWebApiActionMethodsUsingMsTest.html