Implement a full text search using Azure Cognitive Search in ASP.NET Core

This article shows how to implement a full text search in ASP.NET Core using Azure Cognitive Search. The search results are returned using paging and the search index can be created, deleted from an ASP.NET Core Razor Page UI.

Code: https://github.com/damienbod/AspNetCoreAzureSearch

Posts in this series

Creating the Search in the Azure Portal

In the Azure Portal, search for Azure Cognitive Search and create a new search service. Create the search using the portal wizard and choose the correct pricing model as required. The free version supports three indexes but does not support managed identities. This is good for exploring, evaluating the service.

If using the free version, you will need to use API keys to access the search service. This can be found in the Keys blade of the created cognitive search.

Of course the Azure Cognitive Search could also be created using Azure CLI, Arm templates or Powershell. The service can also be created direct from code.

Create an Azure Cognitive Search index

In the ASP.NET Core Razor page application, the Azure.Search.Documents nuget package is used to create and search the Azure Cognitive search service. Add this to your project.

The index and the document field definitions can be created in different ways. We will use attributes and add these to the document search class properties to define the fields of the documents.

public class PersonCity
{
   [SimpleField(IsFilterable = true, IsKey = true)]
   public string Id { get; set; }

   [SearchableField(IsFilterable = true, IsSortable = true)]
   public string Name { get; set; }

   [SearchableField(IsFilterable = true, IsSortable = true)]
   public string FamilyName { get; set; }

   [SearchableField(IsFilterable = true, IsSortable = true)]
   public string Info { get; set; }

   [SearchableField(IsFilterable = true, IsSortable = true, IsFacetable = true)]
   public string CityCountry { get; set; }

   [SearchableField(IsFilterable = true, IsSortable = true, IsFacetable = true)]
   public string Metadata { get; set; }

   [SearchableField(IsFilterable = true, IsSortable = true)]
   public string Web { get; set; }

   [SearchableField(IsFilterable = true, IsSortable = true)]
   public string Github { get; set; }

   [SearchableField(IsFilterable = true, IsSortable = true)]
   public string Twitter { get; set; }

   [SearchableField(IsFilterable = true, IsSortable = true)]
   public string Mvp { get; set; }
}

A SearchProvider class was created to managed and search the indexes and the documents. The SearchProvider was added to the DI in the startup class. The configuration secrets for the Azure search was added to the user secrets of the project. A SearchIndexClient and a SearchClient instance was created using the configurations for your service. The SearchIndexClient can the be used to create a new instance using the CreateIndexAsync method and the field builder which uses the attribute definitions.

public class SearchProvider
{
	private readonly SearchIndexClient _searchIndexClient;
	private readonly SearchClient _searchClient;
	private readonly IConfiguration _configuration;
	private readonly IHttpClientFactory _httpClientFactory;
	private readonly string _index;

	public SearchProvider(IConfiguration configuration, IHttpClientFactory httpClientFactory)
	{
            _configuration = configuration;
            _httpClientFactory = httpClientFactory;
            _index = configuration["PersonCitiesIndexName"];

            Uri serviceEndpoint = new Uri(configuration["PersonCitiesSearchUri"]);
            AzureKeyCredential credential = new AzureKeyCredential(configuration["PersonCitiesSearchApiKey"]);

            _searchIndexClient = new SearchIndexClient(serviceEndpoint, credential);
            _searchClient = new SearchClient(serviceEndpoint, _index, credential);
            
	}

	public async Task CreateIndex()
	{
            FieldBuilder bulder = new FieldBuilder();
            var definition = new SearchIndex(_index, 
               bulder.Build(typeof(PersonCity)));

            await _searchIndexClient.CreateIndexAsync(definition)
             .ConfigureAwait(false);
	}

Once created, VS Code with the Azure Cognitive Search extension can be used to view the index with the created fields. This can also be viewed, managed in the Azure portal.

https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-azurecognitivesearch

Adding search documents

Now that the index exists, it needs some documents so that we can search. Azure Cognitive Search provides many powerful ways of importing data into the search indexes, this is one of its strengths. In this demo, we added some data from a data helper class and uploaded the documents in a batch. This could be the way to add data if using the search as a secondary search engine for your solution.

public async Task AddDocumentsToIndex(List<PersonCity> personCities)
{
	var batch = IndexDocumentsBatch.Upload(personCities);
	await _searchClient.IndexDocumentsAsync(batch)
	  .ConfigureAwait(false);
}

The ASP.NET Razor Search Admin page provides a post method OnPostAddDataAsync to add the index documents.

public async Task<ActionResult> OnPostAddDataAsync()
{
	try
	{
		PersonCityData.CreateTestData();
		await _searchProvider.AddDocumentsToIndex(PersonCityData.Data);
		Messages = new[] {
			new AlertViewModel("success", "Documented added", 
"The Azure Search documents were uploaded! The Document Count takes n seconds to update!"),
		};
		var indexStatus = await _searchProvider.GetIndexStatus();
		IndexExists = indexStatus.Exists;
		DocumentCount = indexStatus.DocumentCount;
		return Page();
	}
	catch (Exception ex)
	{
		Messages = new[] {
			new AlertViewModel("danger", "Error adding documents", ex.Message),
		};
		return Page();
	}
}

The view uses a Bootstrap 4 card to display this and documents can be added to the index.

<div class="card">
	<div class="card-body">
		<h5 class="card-title">Add Documents to index: @Model.IndexName</h5>
		<p class="card-text">Add documents to the Azure Cognitive search index: @Model.IndexName.</p>
	</div>
	<div class="card-footer text-center">
		<form asp-page="/SearchAdmin" asp-page-handler="AddData">
			<button type="submit" class="btn btn-primary col-sm-6">
				Add
			</button>
		</form>
	</div>
</div>

Checking the status of the index

In the ASP.NET Core search administration Razor Page view, we would like to be able to see if the index exists and how many documents exist in the index. The easiest way to do this, is to use the REST API from the Azure search service. The HttpClient is used and the count is returned or a 404.

public async Task<(bool Exists,long DocumentCount)> GetIndexStatus()
{
	try
	{
		var httpClient = _httpClientFactory.CreateClient();
		httpClient.DefaultRequestHeaders.CacheControl = new CacheControlHeaderValue
		{
			NoCache = true,
		};
		httpClient.DefaultRequestHeaders.Add("api-key", _configuration["PersonCitiesSearchApiKey"]);

		var uri = $"{_configuration["PersonCitiesSearchUri"]}/indexes/{_index}/docs/$count?api-version=2020-06-30";
		var data = await httpClient.GetAsync(uri);
		if (data.StatusCode == System.Net.HttpStatusCode.NotFound)
		{
			return (false, 0);
		}
		var payload = await data.Content.ReadAsStringAsync();
		return (true, int.Parse(payload));
	}
	catch
	{
		return (false, 0);
	}
}

When the application is started, the search admin displays the amount of documents, can create or delete the index and add documents to the index.

Implementing a search with Paging

The search is implemented using the QueryPagingFull method which uses the SearchAsync method. The QueryType is set to SearchQueryType.Full in the options so that we can use a fuzzy search. The page size and the range for the paging is defined at the top. The SearchAsync returns a SearchResults object which contains the results. This can then be used as required.

public async Task QueryPagingFull(SearchData model, int page, int leftMostPage)
{
	var pageSize = 4;
	var maxPageRange = 7;
	var pageRangeDelta = maxPageRange - pageSize;

	var options = new SearchOptions
	{
		Skip = page * pageSize,
		Size = pageSize,
		IncludeTotalCount = true, 
		QueryType= SearchQueryType.Full
	};

	model.PersonCities = await _searchClient.SearchAsync<PersonCity>(
	      model.SearchText, options).ConfigureAwait(false);
	model.PageCount = ((int)model.PersonCities.TotalCount + pageSize - 1) / pageSize;
	model.CurrentPage = page;
	if (page == 0)
	{
		leftMostPage = 0;
	}
	else if (page <= leftMostPage)
	{
		leftMostPage = Math.Max(page - pageRangeDelta, 0);
	}
	else if (page >= leftMostPage + maxPageRange - 1)
	{
		leftMostPage = Math.Min(page - pageRangeDelta, model.PageCount - maxPageRange);
	}
	model.LeftMostPage = leftMostPage;
	model.PageRange = Math.Min(model.PageCount - leftMostPage, maxPageRange);
}

The Razor Page uses the SearchProvider and sets up the models so the view can display the data and call the search APIs.

using Azure.Search.Documents.Models;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;
using Microsoft.Extensions.Logging;
using System.Threading.Tasks;

namespace AspNetCoreAzureSearch.Pages
{
    public class SearchModel : PageModel
    {
        private readonly SearchProvider _searchProvider;
        private readonly ILogger<IndexModel> _logger;

        public string SearchText { get; set; }
        public int CurrentPage { get; set; }
        public int PageCount { get; set; }
        public int LeftMostPage { get; set; }
        public int PageRange { get; set; }
        public string Paging { get; set; }
        public int PageNo { get; set; }
        public SearchResults<PersonCity> PersonCities;

        public SearchModel(SearchProvider searchProvider,
            ILogger<IndexModel> logger)
        {
            _searchProvider = searchProvider;
            _logger = logger;
        }

        public void OnGet()
        {
        }

        public async Task<ActionResult> OnGetInitAsync(string searchText)
        {
            SearchData model = new SearchData
            {
                SearchText = searchText
            };

            await _searchProvider.QueryPagingFull(model, 0, 0).ConfigureAwait(false);

            SearchText = model.SearchText;
            CurrentPage = model.CurrentPage;
            PageCount = model.PageCount;
            LeftMostPage = model.LeftMostPage;
            PageRange = model.PageRange;
            Paging = model.Paging;
            PersonCities = model.PersonCities;

            return Page();
        }

        public async Task<ActionResult> OnGetPagingAsync(SearchData model)
        {
            int page;

            switch (model.Paging)
            {
                case "prev":
                    page = PageNo - 1;
                    break;

                case "next":
                    page = PageNo + 1;
                    break;

                default:
                    page = int.Parse(model.Paging);
                    break;
            }

            int leftMostPage = LeftMostPage;

            await _searchProvider.QueryPagingFull(model, page, leftMostPage).ConfigureAwait(false);

            PageNo = page;
            SearchText = model.SearchText;
            CurrentPage = model.CurrentPage;
            PageCount = model.PageCount;
            LeftMostPage = model.LeftMostPage;
            PageRange = model.PageRange;
            Paging = model.Paging;
            PersonCities = model.PersonCities;

            return Page();
        }

    }
}

The view uses Bootstrap 4 and displays the results. All requests are sent using HTTP GET which are cached and can be navigated using the back button. The searchText is added to the query string and also the handler required for the Razor page.

@page "{handler?}"
@model SearchModel
@{
    ViewData["Title"] = "Search with Paging";
}

<form asp-page="/Search" asp-page-handler="Init" method="get">
    <div class="searchBoxForm">
        @Html.TextBoxFor(m => m.SearchText, new { @class = "searchBox" }) 
        <input class="searchBoxSubmit" type="submit" value="">
    </div>
</form>

@if (Model.PersonCities != null)
{
    <p class="sampleText">
        Found @Model.PersonCities.TotalCount Documents
    </p>

    var results = Model.PersonCities.GetResults().ToList();

    @for (var i = 0; i < results.Count; i++)
    {
        <div>
            <b><span><a href="@results[i].Document.Web">@results[i].Document.Name @results[i].Document.FamilyName</a>: @results[i].Document.CityCountry</span></b>
            @if (!string.IsNullOrEmpty(results[i].Document.Twitter))
            {
                <a href="@results[i].Document.Twitter"><img src="/images/socialTwitter.png" /></a>
            }
            @if (!string.IsNullOrEmpty(results[i].Document.Github))
            {
                <a href="@results[i].Document.Github"><img src="/images/github.png" /></a>
            }
            @if (!string.IsNullOrEmpty(results[i].Document.Mvp))
            {
                <a href="@results[i].Document.Mvp"><img src="/images/mvp.png" width="24" /></a>
            }
            <br />
            <em><span>@results[i].Document.Metadata</span></em><br />
            @Html.TextArea($"desc{1}", results[i].Document.Info, new { @class = "infotext" })
            <br />
        </div>
    }
}

@if (Model != null && Model.PageCount > 1)
{
    <table>
        <tr>
            <td>
                @if (Model.CurrentPage > 0)
                {
                    <p class="pageButton">
                        <a href="/Search?handler=Paging&paging=0&SearchText=@Model.SearchText">|<</a>
                    </p>
                }
                else
                {
                    <p class="pageButtonDisabled">|&lt;</p>
                }
            </td>

            <td>
                @if (Model.CurrentPage > 0)
                {
                    <p class="pageButton">
                        <a href="/Search?handler=Paging&paging=prev&SearchText=@Model.SearchText"><</a>
                    </p>
                }
                else
                {
                    <p class="pageButtonDisabled">&lt;</p>
                }
            </td>

            @for (var pn = Model.LeftMostPage; pn < Model.LeftMostPage + Model.PageRange; pn++)
            {
                <td>
                    @if (Model.CurrentPage == pn)
                    {
                        <p class="pageSelected">@(pn + 1)</p>
                    }
                    else
                    {
                        <p class="pageButton">
                            @{var p1 = Model.PageCount - 1;}
                            <a href="/Search?handler=Paging&paging=@pn&SearchText=@Model.SearchText">@(pn + 1)</a>
                        </p>
                    }
                </td>

            }

            <td>
                @if (Model.CurrentPage < Model.PageCount - 1)
                {
                    <p class="pageButton">
                        @{var p1 = Model.PageCount - 1;}
                        <a href="/Search?handler=Paging&paging=next&SearchText=@Model.SearchText">></a>
                    </p>
                }
                else
                {
                    <p class="pageButtonDisabled">&gt;</p>
                }
            </td>

            <td>
                @if (Model.CurrentPage < Model.PageCount - 1)
                {
                    <p class="pageButton">
                        @{var p7 = Model.PageCount - 1;}
                        <a href="/Search?handler=Paging&paging=@p7&SearchText=@Model.SearchText">>|</a>
                    </p>
                }
                else
                {
                    <p class="pageButtonDisabled">&gt;|</p>
                }
            </td>
        </tr>
    </table>
}

Searching, Fuzzy Search

The search can be used by entering a text and clicking the search icon. The results and paging are returned as defined. Ten results were found for “Switzerland” using a full word match.

If you spell the required word incorrectly or leave out a letter, no results will be returned.

This can improved or allowed by using a fuzzy search. The “~” can be added to use a fuzzy search for Azure Cognitive Search. Then the ten results will be found again. Azure Cognitive search supports different types of search queries, search filters and the indexes can be created to support different search types.

Notes

The demo here was built using the Azure search samples found here.

Links

https://docs.microsoft.com/en-us/azure/search

https://docs.microsoft.com/en-us/azure/search/search-what-is-azure-search

https://docs.microsoft.com/en-us/rest/api/searchservice/

https://github.com/Azure-Samples/azure-search-dotnet-samples/

https://channel9.msdn.com/Shows/AI-Show/Azure-Cognitive-Search-Deep-Dive-with-Debug-Sessions

https://channel9.msdn.com/Shows/AI-Show/Azure-Cognitive-Search-Whats-new-in-security

One comment

  1. […] Implement a full text search using Azure Cognitive Search in ASP.NET Core (Damien Bowden) […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: