Angular search with ASP.NET Core and Elasticsearch

This article shows how a website search could be implemented using Angular, ASP.NET Core and Elasticsearch. Most users expect autocomplete and a flexible search like some of known search websites. When the user enters a char in the search input field, an autocomplete using a shingle token filter with a terms aggregation used to suggest possible search terms. When a term is selected, a match query request is sent and uses an edge ngram indexed field to search for hits or matches. Server side paging is then implemented to iterate though the results.

Code: https://github.com/damienbod/Angular2AutoCompleteAspNetCoreElasticsearch

2017.02.10: Updated to VS2017, Angular 2.4.6 and webpack 2.2.1
2017.01.07: Updated to csproj, webpack 2.2.0-rc.3, angular 2.4.1

ASP.NET Core server side search

The Elasticsearch index and queries was built using the ideas from these 2 excellent blogs, bilyachat and qbox.io. ElasticsearchCrud is used as the dotnet core client for Elasticsearch. To setup the index, a mapping needs to be defined as well as the index with the required settings analysis with filters, analyzers and tokenizers. See the Elasticsearch documentation for detailed information.

In this example, 2 custom analyzers are defined, one for the autocomplete and one for the search. The autocomplete analyzer uses a custom shingle token filter called autocompletefilter, a stopwords token filter, lowercase token filter and a stemmer token filter. The edge_ngram_search analyzer uses an edge ngram token filter and a lowercase filter.

private IndexDefinition CreateNewIndexDefinition()
{
	return new IndexDefinition
	{
		IndexSettings =
		{
			Analysis = new Analysis
			{
				Filters =
				{
					CustomFilters = new List<AnalysisFilterBase>
					{
						new StemmerTokenFilter("stemmer"),
						new ShingleTokenFilter("autocompletefilter")
						{
							MaxShingleSize = 5,
							MinShingleSize = 2
						},
						new StopTokenFilter("stopwords"),
						new EdgeNGramTokenFilter("edge_ngram_filter")
						{
							MaxGram = 20,
							MinGram = 2
						}
					}
				},
				Analyzer =
				{
					Analyzers = new List<AnalyzerBase>
					{
						new CustomAnalyzer("edge_ngram_search")
						{
							Tokenizer = DefaultTokenizers.Standard,
							Filter = new List<string> {DefaultTokenFilters.Lowercase, "edge_ngram_filter"},
							CharFilter = new List<string> {DefaultCharFilters.HtmlStrip}
						},
						new CustomAnalyzer("autocomplete")
						{
							Tokenizer = DefaultTokenizers.Standard,
							Filter = new List<string> {DefaultTokenFilters.Lowercase, "autocompletefilter", "stopwords", "stemmer"},
							CharFilter = new List<string> {DefaultCharFilters.HtmlStrip}
						},
						new CustomAnalyzer("default")
						{
							Tokenizer = DefaultTokenizers.Standard,
							Filter = new List<string> {DefaultTokenFilters.Lowercase, "stopwords", "stemmer"},
							CharFilter = new List<string> {DefaultCharFilters.HtmlStrip}
						}
						
					   
					}
				}
			}
		},
	};
}

The PersonCity is used to add and search for documents in Elasticsearch. The default index and type for this class using ElasticsearchCrud is personcitys and personcity.

public class PersonCity
{
	public long Id { get; set; }
	public string Name { get; set; }
	public string FamilyName { get; set; }
	public string Info { get; set; }
	public string CityCountry { get; set; }
	public string Metadata { get; set; }
	public string Web { get; set; }
	public string Github { get; set; }
	public string Twitter { get; set; }
	public string Mvp { get; set; }
}

A PersonCityMapping class is defined so that required mapping from the PersonCityMappingDto mapping class can be defined for the personcitys index and the personcity type. This class overrides the ElasticsearchMapping to define the index and type.

using System;
using ElasticsearchCRUD;

namespace SearchComponent
{
    public class PersonCityMapping : ElasticsearchMapping
    {
        public override string GetIndexForType(Type type)
        {
            return "personcitys";
        }

        public override string GetDocumentType(Type type)
        {
            return "personcity";
        }
    }
}

The PersonCityMapping class is then used to map the C# type PersonCityMappingDto to the default index from the PersonCity class using the PersonCityMapping. The PersonCityMapping maps to the default index of the PersonCity class.

public PersonCitySearchProvider()
{
	_elasticsearchMappingResolver.AddElasticSearchMappingForEntityType(typeof(PersonCityMappingDto), new PersonCityMapping());
	_context = new ElasticsearchContext(ConnectionString, new ElasticsearchSerializerConfiguration(_elasticsearchMappingResolver))
	{
		TraceProvider = new ConsoleTraceProvider()
	};
}

A specific mapping DTO class is used to define the mapping in Elasticsearch. This class is required, if a non default mapping is required in Elasticsearch. The class uses the ElasticsearchString attribute to define a copy mapping. The field in Elasticsearch should be copied to the autocomplete and the searchfield field when adding a new document. The searchfield and the autocomplete field uses the two analyzers which where defined in the index when adding data. This class is only used to define the type mapping in Elasticsearch.

using ElasticsearchCRUD.ContextAddDeleteUpdate.CoreTypeAttributes;

namespace SearchComponent
{
    public class PersonCityMappingDto
    {
        public long Id { get; set; }

        [ElasticsearchString(CopyToList = new[] { "autocomplete", "searchfield" })]
        public string Name { get; set; }

        [ElasticsearchString(CopyToList = new[] { "autocomplete", "searchfield" })]
        public string FamilyName { get; set; }

        [ElasticsearchString(CopyToList = new[] { "autocomplete", "searchfield" })]
        public string Info { get; set; }

        [ElasticsearchString(CopyToList = new[] { "autocomplete", "searchfield" })]
        public string CityCountry { get; set; }

        [ElasticsearchString(CopyToList = new[] { "autocomplete", "searchfield" })]
        public string Metadata { get; set; }

        public string Web { get; set; }

        public string Github { get; set; }

        public string Twitter { get; set; }

        public string Mvp { get; set; }

        [ElasticsearchString(Analyzer = "edge_ngram_search", SearchAnalyzer = "standard", TermVector = TermVector.yes)]
        public string searchfield { get; set; }

        [ElasticsearchString(Analyzer = "autocomplete")]
        public string autocomplete { get; set; }
    }
}

The IndexCreate method creates a new index and mapping in elasticsearch.

public void CreateIndex()
{			
	_context.IndexCreate<PersonCityMappingDto>(CreateNewIndexDefinition());
}

The Elasticsearch settings can be viewed using the HTTP GET:

http://localhost:9200/_settings

{
	"personcitys": {
		"settings": {
			"index": {
				"creation_date": "1477642409728",
				"analysis": {
					"filter": {
						"stemmer": {
							"type": "stemmer"
						},
						"autocompletefilter": {
							"max_shingle_size": "5",
							"min_shingle_size": "2",
							"type": "shingle"
						},
						"stopwords": {
							"type": "stop"
						},
						"edge_ngram_filter": {
							"type": "edgeNGram",
							"min_gram": "2",
							"max_gram": "20"
						}
					},
					"analyzer": {
						"edge_ngram_search": {
							"filter": ["lowercase",
							"edge_ngram_filter"],
							"char_filter": ["html_strip"],
							"type": "custom",
							"tokenizer": "standard"
						},
						"autocomplete": {
							"filter": ["lowercase",
							"autocompletefilter",
							"stopwords",
							"stemmer"],
							"char_filter": ["html_strip"],
							"type": "custom",
							"tokenizer": "standard"
						},
						"default": {
							"filter": ["lowercase",
							"stopwords",
							"stemmer"],
							"char_filter": ["html_strip"],
							"type": "custom",
							"tokenizer": "standard"
						}
					}
				},
				"number_of_shards": "5",
				"number_of_replicas": "1",
				"uuid": "TxS9hdy7SmGPr4FSSNaPiQ",
				"version": {
					"created": "2040199"
				}
			}
		}
	}
}

The Elasticsearch mapping can be viewed using the HTTP GET:

http://localhost:9200/_mapping

{
	"personcitys": {
		"mappings": {
			"personcity": {
				"properties": {
					"autocomplete": {
						"type": "string",
						"analyzer": "autocomplete"
					},
					"citycountry": {
						"type": "string",
						"copy_to": ["autocomplete",
						"searchfield"]
					},
					"familyname": {
						"type": "string",
						"copy_to": ["autocomplete",
						"searchfield"]
					},
					"github": {
						"type": "string"
					},
					"id": {
						"type": "long"
					},
					"info": {
						"type": "string",
						"copy_to": ["autocomplete",
						"searchfield"]
					},
					"metadata": {
						"type": "string",
						"copy_to": ["autocomplete",
						"searchfield"]
					},
					"mvp": {
						"type": "string"
					},
					"name": {
						"type": "string",
						"copy_to": ["autocomplete",
						"searchfield"]
					},
					"searchfield": {
						"type": "string",
						"term_vector": "yes",
						"analyzer": "edge_ngram_search",
						"search_analyzer": "standard"
					},
					"twitter": {
						"type": "string"
					},
					"web": {
						"type": "string"
					}
				}
			}
		}
	}
}

Now documents can be added using the PersonCity class which has no Elasticsearch definitions.

Autocomplete search

A terms aggregation search is used for the autocomplete request. The terms aggregation uses the autocomplete field which only exists in Elasticsearch. A list of strings is returned to the user from this request.

public IEnumerable<string> AutocompleteSearch(string term)
{
	var search = new Search
	{
		Size = 0,
		Aggs = new List<IAggs>
		{
			new TermsBucketAggregation("autocomplete", "autocomplete")
			{
				Order= new OrderAgg("_count", OrderEnum.desc),
				Include = new IncludeExpression(term + ".*")
			}
		}
	};

	var items = _context.Search<PersonCity>(search);
	var aggResult = items.PayloadResult.Aggregations.GetComplexValue<TermsBucketAggregationsResult>("autocomplete");
	IEnumerable<string> results = aggResult.Buckets.Select(t =>  t.Key.ToString());
	return results;
}

The request is sent to Elasticsearch as follows:

POST http://localhost:9200/personcitys/personcity/_search HTTP/1.1
Content-Type: application/json
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Content-Length: 124
Host: localhost:9200

{
	"size": 0,
	"aggs": {
		"autocomplete": {
			"terms": {
				"field": "autocomplete",
				"order": {
					"_count": "desc"
				},
				"include": {
					"pattern": "as.*"
				}
			}
		}
	}
}

Search Query

When an autocomplete string is selected, a search request is sent to Elasticsearch using a Match Query on the searchfield field which returns 10 hits from the 0 document. If the paging request is sent, the from value is a multiple of 10 depending on the page.

public PersonCitySearchResult Search(string term, int from)
{
	var personCitySearchResult = new PersonCitySearchResult();
	var search = new Search
	{
		Size = 10,
		From = from,
		Query = new Query(new MatchQuery("did_you_mean", term))
	};

	var results = _context.Search<PersonCity>(search);

	personCitySearchResult.PersonCities = results.PayloadResult.Hits.HitsResult.Select(t => t.Source);
	personCitySearchResult.Hits = results.PayloadResult.Hits.Total;
	personCitySearchResult.Took = results.PayloadResult.Took;
	return personCitySearchResult;
}

The search query as sent as follows:

POST http://localhost:9200/personcitys/personcity/_search HTTP/1.1
Content-Type: application/json
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Content-Length: 74
Host: localhost:9200

{
	"from": 0,
	"size": 10,
	"query": {
		"match": {
			"searchfield": {
				"query": "asp.net"
			}
		}
	}
}

Angular 2 client side search

The Angular 2 client uses an autocomplete input control and then uses a ngFor to display all the search results. Bootstrap paging is used if more than 10 results are found for the search term.

<div class="panel-group">

    <personcitysearch 
      *ngIf="IndexExists"
      (onTermSelectedEvent)="onTermSelectedEvent($event)"
      [disableAutocomplete]="!IndexExists">
    </personcitysearch>
    
    <em *ngIf="PersonCitySearchData.took > 0" style="font-size:smaller; color:lightgray;">
      <span>Hits: {{PersonCitySearchData.hits}}</span>
    </em><br /> 
    <br />

    <div  *ngFor="let personCity of PersonCitySearchData.personCities">  
        <b><span>{{personCity.name}} {{personCity.familyName}} </span></b> 
        <a *ngIf="personCity.twitter"  href="{{personCity.twitter}}">
          <img src="assets/socialTwitter.png" />
        </a>
        <a *ngIf="personCity.github" href="{{personCity.github}}">
          <img src="assets/github.png" />
        </a>
        <a *ngIf="personCity.mvp" href="{{personCity.mvp}}">
          <img src="assets/mvp.png" width="24" />
        </a><br />

        <em style="font-size:large"><a href="{{personCity.web}}">{{personCity.web}}</a></em><br />  
        <em><span>{{personCity.metadata}}</span></em><br />      
        <span>{{personCity.info}}</span><br />
        <br />
        <br />

    </div>

    <ul class="pagination" *ngIf="ShowPaging">
        <li><a (click)="PreviousPage()" >&laquo;</a></li>

        <li><a *ngFor="let page of Pages" (click)="LoadDataForPage(page)">{{page}}</a></li>

        <li><a (click)="NextPage()">&raquo;</a></li>
    </ul>
</div>

The personcitysearch Angular 2 component implements the autocomplete functionality using the ng2-completer component. When a char is entered into the input, a HTTP request is sent to the server which in turns sends a request to the Elasticsearch server.

import { Component, Inject, EventEmitter, Input, Output, OnInit, AfterViewInit, ElementRef } from '@angular/core';
import { Http, Response } from "@angular/http";

import { Subscription } from 'rxjs/Subscription';
import { Observable } from 'rxjs/Observable';
import { Router } from  '@angular/router';

import { Configuration } from '../app.constants';
import { PersoncityautocompleteDataService } from './personcityautocompleteService';
import { PersonCity } from '../model/personCity';

import { CompleterService, CompleterItem } from 'ng2-completer';

import './personcityautocomplete.component.scss';

@Component({
    selector: 'personcityautocomplete',
  template: `
<ng2-completer [dataService]="dataService" (selected)="onPersonCitySelected($event)" [minSearchLength]="0" [disableInput]="disableAutocomplete"></ng2-completer>

`
})
    
export class PersoncityautocompleteComponent implements OnInit    {

    constructor(private completerService: CompleterService, private http: Http, private _configuration: Configuration) {

        this.dataService = new PersoncityautocompleteDataService(http, _configuration); ////completerService.local("name, info, familyName", 'name');
    }

    @Output() bindModelPersonCityChange = new EventEmitter<PersonCity>();
    @Input() bindModelPersonCity: PersonCity;
    @Input() disableAutocomplete: boolean = false;

    private searchStr: string;
    private dataService: PersoncityautocompleteDataService;

    ngOnInit() {
        console.log("ngOnInit PersoncityautocompleteComponent");
    }

    public onPersonCitySelected(selected: CompleterItem) {
        console.log(selected);
        this.bindModelPersonCityChange.emit(selected.originalObject);
    }
}

And the data service for the CompleterService which is used by the ng2-completer component:

import { Http, Response } from "@angular/http";
import { Subject } from "rxjs/Subject";

import { CompleterData, CompleterItem } from 'ng2-completer';
import { Configuration } from '../app.constants';

export class PersoncityautocompleteDataService extends Subject<CompleterItem[]> implements CompleterData {
    constructor(private http: Http, private _configuration: Configuration) {
        super();

        this.actionUrl = _configuration.Server + 'api/personcity/querystringsearch/';
    }

    private actionUrl: string;

    public search(term: string): void {
        this.http.get(this.actionUrl + term)
            .map((res: Response) => {
                // Convert the result to CompleterItem[]
                let data = res.json();
                let matches: CompleterItem[] = data.map((personcity: any) => {
                    return {
                        title: personcity.name,
                        description: personcity.familyName + ", " + personcity.cityCountry,
                        originalObject: personcity
                    }
                });
                this.next(matches);
            })
            .subscribe();
    }

    public cancel() {
        // Handle cancel
    }
}

The HomeSearchComponent implements the paging for the search results and and also displays the data. The SearchDataService implements the API calls to the MVC ASP.NET Core API service. The paging css uses bootstrap to display the data.

import { Observable } from 'rxjs/Observable';
import { Component, OnInit } from '@angular/core';
import { Http } from '@angular/http';

import { SearchDataService } from '../services/searchDataService';
import { PersonCity } from '../model/personCity';
import { PersonCitySearchResult } from '../model/personCitySearchResult';
import { PersoncitysearchComponent } from '../personcitysearch/personcitysearch.component';

@Component({
    selector: 'homesearchcomponent',
    templateUrl: 'homesearch.component.html',
    providers: [SearchDataService]
})

export class HomeSearchComponent implements OnInit {

    public message: string;
    public PersonCitySearchData: PersonCitySearchResult;
    public SelectedTerm: string;
    public IndexExists: boolean = false;

    constructor(private _dataService: SearchDataService, private _personcitysearchComponent: PersoncitysearchComponent) {
        this.message = "Hello from HomeSearchComponent constructor";
        this.SelectedTerm = "none";
        this.PersonCitySearchData = new PersonCitySearchResult();
    }

    public onTermSelectedEvent(term: string) {
        this.SelectedTerm = term; 
        this.findDataForSearchTerm(term, 0)
    }

    private findDataForSearchTerm(term: string, from: number) {
        console.log("findDataForSearchTerm:" + term);
        this._dataService.FindAllForTerm(term, from)
            .subscribe((data) => {
                console.log(data)
                this.PersonCitySearchData = data;
                this.configurePagingDisplay(this.PersonCitySearchData.hits);
            },
            error => console.log(error),
            () => {
                console.log('PersonCitySearch:findDataForSearchTerm completed');
            }
            );
    }

    ngOnInit() {
        this._dataService
            .IndexExists()
            .subscribe(data => this.IndexExists = data,
            error => console.log(error),
            () => console.log('Get IndexExists complete'));
    }

    public ShowPaging: boolean = false;
    public CurrentPage: number = 0;
    public TotalHits: number = 0;
    public PagesCount: number = 0;
    public Pages: number[] = [];

    public LoadDataForPage(page: number) {
        var from = page * 10;
        this.findDataForSearchTerm(this.SelectedTerm, from)
        this.CurrentPage = page;
    }

    public NextPage() {
        var page = this.CurrentPage;
        console.log("TotalHits" + this.TotalHits + "NextPage: " + ((this.CurrentPage + 1) * 10) + "CurrentPage" + this.CurrentPage );

        if (this.TotalHits > ((this.CurrentPage + 1) * 10)) {
            page = this.CurrentPage + 1;
        }

        this.LoadDataForPage(page);
    }

    public PreviousPage(page: number) {
        var page = this.CurrentPage;

        if (this.CurrentPage > 0) {
            page = this.CurrentPage - 1;
        }

        this.LoadDataForPage(page);
    }

    private configurePagingDisplay(hits: number) {
        this.PagesCount = Math.floor(hits / 10);

        this.Pages = [];
        for (let i = 0; i <= this.PagesCount; i++) {
            this.Pages.push((i));
        }
        
        this.TotalHits = hits;

        if (this.PagesCount <= 1) {
            this.ShowPaging = false;
        } else {
            this.ShowPaging = true;
        }
    }
}

Now when characters are entered into the search input, records are searched for and returned with the amount of hits for the term.

The paging can also be used, to do the server side paging.

The search functions like a web search with which we have come to expect. If different results, searches are required, the server side index creation, query types can be changed as needed. For example, the autocomplete suggestions could be replaced with a fuzzy search, or a query string search.

Links:

https://github.com/oferh/ng2-completer

https://github.com/damienbod/Angular2WebpackVisualStudio

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

https://www.elastic.co/products/elasticsearch

https://www.nuget.org/packages/ElasticsearchCRUD/

https://github.com/damienbod/ElasticsearchCRUD

http://www.bilyachat.com/2015/07/search-like-google-with-elasticsearch.html

http://stackoverflow.com/questions/29753971/elasticsearch-completion-suggest-search-with-multiple-word-inputs

http://rea.tech/implementing-autosuggest-in-elasticsearch/

https://qbox.io/blog/an-introduction-to-ngrams-in-elasticsearch

https://www.elastic.co/guide/en/elasticsearch/guide/current/_ngrams_for_partial_matching.html

Tags: Angular, ASP.NET Core, aspnetcore, autocomplete, dotnet, Elasticsearch, ng2-completer, ngrams, search

4 comments

Mesut A · December 20, 2016 - 11:48 · Reply→

how to debug angular2 typescript in visual studio with breakpoint or its possible?
Dave · January 4, 2017 - 15:20 · Reply→

Great post! I’m about to attempt a similar implementation and this will save me a lot of time. You do GREAT work. Keep it up!

@Mesut – Use SystemJS and set tsconfig “sourceMap”: true if you are letting VS transpile TS (or use gulp-typescript and gulp-sourcemaps) Just make sure the .js.map and .js files are published. Breakpoints will only hit in VS 2015 if you use IE to debug. I haven’t seen it work with Webpack bundling but heard that it’s possible to put sourcemaps inline somehow. Breakpoints DO hit in VS with IE browserlink.
1. damienbod · January 4, 2017 - 17:18 · Reply→
  
  thanks
Abdulrahman · November 21, 2020 - 12:57 · Reply→

Hi thanks for this work,
I have a question. How do i use a MongoDB connectionstring in the code.

Thanks.

Software Engineering

Angular search with ASP.NET Core and Elasticsearch

4 comments

Leave a Reply Cancel reply

Categories

blogs 1 web, general

blogs 2 security

blogs 3 Azure

blogs 4 SSI DID

blogs 5a

blogs 5b

tools

web

Meta

Archives

Software Engineering

Angular search with ASP.NET Core and Elasticsearch

Share this:

Like this:

Related

4 comments

Leave a Reply Cancel reply

Categories

blogs 1 web, general

blogs 2 security

blogs 3 Azure

blogs 4 SSI DID

blogs 5a

blogs 5b

tools

web

Meta

Archives