Azure AI Search: Indexers and Indexes using C# Code

-

In this article, we will implement the Azure AI Search Service Data Source, Index, and Indexer creation using the C# code. We will implement the process using the ASP.NET Core 9 APIs. The application targets not only to configure Azure AI Search but also the application exposes API endpoints to perform search operations on the Data Source configured with Azure AI Search.  Formally known as Azure Cognitive Search, the Azure AI Search is an enterprise-ready information retrieval system. We can build the heterogeneous contents information ingestion into search index e.g. Documents, Relational Data, etc to query so that the data can be retrieved. Teh Azure AI Service is designed for high-performance and scalability. Some of the features of Azure AI Search are as follows:

  • Rich Indexing capabilities with content transformation, data chunking, vectorization for RAG.
  • Vector and full-text search for data retrieval with efficiency.
  • Advanced Query syntax
  • Integration with other Azure Cloud hosted services
  • Security and compliance
Make sure that you have an Azure Subscription and Resource Group created. All the resources mentioned in this article e.g. Azure SQL, and Azure AI Search.
 
In this article, we will use the relational data stored in Azure SQL. I have used the Northwind database which can be downloaded from this link. Use the instnwnd (Azure SQL Database).sql to create the Northwind database. I have used the Query as shown in Listing 1 to create new Table named OrdersReport. This table will be used to store data from multiple table that we will be using for the indexing in Azure AI Search.

SELECT OrderID, Customers.ContactName AS CustomerName, 
       Employees.FirstName + ' ' + Employees.LastName AS EmployeeName, 
       OrderDate, RequiredDate, ShippedDate, Shippers.CompanyName AS ShipperName, 
       Freight, ShipName, ShipAddress, ShipCity, ShipPostalCode, ShipCountry
INTO OrdersReport
FROM Orders, Customers, Employees, Shippers
WHERE Customers.CustomerID = Orders.CustomerID 
  AND Employees.EmployeeID = Orders.EmployeeID 
  AND Shippers.ShipperID = Orders.ShipVia;
Listing 1: The OrdersTable   

You must create Azure SQL Database and run the instnwnd (Azure SQL Database).sql to create all tables.  
Figure 1 shows the Database and Tables created in Azure SQL.


Figure 1: The Northwind database and its Tables
 
Once the Azure SQL Database is created copy its Connection String. We will be using it to connect to Database from Azure AI Search.
 
Let's create an Azure AI Search Service using the Azure Portal as shown in Figure 2.


Figure 2: The Azure AI Service

Click on the Scale to set the scaling (Note that I have used the Pricing Tier as Basic), the scaling can be managed based on the Pricing tier. Once the scaling is set make sure that Azure AI Service is set Endpoint network connectivity to Public. We need to set this so that the Service can be accessed from any machine on the internet. Figure 3 shows the settings.


Figure 3: The Endpoint network connectivity
 
Click on Review+create button to make sure that your Azure AI Search settings are correct as shown in Figure 4.


Figure 4: Final Settings for Azure AI Search

We can connect with the Azure AI Search service from the client application e.g. .NET Client Application, etc. we need use the Azure AI Endpoint as well as Authentication keys. As shown in Figure 4, we need to copy the Primary Key so that we can use it in code.


Figure 4: The Primary Admin Keys

As shown in Figure 5, copy the URL so that we can connect to it from the client application.


Figure 5: The Azure AI Search Service URL
 

Creating the ASP.NET Client Application

Step 1: Open Visual Studio 2022 and create a new ASP.NET Core API project. Name this application as Core_AISearch_Code. Make sure that the Minimal APIs are selected.

Step 2: In this project ass following 2 NuGet Packages

  • Azure.Identity
  • Azure.Search.Documents Version 11.6.0
Make sure that you select this version because version features are very important.

Step 3: Modify the appsettings.json to add new keys for AzureSettings e.g. Azure AI Search Key, Endpoint, Azure SQL Connection string as shown in Listing 2.

{
    "Logging": {
        "LogLevel": {
            "Default": "Information",
            "Microsoft.AspNetCore": "Warning"
        }
    },
    "AllowedHosts": "*",
    "AzureSettings": {
        "AzureAISearchEndPoint": "https://[AZURE-AI-SEARCH-ENDPOINT].search.windows.net",
        "AzureAISearchKey": "[AZURE-AI-SEARCH-PRIMARY-ADMIN-KEY]",
        "AzureAISearchIndexName": "shipping-index",
        "AzureAISearchIndexerName": "shipping-indexer",
        "AzureAISearchDataSourceName": "shipping-datasource",
        "AzureSQLDbConnectionString": "[AZURE-SQL-DATABASE-CONNECTION-STRING]"
    }
}
Listing 2: The appsettings.json 
 
Step 4: In the project add a new folder named Models. In this folder we will add two class files named OrderField.cs and Query.cs. In the OrderField.cs file, we will add code for the ShippingIndex class. This class will contain properties mapped with columns of the OrdersReport table that we created in Northwind database in Azure SQL. This class must have the Key field that matches with the Primary Key of the OrdersReport table (Make sure that once the OrdersReport table is created, we have to explicitly alter this table to add the Primary Key Constraints to the OrderId column.). Other properties from class except the decimal properties must be applied with the SearchFieldAttribute. The SearchFieldAttribute is used in Azure AI Search to define how a field behaves within a search index. This allows developers to specify attributes like filterability, sortability, faceting, and analyzers for search fields in an index.

Key Properties of the SearchFieldAttribute class are as follows:

IsFilterable: Enables filtering using $filter queries.

IsSortable: Allows sorting with $orderby expressions.

IsFacetable: Supports faceted navigation.

IsKey: Marks the field as the primary key.

The code for the ShippingIndex class is shown in Listing 3.

using Azure.Search.Documents.Indexes;

namespace Core_AISearch_Code.Models
{
    public class ShippingIndex
    {
        [SimpleField(IsKey = true)]
        public string OrderID { get; set; }

        [SearchableField(IsFilterable = true)]
        public string? CustomerName { get; set; }

        [SearchableField(IsFilterable = true)]
        public string? EmployeeName { get; set; }

        public DateTime? OrderDate { get; set; }
        public DateTime? RequiredDate { get; set; }
        public DateTime? ShippedDate { get; set; }

        [SearchableField(IsFilterable = true)]
        public string? ShipperName { get; set; }

        [FieldBuilderIgnore]
        public decimal? Freight { get; set; } // Manually define in SearchField

        [SearchableField(IsFilterable = true)]
        public string? ShipName { get; set; }

        public string? ShipAddress { get; set; }

        [SearchableField(IsFilterable = true)]
        public string? ShipCity { get; set; }

        public string? ShipPostalCode { get; set; }

        [SearchableField(IsFilterable = true, IsSortable = true)]
        public string? ShipCountry { get; set; }
    }

}
Listing 3: The ShippingIndex class
  
In the Query.cs file we will add code for Query class; we will use this class to make the POST request to search data from the Azure AI Search service. Listing 4 shows the code for the Query class.

namespace Core_AISearch_Code.Models
{
    public class Query
    {
        public string? SearchText { get; set; }
    }
}
Listing 4: The Query class

Step 5: In the project add a new folder named Services. In this folder add a new class file named SearchServiceManager.cs. This file will contain code for the SearchServiceManager class. This class uses the following classes: 

  • AzureKeyCredential: 
    • This is a key-based authentication mechanism used in Azure SDKs to authenticate requests to Azure services. It allows to securely manage API keys without needing to create a new client when updating credentials. This class has the following features like:
      • Secure authentication: This uses an API key to authenticate requests from the client application.
      • Key rotation support: This allows updating the key dynamically without creating a new client.
  • SearchIndexClient:
    • This is a class in Azure AI Search that allows developers to manage search indexes programmatically. This provides methods for creating, updating, and deleting indexes, as well as configuring search settings. This class has following Key Features:
      • Index Management: Used to Create, update, or delete search indexes.
      • Alias Handling: Used to Manage search aliases for index versioning.
      • Text Analysis: Use analyzers to process text for indexing.
  • SearchIndexerDataSourceConnection:
    • This is a class in Azure AI Search that defines a connection to a data source for an indexer. This class allows to configure how an indexer retrieves data from external sources like Azure Blob Storage, Azure SQL databases, or Cosmos DB, etc. The class has the following properties:
      • ConnectionString: This Specifies the connection string to the data source so that the data can be queried.
      • Container: This is used to define the data container for indexing e.g. Database Table, Azure Blob Container.
      • DataChangeDetectionPolicy: This determines how changes in the data source are detected.      
      • DataDeletionDetectionPolicy: This configures how deleted records are handled.
      • EncryptionKey: This supports encryption using Azure Key Vault for added security.
  • SearchIndexerClient:
    • This class in Azure AI Search is used to provide operations for managing indexers, data source connections, and skillsets. This class allows developers to automate data ingestion and indexing from external sources like Azure Blob Storage, SQL databases, and Cosmos DB. The class has the following features:
      • Create, update, and delete indexers for automated data ingestion.
      • Manage data source connections to link external data sources.
      • Run and reset indexers to refresh indexed data.
      • Retrieve indexer status to monitor indexing operations.
  • SearchIndexer:
    • This component in Azure AI Search is used to automates the process of extracting, transforming, and loading data into a search index. This is used to connects to various data sources, applies AI-powered enrichment, and ensures that indexed content remains up to date. The class has the following features: 
      • Automated Data Ingestion: This pulls data from the configured sources with Azure AI Search service like Azure Blob Storage, SQL databases, and Cosmos DB.
      • AI-Powered Enrichment: This uses skillsets to enhance search results with natural language processing, image analysis, and entity recognition.
      • Incremental Indexing: This detect and updates only changed records to optimize performance.
      • Data Change & Deletion Detection: This ensures accurate indexing by tracking modifications and removals.
  • IndexingParameters:
    • This class in Azure AI Search is used to define parameters for indexer execution. This class allows developers to configure how data is processed during indexing. The class has following key features:
      • Batch Size: This property is used to specify the number of items read from the data source and indexed as a single batch.
      • Max Failed Items: This property sets the maximum number of items that can fail indexing before the process is considered unsuccessful.
      • Max Failed Items Per Batch: This property defines the failure threshold for individual batches.
      • Configuration: This property provides indexer-specific settings, such as parsing mode and data extraction preferences.
  • SearchDocument:
    • This class in Azure AI Search is used to represents an untyped document returned from a search query or document lookup. This class allows developers to access search results dynamically as either a dictionary or a dynamic object. The class has following properties:
      • Flexible Data Access: This can be accessed using key-value pairs or dynamic properties.
      • Supports JSON Serialization: This enables an easy conversion between JSON and search results.
The code for the SearchServiceManager class is shown in Listing 5.
using System.Reflection.Metadata;
using Azure;
using Azure.Search.Documents;
using Azure.Search.Documents.Indexes;
using Azure.Search.Documents.Indexes.Models;
using Azure.Search.Documents.Models;
using Core_AISearch_Code.Models;
using Microsoft.Identity.Client.Platforms.Features.DesktopOs.Kerberos;

namespace Core_AISearch_Code.Services
{
    public class SearchServiceManager
    {
        AzureKeyCredential _credential;
        SearchIndexClient _indexClient;
        IConfiguration configuration;
        string searchKey, searchEndpoint;
        IConfiguration _configuration;

        public SearchServiceManager(IConfiguration configuration)
        {
            _configuration = configuration ?? throw new ArgumentNullException(nameof(configuration));
            searchKey = _configuration["AzureSettings:AzureAISearchKey"] ?? throw new ArgumentNullException("AzureSettings:AzureAISearchKey");
            searchEndpoint = _configuration["AzureSettings:AzureAISearchEndPoint"] ?? throw new ArgumentNullException("AzureSettings:AzureAISearchEndPoint");
            _credential = new AzureKeyCredential(searchKey);

            _indexClient = new SearchIndexClient(new Uri(searchEndpoint), _credential);
        }

        public void CreateIndex()
        {
            var fields = new FieldBuilder().Build(typeof(ShippingIndex));
            var definition = new SearchIndex("shipping-index", fields);
            _indexClient.CreateOrUpdateIndex(definition);

            var connectionString = _configuration["AzureSettings:AzureSQLDbConnectionString"] ?? throw new ArgumentNullException("AzureSettings:AzureSQLDbConnectionString");
            

            var dataSource = new SearchIndexerDataSourceConnection(
                name: "shipping-datasource",
                type: SearchIndexerDataSourceType.AzureSql,
                connectionString: connectionString,
                container: new SearchIndexerDataContainer("OrdersReport")
            );

            var indexerClient = new SearchIndexerClient(new Uri(searchEndpoint), _credential);
            indexerClient.CreateOrUpdateDataSourceConnection(dataSource);

            var indexer = new SearchIndexer(
                name: "shipping-indexer",
                dataSourceName: dataSource.Name,
                targetIndexName: "shipping-index"
            );

            indexer.Parameters = new IndexingParameters
            {
                BatchSize = 1,
                MaxFailedItems = -1,
                MaxFailedItemsPerBatch = -1
            };

            indexerClient.CreateOrUpdateIndexer(indexer);
        }

        public SearchResults<SearchDocument> Search(string searchText)
        {
            var client = new SearchClient(new Uri(searchEndpoint), "shipping-index", _credential);
            var options = new SearchOptions
            {
                Size = 10
            };
            return client.Search<SearchDocument>(searchText, options);
        }
    }
}

Listing 5: The SearchServiceManager class
The code on Listing 5 for the SearchServiceManager class reads the appsettings to read the Azure AI Search Admin Key and its Endpoint to get authenticated and connect with the Azure AI Search Service. The CreateIndex() method of the class creates a Search Index named shipping-index, Data Source named shipping-datasource for Azure AI Search Service as well as the Indexer named shipping-indexer. The Search() method is used to perform the search operations over the index. This is the major part of the application because here we are performing major operations on Azure AI Search service using the code. This offers great scale of the flexibility because instead of using the Azure Portal we are using the C# code. This helps to dynamically choose the Inde Data Source for the Azure AI Search service. This means that based on the available data we can use it for the search.

Step 6: Modify the Program.cs so that we can register the SearchServiceManager class in dependency container as Singleton as shown in Listing 6.

...............
builder.Services.AddSingleton<SearchServiceManager>();
...............
Listing 6: registration of the SearchServiceManager class
 
In Program.cs, lets add API endpoints as shown in Listing 7 to create Index, DataSource, and Indexer.


................

app.MapGet("/index", () =>
{
    
    var searchServiceManager = app.Services.GetRequiredService<SearchServiceManager>();
    searchServiceManager.CreateIndex();
    return Results.Ok("Index created successfully.");
});


app.MapGet("/search", (string searchText) =>
{

    var searchServiceManager = app.Services.GetRequiredService<SearchServiceManager>();
    var results = searchServiceManager.Search(searchText);
    var output = new List<SearchDocument>();
    foreach (var result in results.GetResults())
    {
        output.Add(result.Document);
    }
    return Results.Ok(output);
    
});

app.MapPost("/data", (Query query) =>
{

    var searchServiceManager = app.Services.GetRequiredService<SearchServiceManager>();
    var results = searchServiceManager.Search(query.SearchText);
    var output = new List<SearchDocument>();
    foreach (var result in results.GetResults())
    {
        output.Add(result.Document);
    }
    return Results.Ok(output);

});

................
Listing 7: The API Endpoints to create Index, DataSource, and Indexer
  
Run the application and use the .http file to make the call to API. Listing 8 shows http calls to the APIs.

@Core_AISearch_Code_HostAddress = http://localhost:5019

GET {{Core_AISearch_Code_HostAddress}}/index
Accept: application/json

###
GET {{Core_AISearch_Code_HostAddress}}/search?searchText=CustomerName=Mario
Accept: application/json

###

POST {{Core_AISearch_Code_HostAddress}}/data
Accept: application/json
Content-Type: application/json

{"SearchText":"EmployeeName=Nancy Davolio"}

###
Listing 8: The .http file
 
Once the index endpoint is hit the Azure AI Search service will be added with Data Source, Index, and Indexer as shown in following Figures. Figure 6 Data Source



Figure 6: The Azure AI Search Service Data Source
 
The Figure 7 shows the Index


Figure 7: The Azure AI Search Service Index

Figure 8, shows the Indexer with all 830 documents from the OrdersReport table from the Azure SQL database.


Figure 8: The Azure AI Search Service Indexer
 
Try the Post request as shown in Listing 8, we will get the response for EmployeeName as Nancy Davolio as shown in Figure 9.


Figure 9: The POST request
   
We have received the response as per the query that we sent in the POST request.

Conclusion: The Azure AI Search Service is the most important service for building AI Power solutions. Using the code configuring the Data Source, Index, and Indexer provides greater flexibility for managing the search service so that we can use it easily for Enterprise AI-Powered Search.

Code for this article can be downloaded from this link.

 

Popular posts from this blog

Uploading Excel File to ASP.NET Core 6 application to save data from Excel to SQL Server Database

-
Image
Recently, while conducting a session on ASP.NET Core 6, my client raised a query regarding uploading excel files to the ASP.NET Core MVC application and then further saving data from excel to the SQL Server Database table. I gave him the solution and then thought to write a post on it. As we all know, the Excel Workbook is one of the most heavily used files to save data. But to persist data it is recommended that it should be stored in an SQL Server database.  If the data is stored in an SQL Server database it can be queried easily.  Figure 1, explains an implementation of the application which we are going to discuss in the article. Figure 1: The ASP.NET Core That accepts Excel files and then further store data from Excel Files in SQL Server Database The   ExcelDataReader.DataSet Library To read excel files in ASP.NET Core applications, we need to use the ExcelDataReader.DataSet package. This is a lightweight and fast library for reading Excel files in .NET applications....
Read more

ASP.NET Core 6: Downloading Files from the Server

-
Image
In this article, we will implement the file downloading feature using ASP.NET Core 6 application. In real-world applications, we can have various files like reports, spreadsheets, images, PDF documents stored on the server. (Note that all cloud-hosted applications prefer to store these files on the storage services, e.g. Blob, S3, etc.). But what if the client prefers to store files on the separate file server mapped to the web application host server? Then, in that case, we may need to download these files from the file server. In this article, we will implement exactly the same thing.   Step 1: Open Visual Studio 2022 and create a new ASP.NET Core MVC Application targetted to .NET 6. Name this application as Core6_FileDownload. In this project add a new folder and name it as ServerFiles. In this file add some images. (You can add Excel, PDF, Word, Files). Step 2: Since we need to read the ServerFiles folder using the ASP.NET Core application, we need to add the IFi...
Read more

ASP.NET Core 6: Using Entity Framework Core with Oracle Database with Code-First Approach

-
Image
In this article, we will see how to use Oracle Database with Entity Framework Core with Code-First Approach. The reason behind writing a post on this is, recently while conducting training, one of my students raised the query regarding the EF Core with Oracle. Since Oracle is one of the most powerful databases, various enterprise applications are using it for the past several years and since the .NET Core is the most popular technology for building modern apps, the scenarios for accessing the Oracle database in .NET Core apps the use Entity Framework Core is common. Oracle Data Provider for .NET Entity Framework Core The Oracle Data Provider for .NET i.e. ODP.NET is used to access data from the Oracle database in the .NET Core apps using Entity Framework Core. This uses the Oracle.EntityFrameworkCore.dll, a managed library. This data provider allows using Code-First and Database-First approaches to perform Read/Write Operations with Oracle database.    Implementation To implem...
Read more