using ASP.NET Core 10 Telemetry with OpenTelemetry and ELK

A page is slow, an API returns 500, or a database call fails. The user sees the symptom, but where do you start looking? A console message might tell you that something went wrong. To understand why, you need to connect that message to the request, its dependencies and the state of the process.

That is the problem we'll work through with MVC_App. The solution has an ASP.NET Core application with MVC, Razor Pages and JSON APIs, a Blazor Server interface, and two console clients. We'll follow a request through the code, see how its telemetry reaches the backend, and use Kibana to investigate it.

One distinction matters from the start: the MVC and Blazor applications send telemetry to ELK. The API console generates traffic for them, while the PostgreSQL console sends its own telemetry to Azure Monitor/Application Insights. Keeping those paths separate will save you from looking for data in the wrong place.

Connection strings and keys in the examples are redacted. Supply your own private configuration when running the solution, and rotate any credentials that have already been shared.

Topics Covered in this Article

• What telemetry means
• How ASP.NET Core 10 uses telemetry
• Advantages of telemetry with ELK
• How the solution fits together
• Following a request through the code
• YAML, Logstash and application configuration
• Kibana KQL queries and dashboard panels
• Step-by-step application startup and validation
• Security, reliability and operational limitations
• A closer look at the code
• Conclusion

1. What is telemetry?

Telemetry is the information an application produces while it runs: requests handled, time spent waiting, memory used, exceptions raised, and much more. Sending that information outside the process gives you something to investigate after a request has finished or after the process itself has stopped.

You'll often hear telemetry discussed alongside observability. The distinction is useful: telemetry is the evidence; observability is how well you can use that evidence to understand the system. Monitoring tells you when a known threshold has been crossed. A good set of telemetry also helps you investigate a problem you hadn't anticipated.

The three signals we'll use are logs, metrics and distributed traces. OpenTelemetry, or OTel, provides the APIs, SDKs, instrumentation and Collector that gather and move them. OTLP is the transport protocol. At the other end, ELK, Elasticsearch, Logstash and Kibana stores, transforms and presents the data. “TEL” simply means telemetry here, not a separate component.

Signal	Question Answered	Example in this Solution
Logs	What event occurred, and with which context?	A department conflict, an API-key challenge, a SQL command failure, or a Blazor render exception.
Metrics	How frequently, how slowly, or how much?	HTTP duration, runtime memory/GC observations, PostgreSQL employee operation counters, and a duration histogram.
Traces	Which operations participated in a request, and where was time spent?	A Blazor outgoing HTTP span, the MVC incoming request span, and SQL command child spans.

Think of a trace as the story of an operation. Its spans describe the individual steps: an incoming HTTP request, an outgoing API call, or a SQL command. Each span has timing information, attributes and an identifier; related spans share a trace ID. A log written while a .NET Activity is current can carry that same context, giving you a way to move from a message to the request behind it.

Resource attributes answer a different question: which application produced this? In this solution, service.name and deployment.environment.name identify the service and environment. They describe the producer rather than a particular request.

Suppose the API starts returning 500 responses. Metrics show how often it happens. A trace helps locate the failing dependency. Logs explain the error in more detail. Looking at all three is usually more useful than collecting more of just one.

2. Connecting ASP.NET Core 10 to OpenTelemetry

If you've used ILogger<T>, you've already worked with one of the building blocks. .NET also provides System.Diagnostics.Activity and ActivitySource for tracing, and System.Diagnostics.Metrics.Meter for metrics. OpenTelemetry connects these diagnostics to an export pipeline; ASP.NET Core does not send them to ELK on its own.

Both web projects call builder.AddApplicationTelemetry() during startup. The method brings the pieces together in five steps:

1. Check the configuration. If telemetry is disabled or the endpoint is missing, the method returns without registering this OTel pipeline. Other configured logging providers can still run.
2. Keep the message and its context. AddOpenTelemetry includes formatted messages and logging scopes. The scopes carry fields such as the response status and error category, so you can filter them without searching message text.
3. Follow the request and its dependencies. ASP.NET Core instrumentation observes incoming HTTP requests, HttpClient instrumentation observes outgoing calls, and MVC_App adds SqlClient tracing. Propagated W3C context can connect the spans across services.
4. Collect measurements. HTTP and runtime instrumentation provide metrics. MVC_App also listens to the Kestrel, routing and diagnostics meters; each contributes data when the relevant activity occurs.
5. Send the signals to the Collector. UseOtlpExporter(OtlpExportProtocol.HttpProtobuf, endpointUri) configures export for logs, traces and metrics. The local base URL is http://localhost:4318; the exporter appends paths such as /v1/traces.

The solution targets net10.0, although these diagnostic APIs span several .NET releases. The important separation is between recording an operation and storing it: an Activity still needs a working exporter and backend before you can inspect it in Kibana.

Why Blazor needs more than HTTP middleware

Blazor adds a wrinkle. After the initial page load, the browser works through a server-side circuit. An exception raised by a later component event may occur long after that first HTTP request has ended. To cover those paths, the project uses incoming request middleware, an outgoing ApiLoggingHandler, and LoggingErrorBoundaryLogger. The middleware skips /_blazor traffic to reduce noise. There is no browser OTel SDK here, so don't expect a complete browser trace or assume every click belongs to the initial page trace.

Why the console projects differ

The API console is useful for generating traffic: it calls the backend and prints timings, while the backend produces the telemetry you'll search in ELK. The PostgreSQL console is a separate example. It creates business activities and metrics, adds Npgsql/runtime/process instrumentation, and exports to Azure Monitor. Because a console run may finish quickly, it explicitly flushes its providers before exiting.

3. Advantages of telemetry with ELK

The payoff comes when you need to investigate something across application boundaries. Instead of switching between consoles and guessing which messages belong together, you can ask focused questions about a service, a request or a failure.

• Search both web applications in one place. The service identity tells you whether a record came from MVC or Blazor.
• Filter on meaning, not wording. Stable error categories let you find conflicts, authorization failures and server errors even if their messages change.
• Follow a reported problem. When context is available, the trace ID returned to an API caller connects the visible error to its request, SQL calls and related logs.
• Shape the data before indexing it. Logstash turns OTLP batches into searchable records and repairs a legacy field name along the way.
• See patterns over time. Elasticsearch aggregations and Kibana charts help you compare services, routes, statuses and error categories.
• Keep backend details out of application code. The application uses OTel APIs; it doesn't need to know the name of an Elasticsearch index.
• Check the pipeline with known failures. The diagnostic endpoints and workload clients produce predictable events you can trace from request to dashboard.

There is still a system to operate. Telemetry consumes CPU, network capacity and storage, and the data needs access controls and retention rules. This sample gives you the collection path and query examples; alerting, SLOs, browser real-user monitoring and saved dashboards would be separate additions.

4. How the solution fits together

Figure 1: Whole-solution architecture: applications, databases, ELK and the separate Azure Monitor destination.

Project	Runtime Role	Dependencies / Telemetry Destination
MVC_App	MVC, JSON API, and Razor Pages	Company SQL database and separate Identity SQL database. Telemetry exported via OTLP/HTTP → Collector → ELK.
MVC_App.BlazorClient	Interactive-server Blazor UI	MVC JSON API and shared Identity database. Telemetry exported via OTLP/HTTP → same Collector → ELK.
MVC_App.ApiClient	Scenario and load generator	MVC JSON API. No direct exporter; generates observable backend traffic.
MVC_App.PgSqlClient	PostgreSQL scenario/interactive client	PostgreSQL database. Telemetry sent to Azure Monitor/Application Insights; optional local console exporters.

 Figure 2: Complete flow: browser and API-console traffic, Blazor and MVC database access, OTLP collection, direct trace/metric export, logs through JSONL and Logstash into Elasticsearch and Kibana, and the independent PostgreSQL-to-Azure Monitor telemetry path.

Read the architecture in two parts: application calls at the top, telemetry delivery underneath. Only the logs pass through Logstash. Traces and metrics take a shorter route from the Collector to Elasticsearch, while PostgreSQL telemetry goes to Azure Monitor independently.

The web applications run on the development machine. compose.yaml starts the four observability containers on the telemetry bridge network. Inside that network, the Collector and Logstash reach Elasticsearch at http://elasticsearch:9200. The host applications use the published loopback receiver. This difference is easy to miss: localhost inside a container means that container, not the host or another service.

5. Following a request through the code

Figure 3: Request and error flow through Blazor, ASP.NET Core middleware, controllers and SQL instrumentation.

5.1 An API call from Blazor

Start with a department or employee action in Blazor. The component calls Api/MvcAppApiClient.cs, whose typed HttpClient supplies the API key and runs through ApiLoggingHandler. HttpClient instrumentation can create a client span and pass its trace context to MVC. The backend creates a server span, authenticates the caller and validates the request before reaching the controller's database work.

That work uses CompanyContext. SqlClient instrumentation records database spans, and the custom EF interceptor logs each command's timing and outcome. A single button click can therefore produce several useful records: the outbound call, the incoming request and one or more SQL commands.

The outbound API key authenticates the Blazor server as an API client; it is not the browser user's delegated token. Sharing the Identity database permits the same account credentials in both web applications, but the distinct cookies do not implement shared-session single sign-on.

5.2 A handled conflict and an unhandled fault

Now suppose a department name already exists. The controller raises a domain conflict, and API middleware returns a 409 problem document with an error category and trace ID. That event is logged as Warning, not Error—a useful distinction when you're searching for expected business failures.

The browser MVC flow behaves differently: it redirects to the error page. You may see a 409 in the exception event and a 302 in the final request log, followed by a new request for the error page. Neither record is necessarily wrong; they describe different stages. Unexpected API exceptions follow the 500 path, while an authentication challenge can return 401 without throwing at all.

5.3 SQL telemetry is both logs and spans

DbCommandLoggingInterceptor logs command operation/database/duration and at most 1,000 characters of text. Separately, SqlClient tracing adds db.statement to spans. The filter on Microsoft.EntityFrameworkCore.Database.Command only suppresses that framework category's informational OTel logs; it does not remove command text emitted by the custom interceptor or by span enrichment.

Parameterized commands do not export parameter values through this interceptor, but raw SQL literals and provider exception messages still need scrutiny. Logs inherit whichever Activity is current when they are emitted. Trace-level correlation is useful; do not assume an EF post-execution callback always retains the SQL span as its current Activity.

 6. YAML and configuration, explained

Figure 4: ELK deployment and signal routing: logs through Logstash, traces and metrics directly from the Collector to Elasticsearch.

6.1 compose.yaml: the local infrastructure

Service	Pinned Image / Host Port	Key Behavior
elasticsearch	Elastic 9.5.4 / 127.0.0.1:9200	Single node; security and HTTP TLS disabled; 512 MB Java heap, 1 GB container limit; readiness accepts green or yellow cluster health.
kibana	Elastic 9.5.4 / 127.0.0.1:5601	Connects to Elasticsearch over the bridge network; waits for its health check; 2 GB memory limit.
otel-collector	Contrib 0.161.0 / 127.0.0.1:4318	Read-only config mount; read/write host directory at /otel-output; 256 MB memory limit; waits for Elasticsearch health.
logstash	Elastic 9.5.4 / No published port	Read-only pipeline/log mounts; persistent read offsets in logstash-data; 512 MB Java heap, 1 GB container limit; two workers and batch size eight.

Before starting Docker, allow enough memory for the stack. These four container limits add up to about 4.25 GB, before the applications, databases, Docker itself and the host OS. The table shows the pinned image versions; check their availability in your environment. The optional MVC Dockerfile is separate from this Compose setup.

There is no Filebeat or Elastic APM Server service. Logstash tails the Collector output directly. No named Elasticsearch data volume is declared, so this is not a managed durable-storage design. Depending on the image, anonymous storage may exist; do not rely on container recreation to preserve or reattach your data. The explicitly named logstash-data volume stores input offsets, not the Elasticsearch documents.

6.2 observability/otel-collector.yaml: one receiver, three routes

• Receiver: otlp.protocols.http.endpoint: 0.0.0.0:4318 listens within the container. Compose restricts host access to loopback. OTLP gRPC/4317 is not configured.
• Memory limiter: checks every second with limit_mib: 192 and spike_limit_mib: 48. It can refuse data under pressure; a container memory limit is not a lossless-delivery guarantee.
• Batch: timeout: 5s and send_batch_size: 512 reduce exporter overhead. Send batch size is a trigger, not a configured hard upper bound; there is no send_batch_max_size here.
• Elasticsearch exporter: requires data streams, allows OTel/ECS mapping modes, queues up to 256 export requests and retries a limited number of times. No persistent queue storage is configured. Default stream names are retained to match built-in OTel templates; actual streams should be discovered rather than assumed.
• File exporter: writes one OTLP JSON export envelope per line to /otel-output/logs.jsonl. Rotation uses 10 MB, three days and three backups. One line can contain many log records.
• Pipelines: traces and metrics use the Elasticsearch exporter; logs use only file/logs. Adding an exporter definition alone does not route data to it: the pipeline's exporter list makes that choice.

Do not take the commented logs_index example as an active setting. This Collector's log pipeline does not use its Elasticsearch exporter at all. The custom daily log-index name is configured in Logstash, while trace and metric streams follow the Collector exporter's compatible naming.

6.3 observability/logstash/pipeline/otel-logs.conf: from envelopes to events

Figure 5: Logstash transformation from batched OTLP JSON envelopes to individual searchable log records.

1. The file input watches /otel-output/logs*.jsonl*, including rotated files. It starts new/untracked files at the beginning and stores offsets at /usr/share/logstash/data/sincedb-otel-logs.
2. The json codec parses each already separated line. json_lines would wait for delimiters that this input has already removed.
3. Ruby traverses resourceLogs → scopeLogs → logRecords and decodes OTLP AnyValue strings, numbers, booleans, arrays and key-value lists. Unsupported representations such as bytesValue are not handled by this helper and become nil.
4. It constructs a fresh small event for each record rather than repeatedly deep-copying the entire envelope with nested split filters. This keeps fan-out memory usage much more manageable.
5. It copies message/severity, trace_id/span_id, resource_attributes/log_attributes, service_name, deployment_environment and scope_name. Nanosecond epoch timestamps are converted to Logstash timestamps via a floating-point seconds conversion, so this is not exact nanosecond preservation.
6. It rewrites the legacy http.route.matched attribute to http.route_matched. Otherwise Elasticsearch could need http.route to be both a scalar string and an object, rejecting documents.
7. It cancels the original envelope and indexes the emitted records into mvcapp-logs-%{+YYYY.MM.dd}.

The file hand-off has limits. Resetting sincedb can replay old records, and without deterministic document IDs those replays create duplicates. If Logstash falls behind for long enough, rotation can remove logs before they are read. There is no persistent Logstash queue configured here. Treat observability/otel-output as runtime data, and keep its JSONL files private: they may contain SQL text or personal information.

6.4 Application, project and container configuration

Base appsettings.json files define logging, database connections and application credentials. Development files set the web OTLP endpoint and distinct service names. The custom extension explicitly reads endpoint/name/enable keys from IConfiguration; that does not mean every standard SDK environment variable is automatically read from JSON. In particular, set OTEL_METRIC_EXPORT_INTERVAL as an actual process environment variable when controlling the SDK reader interval unless explicit binding is verified.

launchSettings.json supplies local URLs/environment for IDE or dotnet-run profiles. It is not production deployment configuration. The .csproj files in the solution define target frameworks and package dependencies. .dockerignore excludes appsettings and local launch settings, so a Docker deployment must inject those settings externally.

If older setup notes describe a different route for logs, check the current Compose and pipeline files. The configuration shown here sends logs through Logstash and persists its read offsets in the named volume.

7. Kibana KQL queries and dashboard panels

Figure 6: Investigating a user report in Kibana using service identity, trace IDs and structured error fields.

7.1 Create the data view

1. After starting the applications and generating traffic using section 8, open http://localhost:5601.
2. Open Data Views under Stack Management (or create a data view from Discover, depending on the Kibana navigation). Enter mvcapp-logs-* as the index pattern and choose @timestamp as the time field.
3. Open Discover, select that data view, and select KQL in the query-language selector—not Lucene or ES|QL.
4. Set the time picker to Last 15 minutes, or an absolute range covering the test. Enable a modest auto-refresh interval such as 10 seconds during a demonstration.
5. Add service_name, severity_text, message, trace_id, scope_name and relevant log_attributes fields as columns. Inspect an actual document before choosing aggregation fields.

7.2 Here are some of the Copy-ready Kibana KQL examples, you can use to test the ELK

Paste one query at a time into Discover or a dashboard panel's KQL filter. KQL selects the documents; Lens turns those results into aggregations and charts. These filters use the fields written by the Logstash pipeline. Check those fields in your own index before building panels, and remember that Kibana KQL is different from Azure's Kusto Query Language.

Question	KQL Filter	Interpretation
1. What are both web services logging?	service_name : ("MVC_App" or "MVC_App.BlazorClient")	All matching events, including framework, SQL, and application logs. Not a request count.
2. Which MVC API requests completed?	service_name : "MVC_App" and scope_name : "MVC_App.Middleware.RequestLoggingMiddleware" and log_attributes.http.request.is_api : true	Use this as the base filter for API request/status panels. Escaping exceptions can produce a last-chance failure record from the same middleware.
3. Which API requests returned 5xx?	service_name : "MVC_App" and scope_name : "MVC_App.Middleware.RequestLoggingMiddleware" and log_attributes.http.request.is_api : true and log_attributes.http.response.status_code >= 500	Includes 500 and 503; does not rely on one error-category flag.
4. Which API requests were unauthorized or forbidden?	service_name : "MVC_App" and scope_name : "MVC_App.Middleware.RequestLoggingMiddleware" and log_attributes.http.request.is_api : true and log_attributes.http.response.status_code : (401 or 403)	Includes challenges that did not throw an exception.
5. Which exceptions were actually recorded?	log_attributes.error.type : *	Application-classified exception events. Not every framework/SQL exception record necessarily has this custom field.
6. Where are domain conflicts occurring?	service_name : "MVC_App" and scope_name : "MVC_App.Middleware.ExceptionHandlingMiddleware" and log_attributes.error.category : "conflict"	Uses the exception event rather than counting both the exception and completion logs.
7. Which 404s were unmatched routes?	scope_name : "MVC_App.Middleware.RequestLoggingMiddleware" and log_attributes.http.response.status_code : 404 and log_attributes.http.route_matched : false	The URL did not select an endpoint.
8. Which 404s occurred on matched routes?	scope_name : "MVC_App.Middleware.RequestLoggingMiddleware" and log_attributes.http.response.status_code : 404 and log_attributes.http.route_matched : true	Inspect the associated exception. Missing data and deliberately injected errors can both appear here.
9. What happened during one trace?	trace_id : "REPLACE_WITH_THE_REPORTED_TRACE_ID"	Replace the placeholder with a real trace ID from a problem response or log, then sort by timestamp ascending.
10. Which SQL commands took over 100 ms?	service_name : "MVC_App" and scope_name : "MVC_App.Data.DbCommandLoggingInterceptor" and log_attributes.db.duration_ms > 100	One SQL command is not one HTTP request. This field measures command duration in milliseconds.
11. Which Blazor outbound API calls were slow?	service_name : "MVC_App.BlazorClient" and log_attributes.http.request.direction : "outbound" and log_attributes.http.client.duration_ms > 1000	Calls taking over one second, including slow failures. This is client-observed duration, not server-only execution time.
12. Which outbound API calls were unavailable?	service_name : "MVC_App.BlazorClient" and log_attributes.http.request.direction : "outbound" and log_attributes.error.category : "service_unavailable"	Can represent a real HTTP 503 or a transport failure classified as 503 by ApiLoggingHandler.
13. Which Blazor sign-ins failed?	service_name : "MVC_App.BlazorClient" and log_attributes.event.category : "authentication" and log_attributes.event.action : "sign_in" and log_attributes.event.outcome : "failure"	Inspect auth.failure_reason. Treat user.name as personal data, not a public dashboard label.
14. What was logged for a deliberate 503 test?	service_name : "MVC_App" and log_attributes.url.path : "/api/diagnostics/unavailable"	Useful for checking a known test case. Multiple events can describe the same request.

resource_attributes.service.name is also retained, but the pipeline makes service_name convenient. A bare error.type is not the same field as log_attributes.error.type. The top-level log correlation key is trace_id; trace data-stream schemas can differ, so inspect their mappings before attempting cross-view searches.

Use numeric status filtering for all 5xx. error.is_server_error is based on the exact mapped category; 503 uses service_unavailable. A severity Error filter also omits expected 400/404/409 failures logged as Warning and can include non-request failures. Exception existence, HTTP failure and severity are related but different questions.

For an HTTP completion dashboard, filter to scope_name : "MVC_App.Middleware.RequestLoggingMiddleware" before counting requests. Otherwise SQL, framework, outbound-call and exception events inflate the count. Keep domain-error dashboards separate from final wire-status dashboards, especially for MVC redirects. Prefer framework HTTP duration metrics or suitable span measurements for request latency; the custom request logs do not uniformly provide a duration field.

Useful panels include completion counts by status, domain errors by category/type, failures by service, unmatched routes, and slow SQL command logs. Traces and metrics require their own discovered data streams/views. This stack does not guarantee that raw OTel data streams automatically populate every Elastic APM UI feature.

7.3 Turn the queries into a dashboard

1. Open Dashboards, create a dashboard, and add a Lens visualization using mvcapp-logs-*.
2. Apply the appropriate query as a panel-specific filter. Do not put an MVC-only filter on the whole dashboard if other panels should display Blazor or SQL events.
3. Select the chart, metric and breakdown from the table below. Save the panel with a descriptive title, repeat for the other panels, then save the dashboard.

Panel	Filter / Chart Setup	Caution
API Request Volume by Status	Query 2; stacked bar or area chart, date histogram on @timestamp, document Count, breakdown by log_attributes.http.response.status_code.	Label as requests per time bucket, not requests per second unless you normalize the rate.
API 5xx Percentage	Query 2 as the panel filter; Metric visualization with the Lens formula below; format as Percent.	Do not use Query 3 as the panel filter. That would also restrict the denominator to failures. Empty windows have no meaningful percentage.
Top Domain Exception Types	Filter service_name to MVC_App and scope_name to MVC_App.Middleware.ExceptionHandlingMiddleware; horizontal bars, Count by log_attributes.error.type.	Keep exceptions separate from final HTTP completion records.
SQL Command Duration	Filter to MVC_App.Data.DbCommandLoggingInterceptor; time chart using the 95th percentile of log_attributes.db.duration_ms.	Do not apply the over-100-ms threshold if you want the percentile of all commands.
Blazor Outbound Latency	Filter to MVC_App.BlazorClient and http.request.direction=outbound; 95th percentile of log_attributes.http.client.duration_ms over time.	Do not apply the over-1000-ms threshold to a whole-population latency chart.
Unmatched API Routes	Query 7 plus log_attributes.http.request.is_api : true; table grouped by log_attributes.url.path, with Count.	Raw paths can have high cardinality; limit the displayed top values.

The following is a Lens formula, not a standalone query for the Discover KQL bar. It uses KQL only inside the numerator's filter:

count(kql='log_attributes.http.response.status_code >= 500') / count()

A field can be searchable without being suitable for a Terms breakdown. If that happens, look for an aggregatable keyword subfield, such as log_attributes.error.type.keyword, and verify that it exists in your mapping. Duration fields must be numeric for percentiles. Keep the data source in mind too: this log view contains MVC and Blazor events, not the PostgreSQL console's Azure Monitor records.

8. Run the solution and generate some telemetry

With the collection path clear, it's time to put traffic through it. You'll need Docker, the .NET 10 SDK and your own database connections. Use a training database: several optional scenarios deliberately insert, update and delete rows. Start with the diagnostic requests and read-only workloads before trying those write paths.

8.1 Check prerequisites and build

The commands use PowerShell 7.1 or later (pwsh) so credentials can be entered with masked input. Run them from the solution root; adjust the first path to match your checkout. Start Docker Desktop with Linux containers, and make sure you can reach SQL Server and a Company database matching CompanyContext. Compose starts the telemetry services, not the databases.

Set-Location 'D:\Azure_Adv_Training\MVC_App'
dotnet --list-sdks
docker version
docker compose version
dotnet restore MVC_App.sln
dotnet build MVC_App.sln --configuration Debug
dotnet dev-certs https --trust

Check that a .NET 10 SDK is installed and each command succeeds before continuing. Trust the development certificate when the OS prompts. Prefer this over bypassing certificate verification.

8.2 Start Elasticsearch, Kibana, Collector and Logstash

docker compose config --quiet
docker compose up -d
docker compose ps
docker compose logs --tail 100 elasticsearch otel-collector logstash
Invoke-RestMethod 'http://localhost:9200/_cluster/health'
Invoke-RestMethod 'http://localhost:9200/_cat/indices?v'
Invoke-RestMethod 'http://localhost:9200/_data_stream'

Wait for Elasticsearch health and for Kibana at http://localhost:5601 to become available. Before the first exports, application log indices/data streams may not exist. The Collector accepts telemetry on http://localhost:4318; it is not a browser dashboard. Ports 9200, 5601 and 4318 must be free on the host.

8.3 Configure and start MVC_App in terminal First Instance

Choose a private API key and enter the two SQL connection strings below. Keep this terminal open: the environment settings belong to this shell and the processes it starts. Another terminal—or an already-running Visual Studio instance—won't inherit them. Masked input keeps the values out of the visible command text, although they still exist in process memory.

$env:ConnectionStrings__CompanyContext = Read-Host 'Company SQL connection string' -MaskInput
$env:ConnectionStrings__IdentityConnection = Read-Host 'Identity SQL connection string' -MaskInput
$env:Api__Keys__0 = Read-Host 'Private API key shared with the clients' -MaskInput
$env:Api__HeaderName = 'X-Api-Key'
$env:Telemetry__Enabled = 'true'
$env:OTEL_EXPORTER_OTLP_ENDPOINT = 'http://localhost:4318'
$env:OTEL_SERVICE_NAME = 'MVC_App'
$env:OTEL_METRIC_EXPORT_INTERVAL = '10000'

Identity setup, only when needed: use the existing Identity migrations against the intended training database. This step changes its schema and requires suitable database permissions. It does not create the Department/Employee business schema. No CompanyContext migration is supplied for provisioning that database-first schema. Inspect the target connection before applying migrations.

dotnet ef --version
# If dotnet-ef is not installed, install the version matching this project's EF tooling:
dotnet tool install --global dotnet-ef --version 10.0.12
# Run the update only after verifying the target Identity database:
dotnet ef database update --context IdentityContext --project MVC_App --startup-project MVC_App

dotnet run --project MVC_App --launch-profile https

The install command is conditional: skip it if a compatible tool is already available. If an incompatible global version is installed, use a matching local tool or deliberately align the tooling rather than blindly upgrading it. Leave MVC_App running. Its HTTPS URL is https://localhost:7119; the same profile also binds http://localhost:5121.

8.4 Configure and start Blazor in terminal Second Instance

In another PowerShell terminal at the solution root, enter the same Identity connection string and API key used above. Blazor accesses the account database directly but accesses Company data through the MVC API.

$env:ConnectionStrings__IdentityConnection = Read-Host 'Same Identity SQL connection string as MVC_App' -MaskInput
$env:Api__Key = Read-Host 'Same private API key as MVC_App' -MaskInput
$env:Api__HeaderName = 'X-Api-Key'
$env:Api__BaseUrl = 'https://localhost:7119'
$env:Telemetry__Enabled = 'true'
$env:OTEL_EXPORTER_OTLP_ENDPOINT = 'http://localhost:4318'
$env:OTEL_SERVICE_NAME = 'MVC_App.BlazorClient'
$env:OTEL_METRIC_EXPORT_INTERVAL = '10000'
dotnet run --project MVC_App.BlazorClient --launch-profile https

Leave it running and open https://localhost:7213. Its HTTP endpoint is http://localhost:5072. Register a training account at https://localhost:7119/Account/Register, then sign in separately to MVC and Blazor using the same account. The shared database does not make their cookies or login sessions interchangeable. Account registration writes to Identity; CRUD pages write business data.

Visual Studio alternative: configure multiple startup projects with MVC_App and MVC_App.BlazorClient set to Start and select their HTTPS launch profiles. Do not auto-start the console workload projects unless you intend to run their workloads. Supply the same private configuration to the processes launched by Visual Studio; terminal environment settings made after Visual Studio started will not automatically reach them.

8.5 Generate predictable telemetry without business-data writes terminal Third Instance

Keep both servers running. The anonymous ping proves HTTP liveness, not database health. The diagnostic error endpoints deliberately throw exceptions but do not modify Company rows. Use them only in a local/training environment.

$apiBase = 'https://localhost:7119'
Invoke-RestMethod "$apiBase/api/diagnostics/ping"
$headers = @{ 'X-Api-Key' = (Read-Host 'Same private API key as MVC_App' -MaskInput) }
Invoke-RestMethod "$apiBase/api/diagnostics/whoami" -Headers $headers

# No key: expect 401. SkipHttpErrorCheck permits inspection of the expected error.
(Invoke-WebRequest "$apiBase/api/diagnostics/whoami" -SkipHttpErrorCheck).StatusCode

# Known failures, with a valid key so authorization does not mask the intended error:
foreach ($test in @('bad-request', 'not-found', 'conflict', 'server-error', 'unavailable')) {
    $response = Invoke-WebRequest "$apiBase/api/diagnostics/$test" -Headers $headers -SkipHttpErrorCheck
    "{0}: HTTP {1}" -f $test, $response.StatusCode
    $response.Content
}
$headers.Clear()

Expected results are ping/whoami with a key: 200; whoami without a key: 401; bad-request: 400; not-found: 404; conflict: 409; server-error: 500; unavailable: 503. Copy a real traceId from an error response into query 9. Missing/wrong keys cause 401 before the selected diagnostic executes. Wait for exporter/batch/index-refresh delays, then use the Kibana queries in section 7.

8.6 Run the API console client  optional controlled workload

The default scenario and mutation load create/update/delete Company data. Use a disposable training database, understand the generated IDs and avoid overlapping runs. Start with the read-only load below; its POST/PUT/DELETE counts are explicitly zero. The normal scenario command is a separate, optional write-producing test.

$env:MVCAPP_API_BASEURL = 'https://localhost:7119'
$env:MVCAPP_API_KEY = Read-Host 'Same private API key as MVC_App' -MaskInput
dotnet run --project MVC_App.ApiClient -- --help

# Read-only HTTP load against the configured training API:
dotnet run --project MVC_App.ApiClient -- --load --gets 20 --posts 0 --puts 0 --deletes 0 --concurrency 2 --progress 10

# Optional: full scenario, including database writes and intentional failures:
dotnet run --project MVC_App.ApiClient -- --iterations 1 --delay 250
Remove-Item Env:MVCAPP_API_KEY

This console does not export telemetry itself. Observe the resulting MVC server events in Kibana and use the console's returned trace IDs for correlation. Its local latency report is not an OTel metric stream.

 8.8 Verify ingestion and stop safely

docker compose logs --tail 100 otel-collector logstash
Get-Item observability/otel-output/logs*.jsonl* | Select-Object Name, Length, LastWriteTime
Invoke-RestMethod 'http://localhost:9200/_cat/indices/mvcapp-logs-*?v'
Invoke-RestMethod 'http://localhost:9200/_data_stream'

# First stop the web processes with Ctrl+C in their terminals; allow export to finish.
# Stop infrastructure without removing containers or the sincedb volume:
docker compose stop

Do not publish the runtime JSONL files: they may contain personal data or SQL text. Close the configuration terminals after use to remove their process-scoped secrets. To resume the stopped stack, use docker compose start, then restart the web processes.

8.9 Troubleshooting checklist

1. No web process starts: inspect the build output, startup exception, SDK and occupied HTTP ports. A successful Compose start does not launch either application.
2. Login or CRUD fails: verify each database connection, SQL permissions, the Identity migration and the existing Company tables. Anonymous ping can succeed while the databases are unavailable.
3. All API calls return 401: verify the private key and X-Api-Key header match between MVC and the caller; a valid Blazor login cookie is not an API key.
4. No logs.jsonl growth: inspect the exporting process's effective endpoint/enable settings and Collector logs. Host processes use localhost:4318; a container cannot use its own localhost to reach another container.
5. The file grows but no log index appears: inspect Logstash file access, offsets and parsing/mapping errors. Do not reset sincedb as the first troubleshooting step; replay can create duplicates.
6. Indices exist but Discover is empty: check the data view, time range, query language and current filters. Use service_name rather than an unprefixed service.name; application fields are under log_attributes.
7. A query field is unavailable: generate the relevant event first and refresh the data view's field information if needed. Do not assume a log-only field exists in the trace/metric data streams.

The smoke test answers a narrow but useful question: does the web host send all three signals when enabled, and stop sending them when disabled or unconfigured? It uses temporary loopback ports and checks nonempty protobuf payloads and service identity without database writes. You still need to check the Collector, Logstash and Elasticsearch to verify the complete ingestion path.

Do not use docker compose down -v casually: it removes the named sincedb volume and can cause retained log files to be reprocessed later. There is no configured retention lifecycle for the Elasticsearch daily log indices, so the file exporter's three-day rotation does not define Elasticsearch retention.

Let's have a closer look at the code

With the request flow and local setup in mind, we can look at how the pieces are implemented. Start with the MVC host and telemetry registration, then follow the clients and supporting configuration. The notes beside each file call out the behavior worth noticing as you read.

MVC_App

The backend is a useful starting point because it receives the API calls, applies the business rules and talks to SQL Server. Notice how much of the logging happens around the controllers rather than inside each action.

MVC_App/appsettings.Development.json

These overrides enable detailed errors and point the web app at the local Collector with its own service name. One setting deserves care: the SDK does not necessarily read OTEL_METRIC_EXPORT_INTERVAL from JSON. Set it in the process environment unless you've added explicit options binding.

{
  "DetailedErrors": true,
  "OTEL_SERVICE_NAME": "MVC_App",
  "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4318",
  "OTEL_METRIC_EXPORT_INTERVAL": 10000,
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "Api": {
    "HeaderName": "X-Api-Key",
    "Keys": [
      "<REDACTED_API_KEY>"
    ]
  }
}

MVC_App/appsettings.json

The base settings bring together logging levels, allowed hosts, both SQL connections and the API-key configuration. Supply private values for the redacted entries. The OTLP endpoint comes from the development settings or environment rather than this file.

{
  "ConnectionStrings": {
    "CompanyContext": "<REDACTED_CONNECTION_STRING>",
    "IdentityConnection": "<REDACTED_CONNECTION_STRING>"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "Api": {
    "HeaderName": "X-Api-Key",
    "Keys": [
      "<REDACTED_API_KEY>"
    ]
  }
}

MVC_App/Authentication/ApiKeyAuthentication.cs

The API needs a 401 response when credentials are missing, not a redirect to an HTML login page. This handler reads the configured header, checks the allowed keys and returns JSON challenges or forbids. A valid key creates the caller's claims. The comparison is ordinal and the key represents an API client, not the browser user's delegated identity.

using System.Security.Claims;
using System.Text.Encodings.Web;
using Microsoft.AspNetCore.Authentication;
using Microsoft.Extensions.Options;
using MVC_App.Diagnostics;
using MVC_App.Middleware;

namespace MVC_App.Authentication;

public static class ApiKeyAuthenticationDefaults
{
    public const string Scheme = "ApiKey";
    public const string HeaderName = "X-Api-Key";
}

public sealed class ApiKeyAuthenticationOptions : AuthenticationSchemeOptions
{
    /// <summary>The header the key is read from. Overridable mostly so tests can vary it.</summary>
    public string HeaderName { get; set; } = ApiKeyAuthenticationDefaults.HeaderName;

    /// <summary>
    /// The accepted keys, loaded from configuration (<c>Api:Keys</c>). An empty collection means
    /// no caller can authenticate, which surfaces as a 401 rather than as an open door.
    /// </summary>
    public IList<string> ValidKeys { get; } = new List<string>();
}

/// <summary>
/// Minimal header-based scheme for the JSON API, so a non-interactive caller (the
/// MVC_App.ApiClient console app) can authenticate without the cookie/login round trip that the
/// MVC views use. A missing or unrecognised key fails authentication, which the challenge below
/// turns into a 401 with a JSON body, logged with <c>error.category : "unauthorized"</c>.
/// </summary>
public sealed class ApiKeyAuthenticationHandler : AuthenticationHandler<ApiKeyAuthenticationOptions>
{
    public ApiKeyAuthenticationHandler(
        IOptionsMonitor<ApiKeyAuthenticationOptions> options,
        ILoggerFactory logger,
        UrlEncoder encoder)
        : base(options, logger, encoder)
    {
    }

    protected override Task<AuthenticateResult> HandleAuthenticateAsync()
    {
        if (!Request.Headers.TryGetValue(Options.HeaderName, out var suppliedKey) ||
            string.IsNullOrWhiteSpace(suppliedKey))
        {
            // NoResult rather than Fail: the cookie-authenticated browser also hits endpoints
            // that allow anonymous access, and a hard failure here would mask those.
            return Task.FromResult(AuthenticateResult.NoResult());
        }

        var key = suppliedKey.ToString();
        if (!Options.ValidKeys.Contains(key, StringComparer.Ordinal))
        {
            Logger.LogWarning(
                "API key authentication failed for {Method} {Path}: supplied key was not recognised [{ErrorCategory}]",
                Request.Method, Request.Path.Value, ErrorConditions.Unauthorized);
            return Task.FromResult(AuthenticateResult.Fail("The supplied API key was not recognised."));
        }

        var claims = new[]
        {
            new Claim(ClaimTypes.Name, "api-client"),
            new Claim(ClaimTypes.NameIdentifier, "api-client"),
            new Claim("auth_method", ApiKeyAuthenticationDefaults.Scheme)
        };

        var identity = new ClaimsIdentity(claims, Scheme.Name);
        var ticket = new AuthenticationTicket(new ClaimsPrincipal(identity), Scheme.Name);
        return Task.FromResult(AuthenticateResult.Success(ticket));
    }

    /// <summary>
    /// Writes a 401 as JSON instead of redirecting to the login page. The status code is what
    /// <see cref="RequestLoggingMiddleware"/> records, so this is the single place the API's
    /// "unauthorized" condition originates.
    /// </summary>
    protected override async Task HandleChallengeAsync(AuthenticationProperties properties)
    {
        Response.StatusCode = StatusCodes.Status401Unauthorized;
        Response.Headers.WWWAuthenticate = $"{Scheme.Name} header=\"{Options.HeaderName}\"";

        await ApiProblemDetails.WriteAsync(
            Context,
            StatusCodes.Status401Unauthorized,
            title: "Unauthorized",
            detail: $"Supply a valid API key in the '{Options.HeaderName}' request header.");
    }

    protected override async Task HandleForbiddenAsync(AuthenticationProperties properties)
    {
        Response.StatusCode = StatusCodes.Status403Forbidden;

        await ApiProblemDetails.WriteAsync(
            Context,
            StatusCodes.Status403Forbidden,
            title: "Forbidden",
            detail: "The supplied API key is valid but is not permitted to perform this operation.");
    }
}

public static class ApiKeyAuthenticationExtensions
{
    /// <summary>
    /// Registers the API key scheme alongside Identity's cookie scheme. Identity stays the
    /// default for the MVC views; API controllers opt in explicitly with
    /// <c>[Authorize(AuthenticationSchemes = ApiKeyAuthenticationDefaults.Scheme)]</c>.
    /// </summary>
    public static AuthenticationBuilder AddApiKeyAuthentication(
        this AuthenticationBuilder builder, IConfiguration configuration)
    {
        return builder.AddScheme<ApiKeyAuthenticationOptions, ApiKeyAuthenticationHandler>(
            ApiKeyAuthenticationDefaults.Scheme,
            options =>
            {
                options.HeaderName = configuration["Api:HeaderName"] ?? ApiKeyAuthenticationDefaults.HeaderName;

                foreach (var key in configuration.GetSection("Api:Keys").Get<string[]>() ?? [])
                {
                    if (!string.IsNullOrWhiteSpace(key))
                    {
                        options.ValidKeys.Add(key);
                    }
                }
            });
    }
}

MVC_App/Controllers/AccountController.cs

Registration and sign-in use the separate Identity database, so their SQL operations are observed too. Return URLs are checked before redirecting, and logout requires authentication and antiforgery validation. Those safeguards matter even though the focus of this sample is telemetry.

using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Identity;
using Microsoft.AspNetCore.Mvc;
using MVC_App.Models.Identity;
using MVC_App.ViewModels;

namespace MVC_App.Controllers;

public class AccountController : Controller
{
    private readonly UserManager<ApplicationUser> _userManager;
    private readonly SignInManager<ApplicationUser> _signInManager;
    private readonly ILogger<AccountController> _logger;

    public AccountController(
        UserManager<ApplicationUser> userManager,
        SignInManager<ApplicationUser> signInManager,
        ILogger<AccountController> logger)
    {
        _userManager = userManager;
        _signInManager = signInManager;
        _logger = logger;
    }

    [HttpGet]
    public IActionResult Register(string? returnUrl = null)
    {
        ViewData["ReturnUrl"] = returnUrl;
        return View(new RegisterViewModel());
    }

    [HttpPost]
    [ValidateAntiForgeryToken]
    public async Task<IActionResult> Register(RegisterViewModel model, string? returnUrl = null)
    {
        ViewData["ReturnUrl"] = returnUrl;
        if (!ModelState.IsValid)
        {
            return View(model);
        }

        var user = new ApplicationUser { UserName = model.Email, Email = model.Email };
        var result = await _userManager.CreateAsync(user, model.Password);

        if (result.Succeeded)
        {
            _logger.LogInformation("New user account created for {Email}", model.Email);
            await _signInManager.SignInAsync(user, isPersistent: false);
            return RedirectToLocalOrHome(returnUrl);
        }

        foreach (var error in result.Errors)
        {
            ModelState.AddModelError(string.Empty, error.Description);
        }

        return View(model);
    }

    [HttpGet]
    public IActionResult Login(string? returnUrl = null)
    {
        ViewData["ReturnUrl"] = returnUrl;
        return View(new LoginViewModel());
    }

    [HttpPost]
    [ValidateAntiForgeryToken]
    public async Task<IActionResult> Login(LoginViewModel model, string? returnUrl = null)
    {
        ViewData["ReturnUrl"] = returnUrl;
        if (!ModelState.IsValid)
        {
            return View(model);
        }

        var result = await _signInManager.PasswordSignInAsync(
            model.Email, model.Password, model.RememberMe, lockoutOnFailure: false);

        if (result.Succeeded)
        {
            return RedirectToLocalOrHome(returnUrl);
        }

        ModelState.AddModelError(string.Empty, "Invalid login attempt.");
        return View(model);
    }

    [HttpPost]
    [ValidateAntiForgeryToken]
    [Authorize]
    public async Task<IActionResult> Logout()
    {
        await _signInManager.SignOutAsync();
        return RedirectToPage("/Index");
    }

    [HttpGet]
    public IActionResult AccessDenied()
    {
        return View();
    }

    private IActionResult RedirectToLocalOrHome(string? returnUrl)
    {
        if (!string.IsNullOrEmpty(returnUrl) && Url.IsLocalUrl(returnUrl))
        {
            return Redirect(returnUrl);
        }

        return RedirectToPage("/Index");
    }
}

MVC_App/Controllers/Api/ApiControllerBase.cs

The shared API behavior lives here: API-key authentication, explicit model validation and the simulate query option used for test failures. ThrowIfInvalid sends validation errors through the domain-exception path. Keep the fault-injection option restricted to a controlled environment; it is there to exercise the telemetry, not to serve normal clients.

using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
using MVC_App.Authentication;
using MVC_App.Exceptions;

namespace MVC_App.Controllers.Api;

/// <summary>
/// Shared base for the JSON API controllers. Fixes three things in one place: the API key
/// authentication scheme (so an unauthenticated call is a 401 and not a redirect to the login
/// page), the validation-failure path (so a bad body is a 400 with the same problem shape as
/// every other error), and the <c>?simulate=</c> switch used to generate each error condition
/// on demand for the ELK dashboard.
/// </summary>
[ApiController]
[Authorize(AuthenticationSchemes = ApiKeyAuthenticationDefaults.Scheme)]
[Produces("application/json")]
public abstract class ApiControllerBase : ControllerBase
{
    /// <summary>
    /// Throws the <see cref="AppException"/> matching a <c>?simulate=</c> query value, so each
    /// error condition can be produced deliberately against a real endpoint and real route.
    /// Returns without doing anything when the parameter is absent or unrecognised.
    ///
    /// Supported values: <c>401</c>/<c>unauthorized</c>, <c>403</c>/<c>forbidden</c>,
    /// <c>404</c>/<c>notfound</c>, <c>409</c>/<c>conflict</c>, <c>400</c>/<c>validation</c>,
    /// <c>500</c>/<c>servererror</c>, <c>503</c>/<c>unavailable</c>.
    /// </summary>
    protected void ApplySimulatedFault(string? simulate, string resourceName)
    {
        if (string.IsNullOrWhiteSpace(simulate))
        {
            return;
        }

        switch (simulate.Trim().ToLowerInvariant())
        {
            case "400":
            case "validation":
                throw new ValidationFailedException(
                    $"Simulated validation failure on {resourceName}, requested via ?simulate={simulate}.");

            case "401":
            case "unauthorized":
                throw new UnauthorizedRequestException(
                    $"simulated authentication failure on {resourceName}, requested via ?simulate={simulate}.");

            case "403":
            case "forbidden":
                throw new ForbiddenRequestException(
                    $"simulated authorization failure on {resourceName}, requested via ?simulate={simulate}.");

            case "404":
            case "notfound":
            case "not_found":
                throw new ResourceNotFoundException($"{resourceName} (simulated via ?simulate={simulate})");

            case "409":
            case "conflict":
                throw new DuplicatePrimaryKeyException(resourceName, $"simulated-{simulate}");

            case "500":
            case "servererror":
            case "server_error":
                throw new ServerSideException(
                    $"Simulated server-side failure on {resourceName}, requested via ?simulate={simulate}.",
                    new InvalidOperationException("Simulated downstream fault."));

            case "503":
            case "unavailable":
                throw new DependencyUnavailableException($"{resourceName}-store");
        }
    }

    /// <summary>
    /// Converts a failed <see cref="ControllerBase.ModelState"/> into a
    /// <see cref="ValidationFailedException"/> so a 400 flows through the same middleware, log
    /// fields, and JSON shape as every other error condition. Without this, ApiController's
    /// automatic 400 would short-circuit before the app's own logging ran.
    /// </summary>
    protected void ThrowIfInvalid(string entityName)
    {
        if (ModelState.IsValid)
        {
            return;
        }

        var errors = ModelState
            .Where(entry => entry.Value?.Errors.Count > 0)
            .SelectMany(entry => entry.Value!.Errors.Select(error =>
                $"{entry.Key}: {(string.IsNullOrEmpty(error.ErrorMessage) ? "invalid value" : error.ErrorMessage)}"))
            .ToList();

        throw new ValidationFailedException(entityName, errors);
    }
}

MVC_App/Controllers/Api/DepartmentsApiController.cs

Department operations provide several useful paths to observe: a successful create, a duplicate name, a missing number and an attempted delete while employees still refer to the department. The controller applies those checks and uses CompanyContext for persistence. The surrounding instrumentation produces request spans, SQL spans and command logs without telemetry calls in every action.

using Microsoft.AspNetCore.Mvc;
using Microsoft.EntityFrameworkCore;
using MVC_App.Diagnostics;
using MVC_App.Exceptions;
using MVC_App.Models;
using MVC_App.Models.Api;

namespace MVC_App.Controllers.Api;

/// <summary>
/// CRUD over Department as JSON. Every action documents, and actually produces, the status codes
/// the ELK dashboard charts: 200, 201, 204, 400, 401, 404, 409, 500 and 503.
/// </summary>
[Route("api/departments")]
public sealed class DepartmentsApiController : ApiControllerBase
{
    private const string ResourceName = "Department";

    private readonly CompanyContext _context;
    private readonly ILogger<DepartmentsApiController> _logger;

    public DepartmentsApiController(CompanyContext context, ILogger<DepartmentsApiController> logger)
    {
        _context = context;
        _logger = logger;
    }

    // GET: api/departments
    [HttpGet]
    [ProducesResponseType(typeof(IEnumerable<DepartmentDto>), StatusCodes.Status200OK)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status500InternalServerError)]
    public async Task<ActionResult<IEnumerable<DepartmentDto>>> GetAll([FromQuery] string? simulate = null)
    {
        ApplySimulatedFault(simulate, ResourceName);

        var departments = await _context.Departments
            .OrderBy(d => d.DeptNo)
            .ToListAsync();

        _logger.LogInformation("Returned {Count} departments [{ErrorCategory}]",
            departments.Count, ErrorConditions.None);

        return Ok(departments.Select(DepartmentDto.FromEntity).ToList());
    }

    // GET: api/departments/5
    [HttpGet("{id:int}")]
    [ProducesResponseType(typeof(DepartmentDto), StatusCodes.Status200OK)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status404NotFound)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status500InternalServerError)]
    public async Task<ActionResult<DepartmentDto>> GetById(int id, [FromQuery] string? simulate = null)
    {
        ApplySimulatedFault(simulate, ResourceName);

        var department = await _context.Departments.FirstOrDefaultAsync(d => d.DeptNo == id);
        if (department is null)
        {
            // 404: the not-found condition, logged with error.category "not_found".
            throw new PrimaryKeyNotFoundException(ResourceName, id);
        }

        return Ok(DepartmentDto.FromEntity(department));
    }

    // GET: api/departments/by-name/Sales
    [HttpGet("by-name/{deptName}")]
    [ProducesResponseType(typeof(DepartmentDto), StatusCodes.Status200OK)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status404NotFound)]
    public async Task<ActionResult<DepartmentDto>> GetByName(string deptName, [FromQuery] string? simulate = null)
    {
        ApplySimulatedFault(simulate, ResourceName);

        var department = await _context.Departments.FirstOrDefaultAsync(d => d.DeptName == deptName);
        if (department is null)
        {
            throw new DeptNameNotFoundException(deptName);
        }

        return Ok(DepartmentDto.FromEntity(department));
    }

    // POST: api/departments
    [HttpPost]
    [ProducesResponseType(typeof(DepartmentDto), StatusCodes.Status201Created)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status400BadRequest)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status409Conflict)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status500InternalServerError)]
    public async Task<ActionResult<DepartmentDto>> Create(
        [FromBody] DepartmentDto department, [FromQuery] string? simulate = null)
    {
        ApplySimulatedFault(simulate, ResourceName);
        ThrowIfInvalid(ResourceName);

        if (await _context.Departments.AnyAsync(d => d.DeptNo == department.DeptNo))
        {
            throw new DuplicatePrimaryKeyException(ResourceName, department.DeptNo);
        }

        if (await _context.Departments.AnyAsync(d => d.DeptName == department.DeptName))
        {
            throw new DuplicateDeptNameException(department.DeptName);
        }

        _context.Departments.Add(department.ToEntity());
        await _context.SaveChangesAsync();

        _logger.LogInformation("Created department {DeptNo} ({DeptName})",
            department.DeptNo, department.DeptName);

        // 201 with a Location header pointing at GetById.
        return CreatedAtAction(nameof(GetById), new { id = department.DeptNo }, department);
    }

    // PUT: api/departments/5
    [HttpPut("{id:int}")]
    [ProducesResponseType(typeof(DepartmentDto), StatusCodes.Status200OK)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status400BadRequest)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status404NotFound)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status409Conflict)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status500InternalServerError)]
    public async Task<ActionResult<DepartmentDto>> Update(
        int id, [FromBody] DepartmentDto department, [FromQuery] string? simulate = null)
    {
        ApplySimulatedFault(simulate, ResourceName);
        ThrowIfInvalid(ResourceName);

        if (id != department.DeptNo)
        {
            throw new ValidationFailedException(
                $"Route id '{id}' does not match DeptNo '{department.DeptNo}' in the request body.");
        }

        var existing = await _context.Departments.FirstOrDefaultAsync(d => d.DeptNo == id);
        if (existing is null)
        {
            throw new PrimaryKeyNotFoundException(ResourceName, id);
        }

        if (await _context.Departments.AnyAsync(d => d.DeptName == department.DeptName && d.DeptNo != id))
        {
            throw new DuplicateDeptNameException(department.DeptName);
        }

        existing.DeptName = department.DeptName;
        existing.Location = department.Location;

        try
        {
            await _context.SaveChangesAsync();
        }
        catch (DbUpdateException ex)
        {
            // Reported as 500 rather than letting a raw EF exception escape, so the response
            // shape and the error.category on the log entry stay consistent.
            throw new ServerSideException($"Failed to update {ResourceName} '{id}'.", ex);
        }

        return Ok(DepartmentDto.FromEntity(existing));
    }

    // DELETE: api/departments/5
    [HttpDelete("{id:int}")]
    [ProducesResponseType(StatusCodes.Status204NoContent)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status404NotFound)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status409Conflict)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status500InternalServerError)]
    public async Task<IActionResult> Delete(int id, [FromQuery] string? simulate = null)
    {
        ApplySimulatedFault(simulate, ResourceName);

        var department = await _context.Departments.FirstOrDefaultAsync(d => d.DeptNo == id);
        if (department is null)
        {
            throw new PrimaryKeyNotFoundException(ResourceName, id);
        }

        // Employee.DeptName references Department.DeptName by convention only; there is no foreign
        // key in the database, so without this check the delete succeeds and silently leaves every
        // employee in the department pointing at a name that no longer exists.
        var employeeCount = await _context.Employees.CountAsync(e => e.DeptName == department.DeptName);
        if (employeeCount > 0)
        {
            throw new DepartmentHasEmployeesException(department.DeptName, employeeCount);
        }

        _context.Departments.Remove(department);

        try
        {
            await _context.SaveChangesAsync();
        }
        catch (DbUpdateException ex)
        {
            throw new ServerSideException($"Failed to delete {ResourceName} '{id}'.", ex);
        }

        _logger.LogInformation("Deleted department {DeptNo}", id);
        return NoContent();
    }
}

MVC_App/Controllers/Api/DiagnosticsApiController.cs

Use these endpoints when you want a known error without manipulating business rows. Ping works anonymously; whoami checks authentication; the remaining actions deliberately raise domain errors or an unhandled divide-by-zero. They are handy for validating a dashboard, but should not be exposed as an unrestricted production diagnostic surface.

using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
using MVC_App.Diagnostics;
using MVC_App.Exceptions;

namespace MVC_App.Controllers.Api;

/// <summary>
/// Endpoints whose only job is to produce one specific error condition on demand, so the ELK
/// dashboard can be built and demonstrated without having to provoke a real failure in the
/// Department/Employee data. Each one throws the same exception type the real endpoints throw,
/// so the resulting log documents are indistinguishable apart from <c>url.path</c>.
/// </summary>
[Route("api/diagnostics")]
public sealed class DiagnosticsApiController : ApiControllerBase
{
    private const string ResourceName = "Diagnostics";

    private readonly ILogger<DiagnosticsApiController> _logger;

    public DiagnosticsApiController(ILogger<DiagnosticsApiController> logger)
    {
        _logger = logger;
    }

    /// <summary>
    /// Anonymous liveness check, the one endpoint that does not require an API key. Useful as the
    /// "is the app up at all" baseline next to the error panels on a dashboard.
    /// </summary>
    // GET: api/diagnostics/ping
    [HttpGet("ping")]
    [AllowAnonymous]
    [ProducesResponseType(StatusCodes.Status200OK)]
    public IActionResult Ping()
    {
        _logger.LogInformation("Diagnostics ping [{ErrorCategory}]", ErrorConditions.None);
        return Ok(new
        {
            status = "ok",
            service = "MVC_App",
            utc = DateTimeOffset.UtcNow
        });
    }

    /// <summary>Requires a valid API key; confirms the key works before running the error cases.</summary>
    // GET: api/diagnostics/whoami
    [HttpGet("whoami")]
    [ProducesResponseType(StatusCodes.Status200OK)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    public IActionResult WhoAmI() => Ok(new
    {
        name = User.Identity?.Name,
        authenticated = User.Identity?.IsAuthenticated ?? false,
        scheme = User.Identity?.AuthenticationType
    });

    // GET: api/diagnostics/unauthorized  ->  401, error.category "unauthorized"
    [HttpGet("unauthorized")]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    public IActionResult RaiseUnauthorized()
        => throw new UnauthorizedRequestException(
            "the caller's credentials were rejected by the diagnostics endpoint.");

    // GET: api/diagnostics/forbidden  ->  403, error.category "forbidden"
    [HttpGet("forbidden")]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status403Forbidden)]
    public IActionResult RaiseForbidden()
        => throw new ForbiddenRequestException(
            "the caller is authenticated but lacks permission for this operation.");

    // GET: api/diagnostics/not-found  ->  404, error.category "not_found"
    [HttpGet("not-found")]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status404NotFound)]
    public IActionResult RaiseNotFound()
        => throw new ResourceNotFoundException("api/diagnostics/not-found");

    // GET: api/diagnostics/bad-request  ->  400, error.category "validation_error"
    [HttpGet("bad-request")]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status400BadRequest)]
    public IActionResult RaiseBadRequest()
        => throw new ValidationFailedException(
            "The diagnostics endpoint was asked to produce a validation failure.");

    // GET: api/diagnostics/conflict  ->  409, error.category "conflict"
    [HttpGet("conflict")]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status409Conflict)]
    public IActionResult RaiseConflict()
        => throw new DuplicatePrimaryKeyException(ResourceName, "diagnostics-conflict");

    /// <summary>
    /// 500 via a deliberately raised <see cref="ServerSideException"/>, i.e. a failure the app
    /// knew about and classified.
    /// </summary>
    // GET: api/diagnostics/server-error
    [HttpGet("server-error")]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status500InternalServerError)]
    public IActionResult RaiseServerError()
        => throw new ServerSideException(
            "The diagnostics endpoint was asked to produce a server-side failure.",
            new InvalidOperationException("Simulated downstream fault."));

    /// <summary>
    /// 500 via an exception the app never anticipated. Produces the same status code as
    /// <see cref="RaiseServerError"/>, but with <c>error.type : "DivideByZeroException"</c>,
    /// which is what makes the two distinguishable in Kibana.
    /// </summary>
    // GET: api/diagnostics/unhandled
    [HttpGet("unhandled")]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status500InternalServerError)]
    public IActionResult RaiseUnhandled()
    {
        var divisor = 0;
        // Intentional: an exception type the app has no specific handling for, so the
        // "unexpected server error" branch of ExceptionHandlingMiddleware is exercised.
        return Ok(10 / divisor);
    }

    // GET: api/diagnostics/unavailable  ->  503, error.category "service_unavailable"
    [HttpGet("unavailable")]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status503ServiceUnavailable)]
    public IActionResult RaiseUnavailable()
        => throw new DependencyUnavailableException("Company-database");
}

MVC_App/Controllers/Api/EmployeesApiController.cs

The employee API supports CRUD and department filtering, checking that both the requested employee and any referenced department exist. DTOs keep its contract separate from the EF entities. These checks give the shared middleware concrete validation, conflict and not-found cases to report.

using Microsoft.AspNetCore.Mvc;
using Microsoft.EntityFrameworkCore;
using MVC_App.Diagnostics;
using MVC_App.Exceptions;
using MVC_App.Models;
using MVC_App.Models.Api;

namespace MVC_App.Controllers.Api;

/// <summary>
/// CRUD over Employee as JSON, mirroring <see cref="DepartmentsApiController"/> so both resources
/// produce the same set of status codes and the same error-condition fields in ELK.
/// </summary>
[Route("api/employees")]
public sealed class EmployeesApiController : ApiControllerBase
{
    private const string ResourceName = "Employee";

    private readonly CompanyContext _context;
    private readonly ILogger<EmployeesApiController> _logger;

    public EmployeesApiController(CompanyContext context, ILogger<EmployeesApiController> logger)
    {
        _context = context;
        _logger = logger;
    }

    // GET: api/employees?deptName=Sales
    [HttpGet]
    [ProducesResponseType(typeof(IEnumerable<EmployeeDto>), StatusCodes.Status200OK)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status404NotFound)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status500InternalServerError)]
    public async Task<ActionResult<IEnumerable<EmployeeDto>>> GetAll(
        [FromQuery] string? deptName = null, [FromQuery] string? simulate = null)
    {
        ApplySimulatedFault(simulate, ResourceName);

        var query = _context.Employees.AsQueryable();

        if (!string.IsNullOrWhiteSpace(deptName))
        {
            // Filtering by a department that doesn't exist is a not-found condition, not an
            // empty 200, so the dashboard can tell "no such department" from "no employees".
            if (!await _context.Departments.AnyAsync(d => d.DeptName == deptName))
            {
                throw new DeptNameNotFoundException(deptName);
            }

            query = query.Where(e => e.DeptName == deptName);
        }

        var employees = await query.OrderBy(e => e.EmpNo).ToListAsync();

        _logger.LogInformation("Returned {Count} employees for department filter {DeptName} [{ErrorCategory}]",
            employees.Count, deptName ?? "(none)", ErrorConditions.None);

        return Ok(employees.Select(EmployeeDto.FromEntity).ToList());
    }

    // GET: api/employees/5
    [HttpGet("{id:int}")]
    [ProducesResponseType(typeof(EmployeeDto), StatusCodes.Status200OK)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status404NotFound)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status500InternalServerError)]
    public async Task<ActionResult<EmployeeDto>> GetById(int id, [FromQuery] string? simulate = null)
    {
        ApplySimulatedFault(simulate, ResourceName);

        var employee = await _context.Employees.FirstOrDefaultAsync(e => e.EmpNo == id);
        if (employee is null)
        {
            throw new PrimaryKeyNotFoundException(ResourceName, id);
        }

        return Ok(EmployeeDto.FromEntity(employee));
    }

    // POST: api/employees
    [HttpPost]
    [ProducesResponseType(typeof(EmployeeDto), StatusCodes.Status201Created)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status400BadRequest)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status404NotFound)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status409Conflict)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status500InternalServerError)]
    public async Task<ActionResult<EmployeeDto>> Create(
        [FromBody] EmployeeDto employee, [FromQuery] string? simulate = null)
    {
        ApplySimulatedFault(simulate, ResourceName);
        ThrowIfInvalid(ResourceName);

        if (await _context.Employees.AnyAsync(e => e.EmpNo == employee.EmpNo))
        {
            throw new DuplicatePrimaryKeyException(ResourceName, employee.EmpNo);
        }

        if (!await _context.Departments.AnyAsync(d => d.DeptName == employee.DeptName))
        {
            throw new DeptNameNotFoundException(employee.DeptName);
        }

        _context.Employees.Add(employee.ToEntity());
        await _context.SaveChangesAsync();

        _logger.LogInformation("Created employee {EmpNo} ({EmpName})", employee.EmpNo, employee.EmpName);

        return CreatedAtAction(nameof(GetById), new { id = employee.EmpNo }, employee);
    }

    // PUT: api/employees/5
    [HttpPut("{id:int}")]
    [ProducesResponseType(typeof(EmployeeDto), StatusCodes.Status200OK)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status400BadRequest)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status404NotFound)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status500InternalServerError)]
    public async Task<ActionResult<EmployeeDto>> Update(
        int id, [FromBody] EmployeeDto employee, [FromQuery] string? simulate = null)
    {
        ApplySimulatedFault(simulate, ResourceName);
        ThrowIfInvalid(ResourceName);

        if (id != employee.EmpNo)
        {
            throw new ValidationFailedException(
                $"Route id '{id}' does not match EmpNo '{employee.EmpNo}' in the request body.");
        }

        var existing = await _context.Employees.FirstOrDefaultAsync(e => e.EmpNo == id);
        if (existing is null)
        {
            throw new PrimaryKeyNotFoundException(ResourceName, id);
        }

        if (!await _context.Departments.AnyAsync(d => d.DeptName == employee.DeptName))
        {
            throw new DeptNameNotFoundException(employee.DeptName);
        }

        existing.EmpName = employee.EmpName;
        existing.Salary = employee.Salary;
        existing.DeptName = employee.DeptName;
        existing.Designation = employee.Designation;

        try
        {
            await _context.SaveChangesAsync();
        }
        catch (DbUpdateException ex)
        {
            throw new ServerSideException($"Failed to update {ResourceName} '{id}'.", ex);
        }

        return Ok(EmployeeDto.FromEntity(existing));
    }

    // DELETE: api/employees/5
    [HttpDelete("{id:int}")]
    [ProducesResponseType(StatusCodes.Status204NoContent)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status401Unauthorized)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status404NotFound)]
    [ProducesResponseType(typeof(ApiProblemDetails), StatusCodes.Status500InternalServerError)]
    public async Task<IActionResult> Delete(int id, [FromQuery] string? simulate = null)
    {
        ApplySimulatedFault(simulate, ResourceName);

        var employee = await _context.Employees.FirstOrDefaultAsync(e => e.EmpNo == id);
        if (employee is null)
        {
            throw new PrimaryKeyNotFoundException(ResourceName, id);
        }

        _context.Employees.Remove(employee);

        try
        {
            await _context.SaveChangesAsync();
        }
        catch (DbUpdateException ex)
        {
            throw new ServerSideException($"Failed to delete {ResourceName} '{id}'.", ex);
        }

        _logger.LogInformation("Deleted employee {EmpNo}", id);
        return NoContent();
    }
}

MVC_App/Controllers/DepartmentController.cs

This is the browser-facing version of department CRUD, with cookie authorization and five-row pages. Missing records, duplicates and unsafe deletes become domain exceptions. Compare its failure path with the API controller: the browser can be redirected to an error page instead of receiving a JSON problem document.

using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
using Microsoft.EntityFrameworkCore;
using MVC_App.Exceptions;
using MVC_App.Models;

namespace MVC_App.Controllers;

[Authorize]
public class DepartmentController : Controller
{
    private readonly CompanyContext _context;

    public DepartmentController(CompanyContext context)
    {
        _context = context;
    }

    // GET: Department
    public async Task<IActionResult> Index(int? pageNumber)
    {
        var departments = _context.Departments.OrderBy(d => d.DeptNo);
        int pageSize = 5;
        return View(await PaginatedList<Department>.CreateAsync(departments, pageNumber ?? 1, pageSize));
    }

    // GET: Department/Details/5
    public async Task<IActionResult> Details(int? id)
    {
        if (id == null)
        {
            return NotFound();
        }

        var department = await _context.Departments
            .FirstOrDefaultAsync(m => m.DeptNo == id);
        if (department == null)
        {
            throw new PrimaryKeyNotFoundException(nameof(Department), id);
        }

        // Checked on the way in as well as on the POST, so the confirmation page is never shown
        // for a department that cannot actually be deleted.
        await ThrowIfHasEmployeesAsync(department.DeptName);

        return View(department);
    }

    // GET: Department/Create
    public IActionResult Create()
    {
        return View();
    }

    // POST: Department/Create
    [HttpPost]
    [ValidateAntiForgeryToken]
    public async Task<IActionResult> Create([Bind("DeptNo,DeptName,Location")] Department department)
    {
        if (ModelState.IsValid)
        {
            if (DepartmentExists(department.DeptNo))
            {
                throw new DuplicatePrimaryKeyException(nameof(Department), department.DeptNo);
            }

            if (await DepartmentNameExistsAsync(department.DeptName))
            {
                throw new DuplicateDeptNameException(department.DeptName);
            }

            _context.Add(department);
            await _context.SaveChangesAsync();
            return RedirectToAction(nameof(Index));
        }
        return View(department);
    }

    // GET: Department/Edit/5
    public async Task<IActionResult> Edit(int? id)
    {
        if (id == null)
        {
            return NotFound();
        }

        var department = await _context.Departments.FindAsync(id);
        if (department == null)
        {
            throw new PrimaryKeyNotFoundException(nameof(Department), id);
        }
        return View(department);
    }

    // POST: Department/Edit/5
    [HttpPost]
    [ValidateAntiForgeryToken]
    public async Task<IActionResult> Edit(int id, [Bind("DeptNo,DeptName,Location")] Department department)
    {
        if (id != department.DeptNo)
        {
            return NotFound();
        }

        if (ModelState.IsValid)
        {
            if (await DepartmentNameExistsAsync(department.DeptName, excludeDeptNo: department.DeptNo))
            {
                throw new DuplicateDeptNameException(department.DeptName);
            }

            try
            {
                _context.Update(department);
                await _context.SaveChangesAsync();
            }
            catch (DbUpdateConcurrencyException)
            {
                if (!DepartmentExists(department.DeptNo))
                {
                    throw new PrimaryKeyNotFoundException(nameof(Department), department.DeptNo);
                }
                else
                {
                    throw;
                }
            }
            return RedirectToAction(nameof(Index));
        }
        return View(department);
    }

    // GET: Department/Delete/5
    public async Task<IActionResult> Delete(int? id)
    {
        if (id == null)
        {
            return NotFound();
        }

        var department = await _context.Departments
            .FirstOrDefaultAsync(m => m.DeptNo == id);
        if (department == null)
        {
            throw new PrimaryKeyNotFoundException(nameof(Department), id);
        }

        // Checked on the way in as well as on the POST, so the confirmation page is never shown
        // for a department that cannot actually be deleted.
        await ThrowIfHasEmployeesAsync(department.DeptName);

        return View(department);
    }

    // POST: Department/Delete/5
    [HttpPost, ActionName("Delete")]
    [ValidateAntiForgeryToken]
    public async Task<IActionResult> DeleteConfirmed(int id)
    {
        var department = await _context.Departments.FindAsync(id);
        if (department == null)
        {
            throw new PrimaryKeyNotFoundException(nameof(Department), id);
        }

        await ThrowIfHasEmployeesAsync(department.DeptName);

        _context.Departments.Remove(department);
        await _context.SaveChangesAsync();

        return RedirectToAction(nameof(Index));
    }

    /// <summary>
    /// Blocks deleting a department that still has employees. Employee.DeptName references
    /// Department.DeptName by convention only, with no foreign key in the database, so nothing
    /// below this layer stops the delete from orphaning them.
    ///
    /// This controller writes through the DbContext directly rather than calling the API, so the
    /// identical check in DepartmentsApiController.Delete does not cover it and both need it.
    /// </summary>
    private async Task ThrowIfHasEmployeesAsync(string deptName)
    {
        var employeeCount = await _context.Employees.CountAsync(e => e.DeptName == deptName);
        if (employeeCount > 0)
        {
            throw new DepartmentHasEmployeesException(deptName, employeeCount);
        }
    }

    private bool DepartmentExists(int id)
    {
        return _context.Departments.Any(e => e.DeptNo == id);
    }

    private Task<bool> DepartmentNameExistsAsync(string deptName, int? excludeDeptNo = null)
    {
        return _context.Departments
            .Where(d => d.DeptName == deptName && d.DeptNo != (excludeDeptNo ?? -1))
            .AnyAsync();
    }
}

MVC_App/Controllers/EmployeeController.cs

The MVC employee screens get their data and department choices from this controller. Its paged CRUD and ListByDepartment actions share the same SQL instrumentation as the API, so you can investigate browser-driven database work with the same trace and log fields.

using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.Rendering;
using Microsoft.EntityFrameworkCore;
using MVC_App.Exceptions;
using MVC_App.Models;

namespace MVC_App.Controllers;

[Authorize]
public class EmployeeController : Controller
{
    private readonly CompanyContext _context;

    public EmployeeController(CompanyContext context)
    {
        _context = context;
    }

    // GET: Employee
    public async Task<IActionResult> Index(int? pageNumber)
    {
        var employees = _context.Employees.OrderBy(e => e.EmpNo);
        int pageSize = 5;
        return View(await PaginatedList<Employee>.CreateAsync(employees, pageNumber ?? 1, pageSize));
    }

    // GET: Employee/ListByDepartment?deptName=Sales
    public async Task<IActionResult> ListByDepartment(string deptName)
    {
        var departments = await _context.Departments
            .Select(d => d.DeptName)
            .Distinct()
            .OrderBy(d => d)
            .ToListAsync();

        if (!string.IsNullOrEmpty(deptName) && !departments.Contains(deptName))
        {
            throw new DeptNameNotFoundException(deptName);
        }

        ViewBag.Departments = new SelectList(departments, deptName);
        ViewBag.SelectedDeptName = deptName;

        var employees = _context.Employees.AsQueryable();
        if (!string.IsNullOrEmpty(deptName))
        {
            employees = employees.Where(e => e.DeptName == deptName);
        }

        return View(await employees.OrderBy(e => e.EmpName).ToListAsync());
    }

    // GET: Employee/Details/5
    public async Task<IActionResult> Details(int? id)
    {
        if (id == null)
        {
            return NotFound();
        }

        var employee = await _context.Employees
            .FirstOrDefaultAsync(m => m.EmpNo == id);
        if (employee == null)
        {
            throw new PrimaryKeyNotFoundException(nameof(Employee), id);
        }

        return View(employee);
    }

    // GET: Employee/Create
    public async Task<IActionResult> Create()
    {
        await PopulateDepartmentsDropDownList();
        return View();
    }

    // POST: Employee/Create
    [HttpPost]
    [ValidateAntiForgeryToken]
    public async Task<IActionResult> Create([Bind("EmpNo,EmpName,Salary,DeptName,Designation")] Employee employee)
    {
        if (ModelState.IsValid)
        {
            if (EmployeeExists(employee.EmpNo))
            {
                throw new DuplicatePrimaryKeyException(nameof(Employee), employee.EmpNo);
            }

            _context.Add(employee);
            await _context.SaveChangesAsync();
            return RedirectToAction(nameof(Index));
        }
        await PopulateDepartmentsDropDownList(employee.DeptName);
        return View(employee);
    }

    private async Task PopulateDepartmentsDropDownList(object? selectedDept = null)
    {
        var departments = await _context.Departments
            .Select(d => d.DeptName)
            .Distinct()
            .OrderBy(d => d)
            .ToListAsync();
        ViewBag.Departments = new SelectList(departments, selectedDept);
    }

    // GET: Employee/Edit/5
    public async Task<IActionResult> Edit(int? id)
    {
        if (id == null)
        {
            return NotFound();
        }

        var employee = await _context.Employees.FindAsync(id);
        if (employee == null)
        {
            throw new PrimaryKeyNotFoundException(nameof(Employee), id);
        }
        return View(employee);
    }

    // POST: Employee/Edit/5
    [HttpPost]
    [ValidateAntiForgeryToken]
    public async Task<IActionResult> Edit(int id, [Bind("EmpNo,EmpName,Salary,DeptName,Designation")] Employee employee)
    {
        if (id != employee.EmpNo)
        {
            return NotFound();
        }

        if (ModelState.IsValid)
        {
            try
            {
                _context.Update(employee);
                await _context.SaveChangesAsync();
            }
            catch (DbUpdateConcurrencyException)
            {
                if (!EmployeeExists(employee.EmpNo))
                {
                    throw new PrimaryKeyNotFoundException(nameof(Employee), employee.EmpNo);
                }
                else
                {
                    throw;
                }
            }
            return RedirectToAction(nameof(Index));
        }
        return View(employee);
    }

    // GET: Employee/Delete/5
    public async Task<IActionResult> Delete(int? id)
    {
        if (id == null)
        {
            return NotFound();
        }

        var employee = await _context.Employees
            .FirstOrDefaultAsync(m => m.EmpNo == id);
        if (employee == null)
        {
            throw new PrimaryKeyNotFoundException(nameof(Employee), id);
        }

        return View(employee);
    }

    // POST: Employee/Delete/5
    [HttpPost, ActionName("Delete")]
    [ValidateAntiForgeryToken]
    public async Task<IActionResult> DeleteConfirmed(int id)
    {
        var employee = await _context.Employees.FindAsync(id);
        if (employee == null)
        {
            throw new PrimaryKeyNotFoundException(nameof(Employee), id);
        }

        _context.Employees.Remove(employee);
        await _context.SaveChangesAsync();

        return RedirectToAction(nameof(Index));
    }

    private bool EmployeeExists(int id)
    {
        return _context.Employees.Any(e => e.EmpNo == id);
    }
}

MVC_App/Data/DbCommandLoggingInterceptor.cs

The interceptor gives database calls a readable log alongside their trace spans. Its synchronous and asynchronous callbacks record the database, operation, duration and outcome, with command text limited to 1,000 characters in the message. Parameter values are not logged separately, but SQL literals still need a privacy review. Use the trace ID to connect these events to a request; a post-execution callback may run after the SQL Activity has stopped.

using System.Data.Common;
using Microsoft.EntityFrameworkCore.Diagnostics;

namespace MVC_App.Data;

/// <summary>
/// Logs every database command EF Core executes, success or failure, as a structured entry
/// under the <c>MVC_App.Data.DbCommandLoggingInterceptor</c> category. Registered on both
/// CompanyContext and IdentityContext in Program.cs. Runs inside the ambient Activity created
/// by the SqlClient trace instrumentation (see Telemetry/TelemetryExtensions.cs), so each log
/// entry's trace_id/span_id line up with the matching database span.
/// </summary>
public sealed class DbCommandLoggingInterceptor : DbCommandInterceptor
{
    private const int MaxCommandTextLength = 1000;

    private readonly ILogger<DbCommandLoggingInterceptor> _logger;

    public DbCommandLoggingInterceptor(ILogger<DbCommandLoggingInterceptor> logger)
    {
        _logger = logger;
    }

    public override DbDataReader ReaderExecuted(DbCommand command, CommandExecutedEventData eventData, DbDataReader result)
    {
        LogSuccess(command, eventData);
        return base.ReaderExecuted(command, eventData, result);
    }

    public override ValueTask<DbDataReader> ReaderExecutedAsync(
        DbCommand command, CommandExecutedEventData eventData, DbDataReader result, CancellationToken cancellationToken = default)
    {
        LogSuccess(command, eventData);
        return base.ReaderExecutedAsync(command, eventData, result, cancellationToken);
    }

    public override int NonQueryExecuted(DbCommand command, CommandExecutedEventData eventData, int result)
    {
        LogSuccess(command, eventData);
        return base.NonQueryExecuted(command, eventData, result);
    }

    public override ValueTask<int> NonQueryExecutedAsync(
        DbCommand command, CommandExecutedEventData eventData, int result, CancellationToken cancellationToken = default)
    {
        LogSuccess(command, eventData);
        return base.NonQueryExecutedAsync(command, eventData, result, cancellationToken);
    }

    public override object? ScalarExecuted(DbCommand command, CommandExecutedEventData eventData, object? result)
    {
        LogSuccess(command, eventData);
        return base.ScalarExecuted(command, eventData, result);
    }

    public override ValueTask<object?> ScalarExecutedAsync(
        DbCommand command, CommandExecutedEventData eventData, object? result, CancellationToken cancellationToken = default)
    {
        LogSuccess(command, eventData);
        return base.ScalarExecutedAsync(command, eventData, result, cancellationToken);
    }

    public override void CommandFailed(DbCommand command, CommandErrorEventData eventData)
    {
        LogFailure(command, eventData);
        base.CommandFailed(command, eventData);
    }

    public override Task CommandFailedAsync(DbCommand command, CommandErrorEventData eventData, CancellationToken cancellationToken = default)
    {
        LogFailure(command, eventData);
        return base.CommandFailedAsync(command, eventData, cancellationToken);
    }

    private void LogSuccess(DbCommand command, CommandExecutedEventData eventData)
    {
        using var scope = _logger.BeginScope(new Dictionary<string, object?>
        {
            ["db.system"] = "mssql",
            ["db.name"] = command.Connection?.Database,
            ["db.operation"] = eventData.ExecuteMethod.ToString(),
            ["db.duration_ms"] = eventData.Duration.TotalMilliseconds
        });

        _logger.LogInformation(
            "DB call succeeded: {Operation} against {Database} in {DurationMs}ms: {CommandText}",
            eventData.ExecuteMethod, command.Connection?.Database, eventData.Duration.TotalMilliseconds,
            Truncate(command.CommandText));
    }

    private void LogFailure(DbCommand command, CommandErrorEventData eventData)
    {
        using var scope = _logger.BeginScope(new Dictionary<string, object?>
        {
            ["db.system"] = "mssql",
            ["db.name"] = command.Connection?.Database,
            ["db.operation"] = eventData.ExecuteMethod.ToString(),
            ["db.duration_ms"] = eventData.Duration.TotalMilliseconds
        });

        _logger.LogError(
            eventData.Exception,
            "DB call failed: {Operation} against {Database} after {DurationMs}ms: {CommandText}",
            eventData.ExecuteMethod, command.Connection?.Database, eventData.Duration.TotalMilliseconds,
            Truncate(command.CommandText));
    }

    private static string Truncate(string? commandText)
    {
        if (string.IsNullOrEmpty(commandText))
        {
            return string.Empty;
        }

        return commandText.Length <= MaxCommandTextLength
            ? commandText
            : commandText[..MaxCommandTextLength] + "...";
    }
}

MVC_App/Diagnostics/ApiProblemDetails.cs

When a call fails, this response gives the client a status, description and trace ID it can report back. The extra errorCategory and errorType fields make handling more predictable. WriteAsync uses application/problem+json and avoids rewriting a response that has already started.

using System.Diagnostics;
using System.Text.Json;
using System.Text.Json.Serialization;

namespace MVC_App.Diagnostics;

/// <summary>
/// The single JSON error shape every <c>/api/...</c> endpoint returns, modelled on RFC 7807
/// with two extra members the ELK dashboard relies on: <c>errorCategory</c> (the same slug that
/// goes into <c>log_attributes.error.category</c>) and <c>traceId</c>, which matches the
/// <c>trace_id</c> on the corresponding log and trace documents. A caller that gets a failure can
/// paste its <c>traceId</c> straight into Kibana and find the server-side record of it.
/// </summary>
public sealed class ApiProblemDetails
{
    [JsonPropertyName("status")]
    public int Status { get; init; }

    [JsonPropertyName("title")]
    public string Title { get; init; } = string.Empty;

    [JsonPropertyName("detail")]
    public string? Detail { get; init; }

    [JsonPropertyName("instance")]
    public string? Instance { get; init; }

    /// <summary>The ELK filter value, e.g. "unauthorized", "not_found", "server_error".</summary>
    [JsonPropertyName("errorCategory")]
    public string ErrorCategory { get; init; } = ErrorConditions.None;

    /// <summary>The exception's CLR type name, when the failure came from a thrown exception.</summary>
    [JsonPropertyName("errorType")]
    public string? ErrorType { get; init; }

    /// <summary>Correlates this response with the log and trace documents in Elasticsearch.</summary>
    [JsonPropertyName("traceId")]
    public string? TraceId { get; init; }

    [JsonPropertyName("timestamp")]
    public DateTimeOffset Timestamp { get; init; } = DateTimeOffset.UtcNow;

    private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
    {
        DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
    };

    public static ApiProblemDetails Create(
        HttpContext context, int statusCode, string title, string? detail, string? errorType = null)
        => new()
        {
            Status = statusCode,
            Title = title,
            Detail = detail,
            Instance = context.Request.Path.Value,
            ErrorCategory = ErrorConditions.ForStatusCode(statusCode),
            ErrorType = errorType,
            TraceId = Activity.Current?.TraceId.ToString() ?? context.TraceIdentifier
        };

    /// <summary>
    /// Writes the problem document to the response. No-ops when the response has already
    /// started, since the status code and headers can no longer be changed at that point.
    /// </summary>
    public static async Task WriteAsync(
        HttpContext context, int statusCode, string title, string? detail, string? errorType = null)
    {
        if (context.Response.HasStarted)
        {
            return;
        }

        context.Response.StatusCode = statusCode;
        context.Response.ContentType = "application/problem+json";

        var problem = Create(context, statusCode, title, detail, errorType);
        await context.Response.WriteAsync(JsonSerializer.Serialize(problem, SerializerOptions), context.RequestAborted);
    }
}

MVC_App/Diagnostics/ErrorConditions.cs

Stable category names make dashboard filters less fragile than message searches. This file maps status codes to those categories for both logs and problem responses. One useful detail: 503 maps to service_unavailable, so the server_error flag alone won't find every 5xx response.

namespace MVC_App.Diagnostics;

/// <summary>
/// The canonical set of error-condition slugs this app stamps onto every log entry as
/// <c>error.category</c>. They exist so ELK/Kibana can filter and chart on a single stable
/// string instead of re-deriving meaning from raw status codes in every saved search.
/// </summary>
public static class ErrorConditions
{
    /// <summary>2xx/3xx: the request completed as intended.</summary>
    public const string None = "none";

    /// <summary>400: the request itself was malformed or failed model validation.</summary>
    public const string Validation = "validation_error";

    /// <summary>401: the caller is unauthenticated (missing/invalid credentials or API key).</summary>
    public const string Unauthorized = "unauthorized";

    /// <summary>403: the caller is authenticated but not allowed to perform this operation.</summary>
    public const string Forbidden = "forbidden";

    /// <summary>404: the requested route, or the entity behind it, does not exist.</summary>
    public const string NotFound = "not_found";

    /// <summary>409: the write conflicts with data that already exists.</summary>
    public const string Conflict = "conflict";

    /// <summary>Any other 4xx that isn't one of the named conditions above.</summary>
    public const string ClientError = "client_error";

    /// <summary>500: an unhandled or deliberately raised failure inside the app.</summary>
    public const string ServerError = "server_error";

    /// <summary>502/503/504: a dependency the app relies on was unavailable.</summary>
    public const string Unavailable = "service_unavailable";

    /// <summary>Maps an HTTP status code onto the condition slug it should be reported as.</summary>
    public static string ForStatusCode(int statusCode) => statusCode switch
    {
        StatusCodes.Status400BadRequest or StatusCodes.Status422UnprocessableEntity => Validation,
        StatusCodes.Status401Unauthorized => Unauthorized,
        StatusCodes.Status403Forbidden => Forbidden,
        StatusCodes.Status404NotFound or StatusCodes.Status410Gone => NotFound,
        StatusCodes.Status409Conflict => Conflict,
        StatusCodes.Status502BadGateway
            or StatusCodes.Status503ServiceUnavailable
            or StatusCodes.Status504GatewayTimeout => Unavailable,
        >= 500 => ServerError,
        >= 400 => ClientError,
        _ => None
    };

    /// <summary>
    /// True for the status codes that represent a failed request. 3xx redirects and 2xx
    /// successes are not failures, so <c>event.outcome</c> stays "success" for them.
    /// </summary>
    public static bool IsFailure(int statusCode) => statusCode >= 400;
}

MVC_App/Dockerfile

The optional container build publishes with the .NET 10 SDK, then runs MVC_App.dll in an ASP.NET runtime image as APP_UID on port 8080. Compose does not start this image. If you choose to use it, provide its database and telemetry configuration separately.

# Optional application image; compose.yaml runs telemetry services only.
# Build from the solution root: docker build -f MVC_App/Dockerfile -t mvc-app:local .
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY MVC_App/MVC_App.csproj MVC_App/
RUN dotnet restore MVC_App/MVC_App.csproj
COPY MVC_App/ MVC_App/
RUN dotnet publish MVC_App/MVC_App.csproj -c Release -o /app/publish --no-restore /p:UseAppHost=false

FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS final
WORKDIR /app
ENV ASPNETCORE_HTTP_PORTS=8080
EXPOSE 8080
COPY --from=build /app/publish .
USER $APP_UID
ENTRYPOINT ["dotnet", "MVC_App.dll"]

MVC_App/Exceptions/AppExceptions.cs

These exception types give business failures an explicit meaning and HTTP status instead of treating every problem as a generic 500. The middleware uses them to build consistent responses and logs. DepartmentHasEmployeesException is particularly important: it protects a relationship that the database model does not express as a foreign key.

using MVC_App.Diagnostics;

namespace MVC_App.Exceptions;

/// <summary>
/// Base type for domain/validation exceptions that carry the HTTP status code
/// they should be reported as, so a single middleware can log and render them consistently.
/// <see cref="ErrorCategory"/> is the slug ELK filters on (see <see cref="ErrorConditions"/>).
/// </summary>
public abstract class AppException : Exception
{
    protected AppException(string message, int statusCode, Exception? innerException = null)
        : base(message, innerException)
    {
        StatusCode = statusCode;
    }

    public int StatusCode { get; }

    public string ErrorCategory => ErrorConditions.ForStatusCode(StatusCode);
}

// ---------------------------------------------------------------------------
// 404 - not found
// ---------------------------------------------------------------------------

public sealed class PrimaryKeyNotFoundException : AppException
{
    public PrimaryKeyNotFoundException(string entityName, object key)
        : base($"{entityName} with key '{key}' was not found.", StatusCodes.Status404NotFound)
    {
    }
}

public sealed class DeptNameNotFoundException : AppException
{
    public DeptNameNotFoundException(string deptName)
        : base($"Department name '{deptName}' was not found.", StatusCodes.Status404NotFound)
    {
    }
}

/// <summary>
/// A not-found condition that isn't tied to a primary key, e.g. an unknown sub-resource or a
/// route that resolved but has nothing behind it.
/// </summary>
public sealed class ResourceNotFoundException : AppException
{
    public ResourceNotFoundException(string resource)
        : base($"The requested resource '{resource}' was not found.", StatusCodes.Status404NotFound)
    {
    }
}

// ---------------------------------------------------------------------------
// 409 - conflict
// ---------------------------------------------------------------------------

public sealed class DuplicatePrimaryKeyException : AppException
{
    public DuplicatePrimaryKeyException(string entityName, object key)
        : base($"{entityName} with key '{key}' already exists.", StatusCodes.Status409Conflict)
    {
    }
}

public sealed class DuplicateDeptNameException : AppException
{
    public DuplicateDeptNameException(string deptName)
        : base($"Department name '{deptName}' already exists. Department names must be unique.", StatusCodes.Status409Conflict)
    {
    }
}

/// <summary>
/// A department still has employees assigned to it, so deleting it would leave them pointing at a
/// department that no longer exists.
///
/// The database has no foreign key between Employee.DeptName and Department.DeptName, so nothing
/// below this layer prevents the orphaning. The rule is enforced here, and reported as 409 rather
/// than 400 because the request is well formed; it conflicts with data that already exists.
/// </summary>
public sealed class DepartmentHasEmployeesException : AppException
{
    public DepartmentHasEmployeesException(string deptName, int employeeCount)
        : base($"Department '{deptName}' still has {employeeCount} employee(s) assigned to it and cannot be deleted. " +
               "Reassign or remove them first.", StatusCodes.Status409Conflict)
    {
        DeptName = deptName;
        EmployeeCount = employeeCount;
    }

    public string DeptName { get; }

    public int EmployeeCount { get; }
}

// ---------------------------------------------------------------------------
// 400 - validation
// ---------------------------------------------------------------------------

/// <summary>Model binding or business-rule validation failed on an otherwise well-formed request.</summary>
public sealed class ValidationFailedException : AppException
{
    public ValidationFailedException(string message)
        : base(message, StatusCodes.Status400BadRequest)
    {
    }

    public ValidationFailedException(string entityName, IEnumerable<string> errors)
        : base($"{entityName} failed validation: {string.Join("; ", errors)}", StatusCodes.Status400BadRequest)
    {
    }
}

// ---------------------------------------------------------------------------
// 401 / 403 - authentication and authorization
// ---------------------------------------------------------------------------

/// <summary>
/// The caller presented no credentials, or credentials that could not be verified. Reported as
/// 401 and logged with <c>error.category : "unauthorized"</c>.
/// </summary>
public sealed class UnauthorizedRequestException : AppException
{
    public UnauthorizedRequestException(string reason)
        : base($"Unauthorized: {reason}", StatusCodes.Status401Unauthorized)
    {
    }

    public static UnauthorizedRequestException MissingApiKey(string headerName)
        => new($"the '{headerName}' header was not supplied.");

    public static UnauthorizedRequestException InvalidApiKey(string headerName)
        => new($"the '{headerName}' header did not match a known API key.");
}

/// <summary>
/// The caller is authenticated but is not permitted to perform this operation. Reported as 403
/// and logged with <c>error.category : "forbidden"</c>.
/// </summary>
public sealed class ForbiddenRequestException : AppException
{
    public ForbiddenRequestException(string reason)
        : base($"Forbidden: {reason}", StatusCodes.Status403Forbidden)
    {
    }
}

// ---------------------------------------------------------------------------
// 5xx - server-side failures
// ---------------------------------------------------------------------------

/// <summary>
/// An operation failed for a reason that is the server's fault, not the caller's. Reported as
/// 500 and logged with <c>error.category : "server_error"</c>. Prefer this over a bare
/// <see cref="Exception"/> so the failure carries its intended status code through the pipeline.
/// </summary>
public sealed class ServerSideException : AppException
{
    public ServerSideException(string message, Exception? innerException = null)
        : base(message, StatusCodes.Status500InternalServerError, innerException)
    {
    }
}

/// <summary>
/// A dependency the app needs (database, downstream service) could not be reached. Reported as
/// 503 and logged with <c>error.category : "service_unavailable"</c>.
/// </summary>
public sealed class DependencyUnavailableException : AppException
{
    public DependencyUnavailableException(string dependencyName, Exception? innerException = null)
        : base($"The dependency '{dependencyName}' is currently unavailable.",
            StatusCodes.Status503ServiceUnavailable, innerException)
    {
    }
}

MVC_App/Middleware/ExceptionHandlingMiddleware.cs

An API client needs an error response it can parse; a browser user needs a page. This middleware makes that choice, returning problem JSON for API calls and redirecting browser requests for known domain failures. Unexpected API exceptions become sanitized 500 responses, while unexpected browser exceptions go to the environment handler. This explains why a browser failure can have an error log followed by a 302 completion log.

using MVC_App.Diagnostics;
using MVC_App.Exceptions;

namespace MVC_App.Middleware;

/// <summary>
/// Catches domain validation exceptions (<see cref="AppException"/>) close to the endpoint,
/// logs them at a severity ELK can filter/dashboard on, and renders them in the shape the caller
/// expects: a redirect to the shared error page for browser requests, an
/// <see cref="ApiProblemDetails"/> JSON document for <c>/api/...</c> requests.
///
/// Any other exception is logged as an unexpected server error. For API requests it is converted
/// into a 500 problem document so the caller always gets JSON back; for browser requests it is
/// rethrown so the environment's own handler (developer exception page, or app.UseExceptionHandler)
/// still runs.
/// </summary>
public sealed class ExceptionHandlingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<ExceptionHandlingMiddleware> _logger;

    public ExceptionHandlingMiddleware(RequestDelegate next, ILogger<ExceptionHandlingMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        try
        {
            await _next(context);
        }
        catch (AppException appException)
        {
            RequestLogging.LogByStatusCode(_logger, context, appException.StatusCode, appException);

            if (context.Response.HasStarted)
            {
                throw;
            }

            context.Response.Clear();

            if (RequestLogging.IsApiRequest(context))
            {
                await ApiProblemDetails.WriteAsync(
                    context,
                    appException.StatusCode,
                    title: TitleFor(appException.StatusCode),
                    detail: appException.Message,
                    errorType: appException.GetType().Name);
                return;
            }

            context.Response.StatusCode = appException.StatusCode;
            context.Response.Redirect(
                $"/Error?message={Uri.EscapeDataString(appException.Message)}&statusCode={appException.StatusCode}");
        }
        catch (Exception ex)
        {
            RequestLogging.LogByStatusCode(_logger, context, StatusCodes.Status500InternalServerError, ex);

            if (!RequestLogging.IsApiRequest(context) || context.Response.HasStarted)
            {
                throw;
            }

            context.Response.Clear();
            await ApiProblemDetails.WriteAsync(
                context,
                StatusCodes.Status500InternalServerError,
                title: "Internal Server Error",
                detail: "The request could not be completed because of an unexpected server-side failure.",
                errorType: ex.GetType().Name);
        }
    }

    private static string TitleFor(int statusCode) => statusCode switch
    {
        StatusCodes.Status400BadRequest => "Bad Request",
        StatusCodes.Status401Unauthorized => "Unauthorized",
        StatusCodes.Status403Forbidden => "Forbidden",
        StatusCodes.Status404NotFound => "Not Found",
        StatusCodes.Status409Conflict => "Conflict",
        StatusCodes.Status503ServiceUnavailable => "Service Unavailable",
        >= 500 => "Internal Server Error",
        _ => "Request Failed"
    };
}

public static class ExceptionHandlingMiddlewareExtensions
{
    public static IApplicationBuilder UseAppExceptionHandling(this IApplicationBuilder app)
        => app.UseMiddleware<ExceptionHandlingMiddleware>();
}

MVC_App/Middleware/RequestLogging.cs

Rather than let every controller invent its own log format, this helper gives requests a common set of fields and severity rules: Information for 2xx/3xx, Warning for 4xx and Error for 5xx. It also tracks which exception was already logged. Pay attention to http.route_matched: the underscore prevents Elasticsearch from treating the existing http.route string as an object with a child field.

using MVC_App.Diagnostics;

namespace MVC_App.Middleware;

/// <summary>
/// Maps an HTTP status code to the log level ELK should surface it at, and writes a single
/// structured log entry per occurrence. 401/402/403 and 5xx are always Warning/Error so they are
/// never filtered out; other 4xx are Warning; 2xx/3xx are Information.
///
/// Every entry also carries an explicit error condition (<c>error.category</c>, plus the three
/// boolean flags <c>error.is_unauthorized</c> / <c>error.is_not_found</c> / <c>error.is_server_error</c>)
/// so Kibana can filter on a stable value rather than re-deriving meaning from the status code.
/// </summary>
internal static class RequestLogging
{
    /// <summary>
    /// <see cref="HttpContext.Items"/> key holding the exception that <see cref="LogByStatusCode"/>
    /// has already written an entry for. It lets the outermost middleware tell "this was logged
    /// downstream, only the final status code is still missing" apart from "nothing logged this at
    /// all, and I am the last chance to", without logging the same failure twice.
    /// </summary>
    private const string LoggedExceptionKey = "MVC_App.LoggedException";

    /// <summary>
    /// True when <paramref name="exception"/> has already been written out for this request by an
    /// inner middleware. Compared by reference, so a different exception thrown later in the same
    /// request (e.g. while rendering the error response) is still treated as unlogged.
    /// </summary>
    public static bool WasAlreadyLogged(HttpContext context, Exception exception)
        => context.Items.TryGetValue(LoggedExceptionKey, out var logged) && ReferenceEquals(logged, exception);

    public static LogLevel LevelForStatusCode(int statusCode) => statusCode switch
    {
        >= 500 => LogLevel.Error,
        401 or 402 or 403 => LogLevel.Warning,
        >= 400 => LogLevel.Warning,
        _ => LogLevel.Information
    };

    public static void LogByStatusCode(ILogger logger, HttpContext context, int statusCode, Exception? exception)
    {
        var method = context.Request.Method;
        var path = context.Request.Path.Value;
        var logLevel = LevelForStatusCode(statusCode);
        var condition = ErrorConditions.ForStatusCode(statusCode);
        var isFailure = ErrorConditions.IsFailure(statusCode);

        // Null when the URL matched no endpoint at all, which is how a "no such route" 404 is
        // told apart from a "route was fine, the row wasn't there" 404.
        var routePattern = (context.GetEndpoint() as RouteEndpoint)?.RoutePattern.RawText;

        var fields = new Dictionary<string, object?>
        {
            ["http.request.method"] = method,
            ["http.response.status_code"] = statusCode,
            ["url.path"] = path,

            // Logged separately from url.path, which excludes it. Needed to tell a deliberately
            // faulted call (?simulate=500) apart from a real failure on the same route.
            ["url.query"] = context.Request.QueryString.HasValue ? context.Request.QueryString.Value : null,
            ["http.route"] = routePattern,

            // Deliberately NOT named "http.route.matched": Elasticsearch would have to make
            // `http.route` an object to hold a child field, which conflicts with the string
            // above, and every document carrying both would be rejected at index time.
            ["http.route_matched"] = routePattern is not null,

            // Error-condition fields. Present on *every* log entry, including successful ones
            // (error.category = "none"), so a Kibana filter never silently drops documents just
            // because a field is missing on the healthy path.
            ["error.category"] = condition,
            ["error.is_unauthorized"] = condition == ErrorConditions.Unauthorized,
            ["error.is_not_found"] = condition == ErrorConditions.NotFound,
            ["error.is_server_error"] = condition == ErrorConditions.ServerError,
            ["error.type"] = exception?.GetType().Name,
            ["event.outcome"] = isFailure ? "failure" : "success",
            ["event.kind"] = isFailure ? "error" : "event",
            ["http.request.is_api"] = IsApiRequest(context)
        };

        // Drop the nulls rather than exporting them. Elasticsearch treats a null the same as a
        // missing field for `exists`, so a query like `error.type : *` behaves identically either
        // way, but an absent field keeps the documents readable in Discover.
        foreach (var key in fields.Where(f => f.Value is null).Select(f => f.Key).ToList())
        {
            fields.Remove(key);
        }

        using var scope = logger.BeginScope(fields);

        if (exception is not null)
        {
            logger.Log(logLevel, exception,
                "{ExceptionType} while handling {Method} {Path} -> {StatusCode} [{ErrorCategory}]: {Message}",
                exception.GetType().Name, method, path, statusCode, condition, exception.Message);

            // Record that this exception is now on the wire, so the outermost middleware does not
            // emit a second entry for it (see WasAlreadyLogged).
            context.Items[LoggedExceptionKey] = exception;
        }
        else
        {
            logger.Log(logLevel,
                "{Method} {Path} responded {StatusCode} [{ErrorCategory}]",
                method, path, statusCode, condition);
        }
    }

    /// <summary>
    /// True when the request targets the JSON API surface (<c>/api/...</c>) rather than a
    /// server-rendered MVC view or Razor Page. Used both to pick the error response format and
    /// to let ELK split API traffic from browser traffic.
    /// </summary>
    public static bool IsApiRequest(HttpContext context)
        => context.Request.Path.StartsWithSegments("/api", StringComparison.OrdinalIgnoreCase);
}

MVC_App/Middleware/RequestLoggingMiddleware.cs

Placing this middleware first lets it record the final response after the rest of the pipeline has run. It also handles exceptions that escape the inner handlers, without attaching the same exception twice if it was already logged. The completion event still matters even when another log describes the failure; that distinction is important when counting requests in Kibana.

using MVC_App.Exceptions;

namespace MVC_App.Middleware;

/// <summary>
/// Outermost middleware: logs the final outcome of every request (status code + route) at a
/// level based on that status code, so the ELK dashboard can chart 2xx/3xx traffic alongside
/// 401/402/5xx errors. Registered before any exception-handling middleware so it always sees
/// the status code that was ultimately written to the response.
/// </summary>
public sealed class RequestLoggingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<RequestLoggingMiddleware> _logger;

    public RequestLoggingMiddleware(RequestDelegate next, ILogger<RequestLoggingMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        try
        {
            await _next(context);
            RequestLogging.LogByStatusCode(_logger, context, context.Response.StatusCode, exception: null);
        }
        catch (Exception exception)
        {
            // Last line of defence: nothing outside this middleware can log, so an exception that
            // reaches here unlogged would be lost entirely.
            //
            // ExceptionHandlingMiddleware covers everything registered inside it, which is every
            // API and MVC controller, Razor Page, and the auth/routing middleware. For those this
            // only needs to add the final status code, and passing the exception again would
            // duplicate the entry. But an exception raised *outside* that middleware (the
            // UseStatusCodePages callback, or the environment's own handler while it renders the
            // error response) never passed through it, so here it is still unrecorded and this is
            // the only place left that can write it out with its stack trace.
            var alreadyLogged = RequestLogging.WasAlreadyLogged(context, exception);

            // An AppException rethrown because the response had already started keeps its own
            // status code; anything else genuinely is a 500.
            var statusCode = exception is AppException appException
                ? appException.StatusCode
                : StatusCodes.Status500InternalServerError;

            RequestLogging.LogByStatusCode(_logger, context, statusCode, alreadyLogged ? null : exception);
            throw;
        }
    }
}

public static class RequestLoggingMiddlewareExtensions
{
    public static IApplicationBuilder UseAppRequestLogging(this IApplicationBuilder app)
        => app.UseMiddleware<RequestLoggingMiddleware>();
}

MVC_App/Models/Api/ApiDtos.cs

The API exchanges these DTOs rather than exposing EF entities directly. Their annotations describe valid input, and the FromEntity/ToEntity methods keep the conversion explicit. Failed validation then follows the application's common error path.

using System.ComponentModel.DataAnnotations;

namespace MVC_App.Models.Api;

/// <summary>
/// Request/response shape for <c>/api/departments</c>. Kept separate from the EF entity so the
/// API contract (and its validation rules, which drive the 400 condition) can change without
/// touching the database model.
/// </summary>
public sealed class DepartmentDto
{
    [Range(1, int.MaxValue, ErrorMessage = "DeptNo must be a positive number.")]
    public int DeptNo { get; set; }

    [Required(ErrorMessage = "DeptName is required.")]
    [StringLength(200, MinimumLength = 2, ErrorMessage = "DeptName must be between 2 and 200 characters.")]
    public string DeptName { get; set; } = null!;

    [Required(ErrorMessage = "Location is required.")]
    [StringLength(40, MinimumLength = 2, ErrorMessage = "Location must be between 2 and 40 characters.")]
    public string Location { get; set; } = null!;

    public static DepartmentDto FromEntity(Department department) => new()
    {
        DeptNo = department.DeptNo,
        DeptName = department.DeptName,
        Location = department.Location
    };

    public Department ToEntity() => new()
    {
        DeptNo = DeptNo,
        DeptName = DeptName,
        Location = Location
    };
}

/// <summary>Request/response shape for <c>/api/employees</c>.</summary>
public sealed class EmployeeDto
{
    [Range(1, int.MaxValue, ErrorMessage = "EmpNo must be a positive number.")]
    public int EmpNo { get; set; }

    [Required(ErrorMessage = "EmpName is required.")]
    [StringLength(100, MinimumLength = 2, ErrorMessage = "EmpName must be between 2 and 100 characters.")]
    public string EmpName { get; set; } = null!;

    [Range(1, 10_000_000, ErrorMessage = "Salary must be greater than zero.")]
    public int Salary { get; set; }

    [Required(ErrorMessage = "DeptName is required.")]
    [StringLength(100, MinimumLength = 2, ErrorMessage = "DeptName must be between 2 and 100 characters.")]
    public string DeptName { get; set; } = null!;

    [Required(ErrorMessage = "Designation is required.")]
    [StringLength(100, MinimumLength = 2, ErrorMessage = "Designation must be between 2 and 100 characters.")]
    public string Designation { get; set; } = null!;

    public static EmployeeDto FromEntity(Employee employee) => new()
    {
        EmpNo = employee.EmpNo,
        EmpName = employee.EmpName,
        Salary = employee.Salary,
        DeptName = employee.DeptName,
        Designation = employee.Designation
    };

    public Employee ToEntity() => new()
    {
        EmpNo = EmpNo,
        EmpName = EmpName,
        Salary = Salary,
        DeptName = DeptName,
        Designation = Designation
    };
}

MVC_App/Models/CompanyContext.cs

CompanyContext maps the existing Department and Employee tables, including their keys, string lengths and unique department-name index. There is no employee-to-department foreign key configured here, which is why the controller checks matter. Program.cs supplies the connection and interceptor when it registers the context.

using System;
using System.Collections.Generic;
using Microsoft.EntityFrameworkCore;

namespace MVC_App.Models;

public partial class CompanyContext : DbContext
{
    public CompanyContext()
    {
    }

    public CompanyContext(DbContextOptions<CompanyContext> options)
        : base(options)
    {
    }

    public virtual DbSet<Department> Departments { get; set; }

    public virtual DbSet<Employee> Employees { get; set; }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<Department>(entity =>
        {
            entity.HasKey(e => e.DeptNo).HasName("PK__Departme__0148CAAE7A959546");

            entity.ToTable("Department");

            entity.Property(e => e.DeptNo).ValueGeneratedNever();
            entity.Property(e => e.DeptName)
                .HasMaxLength(200)
                .IsUnicode(false);
            entity.Property(e => e.Location)
                .HasMaxLength(40)
                .IsUnicode(false);
            entity.HasIndex(e => e.DeptName).IsUnique();
        });

        modelBuilder.Entity<Employee>(entity =>
        {
            entity.HasKey(e => e.EmpNo);

            entity.ToTable("Employee");

            entity.Property(e => e.EmpNo).ValueGeneratedNever();
            entity.Property(e => e.DeptName)
                .HasMaxLength(100)
                .IsUnicode(false);
            entity.Property(e => e.Designation)
                .HasMaxLength(100)
                .IsUnicode(false);
            entity.Property(e => e.EmpName)
                .HasMaxLength(100)
                .IsUnicode(false);
        });

        OnModelCreatingPartial(modelBuilder);
    }

    partial void OnModelCreatingPartial(ModelBuilder modelBuilder);
}

MVC_App/Models/Department.cs

The department entity is deliberately small: a number, name and location. The controller rules and CompanyContext mapping supply the behavior around those values.

using System;
using System.Collections.Generic;

namespace MVC_App.Models;

public partial class Department
{
    public int DeptNo { get; set; }

    public string DeptName { get; set; } = null!;

    public string Location { get; set; } = null!;
}

MVC_App/Models/Employee.cs

This is the SQL Server employee shape used by the MVC application. Salary and employee details belong to business data; avoid copying them indiscriminately into log fields just because they are available on the entity.

using System;
using System.Collections.Generic;

namespace MVC_App.Models;

public partial class Employee
{
    public int EmpNo { get; set; }

    public string EmpName { get; set; } = null!;

    public int Salary { get; set; }

    public string DeptName { get; set; } = null!;

    public string Designation { get; set; } = null!;
}

MVC_App/Models/PaginatedList.cs

Pagination needs both the rows for the current page and a total count. CreateAsync performs those queries and calculates the navigation flags. When you inspect a page request, remember that this helper can account for more than one SQL command.

using Microsoft.EntityFrameworkCore;

namespace MVC_App.Models;

public class PaginatedList<T> : List<T>
{
    public int PageIndex { get; private set; }
    public int TotalPages { get; private set; }

    public PaginatedList(List<T> items, int count, int pageIndex, int pageSize)
    {
        PageIndex = pageIndex;
        TotalPages = (int)Math.Ceiling(count / (double)pageSize);

        this.AddRange(items);
    }

    public bool HasPreviousPage => PageIndex > 1;

    public bool HasNextPage => PageIndex < TotalPages;

    public static async Task<PaginatedList<T>> CreateAsync(IQueryable<T> source, int pageIndex, int pageSize)
    {
        var count = await source.CountAsync();
        var items = await source.Skip((pageIndex - 1) * pageSize).Take(pageSize).ToListAsync();
        return new PaginatedList<T>(items, count, pageIndex, pageSize);
    }
}

 

MVC_App/Pages/Error.cshtml

The page displays the status, message and request identifier. Razor encodes normal expressions, but that doesn't decide whether an error message is appropriate to share. Review detailed business errors before showing them publicly.

@page
@model ErrorModel
@{
    ViewData["Title"] = "Error";
}

<h1 class="text-danger">Error @(Model.ErrorStatusCode > 0 ? $"({Model.ErrorStatusCode})" : "").</h1>
@if (!string.IsNullOrEmpty(Model.Message))
{
    <h2 class="text-danger">@Model.Message</h2>
}
else
{
    <h2 class="text-danger">An error occurred while processing your request.</h2>
}

@if (Model.ShowRequestId)
{
    <p>
        <strong>Request ID:</strong> <code>@Model.RequestId</code>
    </p>
}

<h3>Development Mode</h3>
<p>
    Swapping to the <strong>Development</strong> environment displays detailed information about the error that occurred.
</p>
<p>
    <strong>The Development environment shouldn't be enabled for deployed applications.</strong>
    It can result in displaying sensitive information from exceptions to end users.
    For local debugging, enable the <strong>Development</strong> environment by setting the <strong>ASPNETCORE_ENVIRONMENT</strong> environment variable to <strong>Development</strong>
    and restarting the app.
</p>

MVC_App/Pages/Error.cshtml.cs

The error page reads the supplied status and message, disables caching and shows an Activity-based or HTTP request identifier. If the browser arrived through a redirect, that identifier may belong to the new request. Look back to the original error event when you need the failing request's trace.

using System.Diagnostics;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;

namespace MVC_App.Pages
{
    [ResponseCache(Duration = 0, Location = ResponseCacheLocation.None, NoStore = true)]
    [IgnoreAntiforgeryToken]
    public class ErrorModel : PageModel
    {
        public string? RequestId { get; set; }

        public bool ShowRequestId => !string.IsNullOrEmpty(RequestId);

        public string? Message { get; set; }

        public int ErrorStatusCode { get; set; }

        private readonly ILogger<ErrorModel> _logger;

        public ErrorModel(ILogger<ErrorModel> logger)
        {
            _logger = logger;
        }

        public void OnGet(string? message, int? statusCode)
        {
            RequestId = Activity.Current?.Id ?? HttpContext.TraceIdentifier;
            Message = message;
            ErrorStatusCode = statusCode ?? HttpContext.Response.StatusCode;
        }
    }

}

MVC_App/Program.cs

Start here to see how the application is wired together. Telemetry is registered before the host is built, and both SQL contexts receive the command-logging interceptor. Cookies protect the browser pages; API controllers use the separate API-key scheme. Notice the middleware order: request logging sits outside the exception handlers so it can see the final result. Automatic API validation responses are suppressed so validation failures pass through the same error-handling path as other domain errors.

using Microsoft.AspNetCore.Identity;
using Microsoft.EntityFrameworkCore;
using MVC_App.Authentication;
using MVC_App.Data;
using MVC_App.Diagnostics;
using MVC_App.Middleware;
using MVC_App.Models;
using MVC_App.Models.Identity;
using MVC_App.Telemetry;

var builder = WebApplication.CreateBuilder(args);
builder.AddApplicationTelemetry();

// Add services to the container.
builder.Services.AddRazorPages();
builder.Services.AddControllersWithViews();

// The JSON API controllers validate their own models (ApiControllerBase.ThrowIfInvalid) so that a
// 400 flows through ExceptionHandlingMiddleware and picks up the same error.category fields as
// every other failure. ApiController's automatic 400 would short-circuit before that runs.
builder.Services.Configure<Microsoft.AspNetCore.Mvc.ApiBehaviorOptions>(options =>
{
    options.SuppressModelStateInvalidFilter = true;
});

// Logs every DB command (success and failure) on both DbContexts; see Data/DbCommandLoggingInterceptor.cs.
builder.Services.AddSingleton<DbCommandLoggingInterceptor>();

builder.Services.AddDbContext<CompanyContext>((sp, options) =>
    options.UseSqlServer(builder.Configuration.GetConnectionString("CompanyContext"))
        .AddInterceptors(sp.GetRequiredService<DbCommandLoggingInterceptor>()));

// Microsoft Identity is stored in its own code-first database on the local SQL Server
// instance, kept separate from the existing (database-first) CompanyContext.
builder.Services.AddDbContext<IdentityContext>((sp, options) =>
    options.UseSqlServer(builder.Configuration.GetConnectionString("IdentityConnection"))
        .AddInterceptors(sp.GetRequiredService<DbCommandLoggingInterceptor>()));

builder.Services.AddIdentity<ApplicationUser, IdentityRole>(options =>
    {
        options.Password.RequireNonAlphanumeric = false;
        options.SignIn.RequireConfirmedAccount = false;
    })
    .AddEntityFrameworkStores<IdentityContext>()
    .AddDefaultTokenProviders();

// Identity's cookie stays the default scheme for the MVC views. The API key scheme is additive:
// /api controllers opt into it explicitly, which is what lets an unauthenticated API call return
// a 401 instead of a 302 redirect to the login page.
builder.Services.AddAuthentication()
    .AddApiKeyAuthentication(builder.Configuration);

builder.Services.ConfigureApplicationCookie(options =>
{
    options.LoginPath = "/Account/Login";
    options.AccessDeniedPath = "/Account/AccessDenied";
});

var app = builder.Build();

// Logs the final status code of every request (2xx/3xx as Information, 401/402/403/4xx as
// Warning, 5xx as Error) along with its error condition (error.category, error.is_unauthorized,
// error.is_not_found, error.is_server_error) so the ELK dashboard can chart the full
// status-code distribution and filter on a stable condition slug.
app.UseAppRequestLogging();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error");
    // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
    app.UseHsts();
}
else
{
    app.UseDeveloperExceptionPage();
}

// Catches domain validation exceptions (UnauthorizedRequestException, ResourceNotFoundException,
// ServerSideException, PrimaryKeyNotFoundException, DuplicateDeptNameException, ...), logs them
// with their error condition, and either redirects to /Error (browser) or writes an
// ApiProblemDetails document (/api).
//
// Registered ahead of UseStatusCodePages so it also wraps that callback. Everything downstream of
// this line -- every API and MVC controller, every Razor Page, routing, authentication and the
// status-code pages -- has its exceptions logged here. Only the environment's own handler above
// sits outside it, and RequestLoggingMiddleware catches anything that escapes that far.
app.UseAppExceptionHandling();

// Gives /api responses that the framework produced without a body (an unmatched route's 404, a
// 405 from the wrong verb, a 401 challenge) the same JSON problem shape as the handled errors,
// so an API caller never has to special-case an empty body.
app.UseStatusCodePages(async context =>
{
    var response = context.HttpContext.Response;
    if (!RequestLogging.IsApiRequest(context.HttpContext) ||
        response.ContentLength.HasValue ||
        !string.IsNullOrEmpty(response.ContentType))
    {
        return;
    }

    await ApiProblemDetails.WriteAsync(
        context.HttpContext,
        response.StatusCode,
        title: ReasonPhrases.For(response.StatusCode),
        detail: $"The request to '{context.HttpContext.Request.Path}' ended with status {response.StatusCode}.");
});

app.UseHttpsRedirection();
app.UseStaticFiles();

app.UseRouting();

app.UseAuthentication();
app.UseAuthorization();

app.MapControllerRoute(
    name: "default",
    pattern: "{controller=Home}/{action=Index}/{id?}");

app.MapRazorPages();

app.Run();

/// <summary>Minimal status-code-to-title map for the problem documents written above.</summary>
internal static class ReasonPhrases
{
    public static string For(int statusCode) => statusCode switch
    {
        StatusCodes.Status400BadRequest => "Bad Request",
        StatusCodes.Status401Unauthorized => "Unauthorized",
        StatusCodes.Status403Forbidden => "Forbidden",
        StatusCodes.Status404NotFound => "Not Found",
        StatusCodes.Status405MethodNotAllowed => "Method Not Allowed",
        StatusCodes.Status409Conflict => "Conflict",
        StatusCodes.Status415UnsupportedMediaType => "Unsupported Media Type",
        StatusCodes.Status503ServiceUnavailable => "Service Unavailable",
        >= 500 => "Internal Server Error",
        _ => "Request Failed"
    };
}

MVC_App/Telemetry/TelemetryExtensions.cs

This method connects the application's diagnostics to the Collector. It checks the endpoint and enable switch, adds service metadata, and registers logs, HTTP/SQL traces and runtime/server metrics. The SQL category filter reduces duplicate informational logs, while the enrichment callback puts command text on database spans. The numbered walkthrough after the code explains each registration and the limits worth keeping in mind.

using System.Data.Common;
using OpenTelemetry;
using OpenTelemetry.Exporter;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

namespace MVC_App.Telemetry;

public static class TelemetryExtensions
{
    public static WebApplicationBuilder AddApplicationTelemetry(this WebApplicationBuilder builder)
    {
        var endpoint = builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"];
        if (!builder.Configuration.GetValue("Telemetry:Enabled", true) || string.IsNullOrWhiteSpace(endpoint))
        {
            return builder;
        }

        if (!Uri.TryCreate(endpoint, UriKind.Absolute, out var endpointUri) ||
            (endpointUri.Scheme != Uri.UriSchemeHttp && endpointUri.Scheme != Uri.UriSchemeHttps))
        {
            throw new InvalidOperationException("OTEL_EXPORTER_OTLP_ENDPOINT must be an absolute HTTP or HTTPS base URL.");
        }

        builder.Logging.AddOpenTelemetry(options =>
        {
            options.IncludeFormattedMessage = true;
            options.IncludeScopes = true;
        });

        // Do not export EF Core's informational SQL command text as structured logs.
        builder.Logging.AddFilter<OpenTelemetryLoggerProvider>(
            "Microsoft.EntityFrameworkCore.Database.Command", LogLevel.Warning);

        builder.Services.AddOpenTelemetry()
            .ConfigureResource(resource => resource
                .AddService(builder.Configuration["OTEL_SERVICE_NAME"] ?? builder.Environment.ApplicationName)
                .AddAttributes(new Dictionary<string, object>
                {
                    ["deployment.environment.name"] = builder.Environment.EnvironmentName
                }))
            .WithTracing(tracing => tracing
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                // Traces every ADO.NET command EF Core issues (both CompanyContext and
                // IdentityContext use Microsoft.Data.SqlClient under the hood), success or
                // failure, as a child span of the request that triggered it. RecordException
                // marks failed commands with the exception on the span itself; EnrichWithSqlCommand
                // adds the parameterized SQL text (placeholders, not literal values, since
                // sensitive-data logging is never enabled) so failed/slow queries are identifiable.
                .AddSqlClientInstrumentation(options =>
                {
                    options.RecordException = true;
                    options.EnrichWithSqlCommand = (activity, command) =>
                    {
                        if (command is DbCommand dbCommand)
                        {
                            activity.SetTag("db.statement", dbCommand.CommandText);
                        }
                    };
                }))
            .WithMetrics(metrics => metrics
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation()
                .AddMeter("Microsoft.AspNetCore.Server.Kestrel",
                    "Microsoft.AspNetCore.Routing", "Microsoft.AspNetCore.Diagnostics"))
            // Registers one exporter per signal, including logs. Paths are appended automatically.
            .UseOtlpExporter(OtlpExportProtocol.HttpProtobuf, endpointUri);

        return builder;
    }
}

Walking through TelemetryExtensions.cs

Most of the application's telemetry setup comes together in this one method. Read it from top to bottom: first it checks configuration, then it identifies the service and chooses the signals, and finally it connects them to the Collector.

1. Start with the namespaces. The OpenTelemetry imports make the logging, tracing, metrics, resource and exporter extension methods available. Their implementations come from the project's NuGet references. System.Data.Common has a more specific job: it supplies DbCommand for the SQL enrichment callback near the middle of the method.
2. The extension keeps Program.cs readable. The this WebApplicationBuilder builder parameter lets startup code call builder.AddApplicationTelemetry(). Returning the builder allows further configuration to be chained. All of this runs before builder.Build(), rather than once for each request.
3. Read the Collector address from configuration. builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"] gets the base URL, with the usual ASP.NET Core providers allowing environment values to override JSON. A host process can reach the local Collector at http://localhost:4318. Inside a container, however, localhost means that container; use an address the application can actually reach.
4. Allow telemetry to be switched off. GetValue("Telemetry:Enabled", true) defaults to enabled, but an endpoint must also be present. Either Telemetry__Enabled=false or a blank endpoint makes the method return early. This skips the OTel registration here; it doesn't turn off ordinary logging providers, request middleware or diagnostic endpoints.
5. Catch an invalid URL before serving requests. Uri.TryCreate checks for an absolute address, and the next condition permits only HTTP or HTTPS. An invalid setting raises InvalidOperationException at startup. This is a configuration check, not a connectivity test: a valid URL can still point to a stopped Collector.
6. Connect existing ILogger calls to OTel. builder.Logging.AddOpenTelemetry adds a provider, so application logs can enter the telemetry pipeline without rewriting every ILogger<T> call. Logging filters still apply. The destination is configured later by the shared exporter.
7. Keep the rendered log message. IncludeFormattedMessage = true makes the message readable without reconstructing it from a template and argument list. That is convenient in Discover, but it also means any sensitive argument can end up in the rendered text. Formatting is not redaction.
8. Keep the context carried by scopes. With IncludeScopes = true, fields supplied by request middleware and the SQL interceptor travel with the log. Examples include http.response.status_code, error.category, url.path and db.duration_ms. Logstash places these under log_attributes, which is why the Kibana queries use that prefix. An active Activity can also supply trace/span IDs; startup logs and work without one may have no such correlation.
9. Reduce duplicate SQL logging without hiding failures. AddFilter<OpenTelemetryLoggerProvider> raises the minimum level for Microsoft.EntityFrameworkCore.Database.Command to Warning on the OTel provider. Warning, Error and Critical events remain eligible. The custom MVC_App.Data.DbCommandLoggingInterceptor has a different category and still logs its commands. This filter neither changes other providers globally nor removes SQL text from spans.
10. Let the host own the providers. builder.Services.AddOpenTelemetry() starts the shared tracing and metrics configuration and ties provider lifecycle to the application host. There is no need to construct a provider inside each controller. The remaining fluent calls specify what that shared setup should collect.
11. Give every signal a service identity. ConfigureResource uses AddService with OTEL_SERVICE_NAME and adds deployment.environment.name. These attributes distinguish MVC_App from Blazor and Development from Production. The ?? fallback to builder.Environment.ApplicationName handles a null setting, not an empty string. Resource values describe the producer, so they are reused across requests and signal types.
12. Observe incoming HTTP requests. Under WithTracing, AddAspNetCoreInstrumentation() collects the framework's request activities as server spans. A valid incoming trace context can connect the request to its caller. This doesn't automatically add a business span for every Razor or Blazor interaction, and recording/export still depends on sampling and SDK configuration.
13. Observe outgoing HTTP calls too. AddHttpClientInstrumentation() collects supported HttpClient activities and propagates trace context. An outbound call made during an active request can then belong to the same trace. It only observes calls that actually happen. The Blazor client span is registered in Blazor's own extension; this MVC extension observes the receiving server request.
14. Follow the request into SQL Server. AddSqlClientInstrumentation subscribes to the supported SQL client diagnostics. Both CompanyContext and IdentityContext use SQL Server, so their commands can be observed—not just employee and department queries. A command can inherit an active request's context, but background work won't acquire an HTTP parent on its own. Npgsql needs the separate instrumentation used in PgSqlClient.
15. Keep exception details on failed SQL spans. options.RecordException = true asks the instrumentation to attach exception details when it observes a failure. That helps explain a failed dependency in the trace. It doesn't catch the exception for the application or decide the response status; normal propagation and ExceptionHandlingMiddleware still do that work.
16. Notice exactly where command text is added. EnrichWithSqlCommand receives the Activity and command. After checking command is DbCommand dbCommand, it calls activity.SetTag("db.statement", dbCommand.CommandText). Parameterized SQL usually contains placeholders, but literals can still expose data. This callback neither redacts nor truncates the statement. The interceptor's 1,000-character limit applies to its log message, not to this span attribute.
17. Use HTTP metrics for the broader picture. WithMetrics adds ASP.NET Core and HttpClient measurements such as request and dependency duration. A trace explains one operation; metrics help show whether the same behavior is common. Available instruments depend on the framework and package versions, and arbitrary log messages are not automatically turned into metrics.
18. Check the runtime when several endpoints slow down. AddRuntimeInstrumentation() collects supported measurements for garbage collection, managed memory and the thread pool. Those observations can help distinguish one slow query from process-wide pressure. They are separate from the custom employee counters and salary gauges defined in PgSqlClient's AppDiagnostics.
19. Subscribe to the framework meters you need. AddMeter names the Kestrel, routing and diagnostics sources: Microsoft.AspNetCore.Server.Kestrel, Microsoft.AspNetCore.Routing and Microsoft.AspNetCore.Diagnostics. It listens for their instruments rather than creating new ones. You may need to exercise the relevant behavior before a measurement appears.
20. Send all three signals through OTLP. UseOtlpExporter(OtlpExportProtocol.HttpProtobuf, endpointUri) connects logs, traces and metrics to the Collector. This is protobuf over HTTP—not JSON or OTLP/gRPC. From the configured base URL, the exporters use paths such as /v1/logs, /v1/traces and /v1/metrics. Neither a Kibana address nor an Elasticsearch index name belongs here.
21. Don't confuse a successful request with a successful export. The host manages the providers and their disposal; processors and readers control export timing. This method doesn't explicitly choose a sampler, durable queue, retry policy or custom metric interval. Set OTEL_METRIC_EXPORT_INTERVAL in the actual process environment, or bind it explicitly, rather than assuming the JSON key configures the reader. Shutdown and network failures can still lose telemetry, so check the pipeline beyond the application.
22. Follow the Collector's routing for the final step. Its memory_limiter and batch processors run before export. Traces and metrics go directly to Elasticsearch OTel data streams. Logs take the rotating JSONL path through Logstash into mvcapp-logs-YYYY.MM.dd, where Kibana can query them. Keeping this routing outside the application lets you change storage and transformations without rewriting every instrumentation call.

The method's job ends at collecting and exporting the signals. To investigate a real request, combine that setup with the context added by middleware and SQL interceptors, then follow the data through the Collector to the appropriate Kibana view.

MVC_App/Views/Department/Create.cshtml

The form collects the department number, name and location. Client validation gives quick feedback, but the controller and database still decide whether the number and name are unique.

@model MVC_App.Models.Department

@{
    ViewData["Title"] = "Create Department";
}

<h1>Create Department</h1>

<hr />
<div class="row">
    <div class="col-md-4">
        <form asp-action="Create">
            <div asp-validation-summary="ModelOnly" class="text-danger"></div>
            <div class="form-group">
                <label asp-for="DeptNo" class="control-label"></label>
                <input asp-for="DeptNo" class="form-control" />
                <span asp-validation-for="DeptNo" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="DeptName" class="control-label"></label>
                <input asp-for="DeptName" class="form-control" />
                <span asp-validation-for="DeptName" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="Location" class="control-label"></label>
                <input asp-for="Location" class="form-control" />
                <span asp-validation-for="Location" class="text-danger"></span>
            </div>
            <div class="form-group mt-2">
                <input type="submit" value="Create" class="btn btn-primary" />
            </div>
        </form>
    </div>
</div>

<div>
    <a asp-action="Index">Back to List</a>
</div>

@section Scripts {
    @{await Html.RenderPartialAsync("_ValidationScriptsPartial");}
}

MVC_App/Views/Department/Delete.cshtml

The confirmation form posts the department number back to the controller. The important business check—whether employees still use that department—happens on the server.

@model MVC_App.Models.Department

@{
    ViewData["Title"] = "Delete Department";
}

<h1>Delete Department</h1>

<h3>Are you sure you want to delete this?</h3>
<div>
    <hr />
    <dl class="row">
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.DeptNo)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.DeptNo)</dd>
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.DeptName)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.DeptName)</dd>
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.Location)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.Location)</dd>
    </dl>

    <form asp-action="Delete">
        <input type="hidden" asp-for="DeptNo" />
        <input type="submit" value="Delete" class="btn btn-danger" /> |
        <a asp-action="Index">Back to List</a>
    </form>
</div>

MVC_App/Views/Department/Details.cshtml

After the controller finds a department, this view displays its fields with edit and back navigation. A missing record takes the error path before the view can render.

@model MVC_App.Models.Department

@{
    ViewData["Title"] = "Department Details";
}

<h1>Department Details</h1>

<div>
    <hr />
    <dl class="row">
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.DeptNo)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.DeptNo)</dd>
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.DeptName)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.DeptName)</dd>
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.Location)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.Location)</dd>
    </dl>
</div>
<div>
    <a asp-action="Edit" asp-route-id="@Model?.DeptNo">Edit</a> |
    <a asp-action="Index">Back to List</a>
</div>

MVC_App/Views/Department/Edit.cshtml

The department number stays read-only while the user edits the name and location. The server must still validate the submitted values; a read-only input is a UI choice, not an authorization rule.

@model MVC_App.Models.Department

@{
    ViewData["Title"] = "Edit Department";
}

<h1>Edit Department</h1>

<hr />
<div class="row">
    <div class="col-md-4">
        <form asp-action="Edit">
            <div asp-validation-summary="ModelOnly" class="text-danger"></div>
            <div class="form-group">
                <label asp-for="DeptNo" class="control-label"></label>
                <input asp-for="DeptNo" class="form-control" readonly />
                <span asp-validation-for="DeptNo" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="DeptName" class="control-label"></label>
                <input asp-for="DeptName" class="form-control" />
                <span asp-validation-for="DeptName" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="Location" class="control-label"></label>
                <input asp-for="Location" class="form-control" />
                <span asp-validation-for="Location" class="text-danger"></span>
            </div>
            <div class="form-group mt-2">
                <input type="submit" value="Save" class="btn btn-primary" />
            </div>
        </form>
    </div>
</div>

<div>
    <a asp-action="Index">Back to List</a>
</div>

@section Scripts {
    @{await Html.RenderPartialAsync("_ValidationScriptsPartial");}
}

MVC_App/Views/Department/Index.cshtml

The department table links to each CRUD action and to the employees in a selected department. Its previous/next controls are a convenient way to observe the paging queries behind a normal browser request.

@model MVC_App.Models.PaginatedList<MVC_App.Models.Department>

@{
    ViewData["Title"] = "Departments";
}

<h1>Departments</h1>

<p>
    <a asp-action="Create" class="btn btn-primary">Create New</a>
</p>
<table class="table">
    <thead>
        <tr>
            <th>@Html.DisplayNameFor(model => model[0].DeptNo)</th>
            <th>@Html.DisplayNameFor(model => model[0].DeptName)</th>
            <th>@Html.DisplayNameFor(model => model[0].Location)</th>
            <th></th>
        </tr>
    </thead>
    <tbody>
@foreach (var item in Model)
{
        <tr>
            <td>@Html.DisplayFor(modelItem => item.DeptNo)</td>
            <td>@Html.DisplayFor(modelItem => item.DeptName)</td>
            <td>@Html.DisplayFor(modelItem => item.Location)</td>
            <td>
                <a asp-action="Edit" asp-route-id="@item.DeptNo">Edit</a> |
                <a asp-action="Details" asp-route-id="@item.DeptNo">Details</a> |
                <a asp-action="Delete" asp-route-id="@item.DeptNo">Delete</a> |
                <a asp-controller="Employee" asp-action="ListByDepartment" asp-route-deptName="@item.DeptName">Employees</a>
            </td>
        </tr>
}
    </tbody>
</table>

<nav aria-label="Department pagination">
    <ul class="pagination">
        <li class="page-item @(Model.HasPreviousPage ? "" : "disabled")">
            <a class="page-link" asp-action="Index" asp-route-pageNumber="@(Model.PageIndex - 1)">Previous</a>
        </li>
        @for (int i = 1; i <= Model.TotalPages; i++)
        {
            <li class="page-item @(i == Model.PageIndex ? "active" : "")">
                <a class="page-link" asp-action="Index" asp-route-pageNumber="@i">@i</a>
            </li>
        }
        <li class="page-item @(Model.HasNextPage ? "" : "disabled")">
            <a class="page-link" asp-action="Index" asp-route-pageNumber="@(Model.PageIndex + 1)">Next</a>
        </li>
    </ul>
</nav>

MVC_App/Views/Employee/Create.cshtml

The create form gets its department choices from ViewBag.Departments, so users can select a known name rather than type one. Validation messages appear beside the submitted fields.

@model MVC_App.Models.Employee

@{
    ViewData["Title"] = "Create Employee";
}

<h1>Create Employee</h1>

<hr />
<div class="row">
    <div class="col-md-4">
        <form asp-action="Create">
            <div asp-validation-summary="ModelOnly" class="text-danger"></div>
            <div class="form-group">
                <label asp-for="EmpNo" class="control-label"></label>
                <input asp-for="EmpNo" class="form-control" />
                <span asp-validation-for="EmpNo" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="EmpName" class="control-label"></label>
                <input asp-for="EmpName" class="form-control" />
                <span asp-validation-for="EmpName" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="Salary" class="control-label"></label>
                <input asp-for="Salary" class="form-control" />
                <span asp-validation-for="Salary" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="DeptName" class="control-label"></label>
                <select asp-for="DeptName" asp-items="ViewBag.Departments" class="form-select">
                    <option value="">-- Select Department --</option>
                </select>
                <span asp-validation-for="DeptName" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="Designation" class="control-label"></label>
                <input asp-for="Designation" class="form-control" />
                <span asp-validation-for="Designation" class="text-danger"></span>
            </div>
            <div class="form-group mt-2">
                <input type="submit" value="Create" class="btn btn-primary" />
            </div>
        </form>
    </div>
</div>

<div>
    <a asp-action="Index">Back to List</a>
</div>

@section Scripts {
    @{await Html.RenderPartialAsync("_ValidationScriptsPartial");}
}

MVC_App/Views/Employee/Delete.cshtml

The page asks for confirmation and posts the hidden employee key. The delete itself belongs to the controller's POST action, not to rendering this page.

@model MVC_App.Models.Employee

@{
    ViewData["Title"] = "Delete Employee";
}

<h1>Delete Employee</h1>

<h3>Are you sure you want to delete this?</h3>
<div>
    <hr />
    <dl class="row">
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.EmpNo)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.EmpNo)</dd>
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.EmpName)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.EmpName)</dd>
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.Salary)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.Salary)</dd>
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.DeptName)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.DeptName)</dd>
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.Designation)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.Designation)</dd>
    </dl>

    <form asp-action="Delete">
        <input type="hidden" asp-for="EmpNo" />
        <input type="submit" value="Delete" class="btn btn-danger" /> |
        <a asp-action="Index">Back to List</a>
    </form>
</div>

MVC_App/Views/Employee/Details.cshtml

A read-only view of the selected employee, with links back to the list or into the edit screen. The controller has already done the lookup by the time this renders.

@model MVC_App.Models.Employee

@{
    ViewData["Title"] = "Employee Details";
}

<h1>Employee Details</h1>

<div>
    <hr />
    <dl class="row">
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.EmpNo)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.EmpNo)</dd>
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.EmpName)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.EmpName)</dd>
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.Salary)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.Salary)</dd>
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.DeptName)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.DeptName)</dd>
        <dt class="col-sm-2">@Html.DisplayNameFor(model => model.Designation)</dt>
        <dd class="col-sm-10">@Html.DisplayFor(model => model.Designation)</dd>
    </dl>
</div>
<div>
    <a asp-action="Edit" asp-route-id="@Model?.EmpNo">Edit</a> |
    <a asp-action="Index">Back to List</a>
</div>

MVC_App/Views/Employee/Edit.cshtml

Notice the difference from the create form: DeptName is a text input here, while EmpNo is read-only. That makes the server's department validation especially important when an update is submitted.

@model MVC_App.Models.Employee

@{
    ViewData["Title"] = "Edit Employee";
}

<h1>Edit Employee</h1>

<hr />
<div class="row">
    <div class="col-md-4">
        <form asp-action="Edit">
            <div asp-validation-summary="ModelOnly" class="text-danger"></div>
            <div class="form-group">
                <label asp-for="EmpNo" class="control-label"></label>
                <input asp-for="EmpNo" class="form-control" readonly />
                <span asp-validation-for="EmpNo" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="EmpName" class="control-label"></label>
                <input asp-for="EmpName" class="form-control" />
                <span asp-validation-for="EmpName" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="Salary" class="control-label"></label>
                <input asp-for="Salary" class="form-control" />
                <span asp-validation-for="Salary" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="DeptName" class="control-label"></label>
                <input asp-for="DeptName" class="form-control" />
                <span asp-validation-for="DeptName" class="text-danger"></span>
            </div>
            <div class="form-group">
                <label asp-for="Designation" class="control-label"></label>
                <input asp-for="Designation" class="form-control" />
                <span asp-validation-for="Designation" class="text-danger"></span>
            </div>
            <div class="form-group mt-2">
                <input type="submit" value="Save" class="btn btn-primary" />
            </div>
        </form>
    </div>
</div>

<div>
    <a asp-action="Index">Back to List</a>
</div>

@section Scripts {
    @{await Html.RenderPartialAsync("_ValidationScriptsPartial");}
}

MVC_App/Views/Employee/Index.cshtml

This table is the starting point for the MVC employee screens. Paging, CRUD links and department filtering give you several ordinary browser actions to compare in the request logs.

@model MVC_App.Models.PaginatedList<MVC_App.Models.Employee>

@{
    ViewData["Title"] = "Employees";
}

<h1>Employees</h1>

<p>
    <a asp-action="Create" class="btn btn-primary">Create New</a>
    <a asp-action="ListByDepartment" class="btn btn-secondary">List By Department</a>
</p>
<table class="table">
    <thead>
        <tr>
            <th>@Html.DisplayNameFor(model => model[0].EmpNo)</th>
            <th>@Html.DisplayNameFor(model => model[0].EmpName)</th>
            <th>@Html.DisplayNameFor(model => model[0].Salary)</th>
            <th>@Html.DisplayNameFor(model => model[0].DeptName)</th>
            <th>@Html.DisplayNameFor(model => model[0].Designation)</th>
            <th></th>
        </tr>
    </thead>
    <tbody>
@foreach (var item in Model)
{
        <tr>
            <td>@Html.DisplayFor(modelItem => item.EmpNo)</td>
            <td>@Html.DisplayFor(modelItem => item.EmpName)</td>
            <td>@Html.DisplayFor(modelItem => item.Salary)</td>
            <td>@Html.DisplayFor(modelItem => item.DeptName)</td>
            <td>@Html.DisplayFor(modelItem => item.Designation)</td>
            <td>
                <a asp-action="Edit" asp-route-id="@item.EmpNo">Edit</a> |
                <a asp-action="Details" asp-route-id="@item.EmpNo">Details</a> |
                <a asp-action="Delete" asp-route-id="@item.EmpNo">Delete</a>
            </td>
        </tr>
}
    </tbody>
</table>

<nav aria-label="Employee pagination">
    <ul class="pagination">
        <li class="page-item @(Model.HasPreviousPage ? "" : "disabled")">
            <a class="page-link" asp-action="Index" asp-route-pageNumber="@(Model.PageIndex - 1)">Previous</a>
        </li>
        @for (int i = 1; i <= Model.TotalPages; i++)
        {
            <li class="page-item @(i == Model.PageIndex ? "active" : "")">
                <a class="page-link" asp-action="Index" asp-route-pageNumber="@i">@i</a>
            </li>
        }
        <li class="page-item @(Model.HasNextPage ? "" : "disabled")">
            <a class="page-link" asp-action="Index" asp-route-pageNumber="@(Model.PageIndex + 1)">Next</a>
        </li>
    </ul>
</nav>

MVC_App/Views/Employee/ListByDepartment.cshtml

The GET form filters the table by department and shows which choice is active. Use it when you want repeatable read traffic through MVC rather than through the JSON API.

@model IEnumerable<MVC_App.Models.Employee>

@{
    ViewData["Title"] = "Employees By Department";
    var selectedDept = ViewBag.SelectedDeptName as string;
}

<h1>Employees By Department</h1>

<form asp-action="ListByDepartment" method="get" class="row g-2 mb-3">
    <div class="col-auto">
        <select name="deptName" asp-items="ViewBag.Departments" class="form-select">
            <option value="">-- All Departments --</option>
        </select>
    </div>
    <div class="col-auto">
        <button type="submit" class="btn btn-primary">Filter</button>
    </div>
</form>

@if (!string.IsNullOrEmpty(selectedDept))
{
    <p>Showing employees in department: <strong>@selectedDept</strong></p>
}

<table class="table">
    <thead>
        <tr>
            <th>@Html.DisplayNameFor(model => model.EmpNo)</th>
            <th>@Html.DisplayNameFor(model => model.EmpName)</th>
            <th>@Html.DisplayNameFor(model => model.Salary)</th>
            <th>@Html.DisplayNameFor(model => model.DeptName)</th>
            <th>@Html.DisplayNameFor(model => model.Designation)</th>
            <th></th>
        </tr>
    </thead>
    <tbody>
@foreach (var item in Model)
{
        <tr>
            <td>@Html.DisplayFor(modelItem => item.EmpNo)</td>
            <td>@Html.DisplayFor(modelItem => item.EmpName)</td>
            <td>@Html.DisplayFor(modelItem => item.Salary)</td>
            <td>@Html.DisplayFor(modelItem => item.DeptName)</td>
            <td>@Html.DisplayFor(modelItem => item.Designation)</td>
            <td>
                <a asp-action="Edit" asp-route-id="@item.EmpNo">Edit</a> |
                <a asp-action="Details" asp-route-id="@item.EmpNo">Details</a> |
                <a asp-action="Delete" asp-route-id="@item.EmpNo">Delete</a>
            </td>
        </tr>
}
    </tbody>
</table>

<div>
    <a asp-action="Index">Back to Full List</a>
</div>

MVC_App.BlazorClient

On the Blazor side, follow a component action into the typed API client. The outgoing handler and error boundary fill the gaps left by ordinary HTTP request logging in an interactive-server application.

MVC_App.BlazorClient/Api/ApiDtos.cs

These local DTOs mirror the backend's department, employee and problem-response contracts. They make typed calls straightforward, but there is no shared contract assembly to keep the two copies synchronized automatically.

using System.ComponentModel.DataAnnotations;

namespace MVC_App.BlazorClient.Api;

/// <summary>
/// Client-side copy of <c>MVC_App/Models/Api/ApiDtos.cs</c>. The validation attributes are
/// repeated verbatim so an <c>EditForm</c> rejects a bad value in the browser with the same
/// message the API would return as a 400, which keeps the two in agreement without a round trip.
///
/// Deliberately a copy rather than a project reference: referencing MVC_App would pull EF Core and
/// Identity into this client, and these DTOs are the published API contract, which is exactly the
/// thing a client is supposed to hold its own copy of.
/// </summary>
public sealed class DepartmentDto
{
    [Range(1, int.MaxValue, ErrorMessage = "DeptNo must be a positive number.")]
    public int DeptNo { get; set; }

    [Required(ErrorMessage = "DeptName is required.")]
    [StringLength(200, MinimumLength = 2, ErrorMessage = "DeptName must be between 2 and 200 characters.")]
    public string DeptName { get; set; } = string.Empty;

    [Required(ErrorMessage = "Location is required.")]
    [StringLength(40, MinimumLength = 2, ErrorMessage = "Location must be between 2 and 40 characters.")]
    public string Location { get; set; } = string.Empty;

    public DepartmentDto Clone() => new()
    {
        DeptNo = DeptNo,
        DeptName = DeptName,
        Location = Location
    };
}

/// <summary>Client-side copy of the <c>/api/employees</c> contract.</summary>
public sealed class EmployeeDto
{
    [Range(1, int.MaxValue, ErrorMessage = "EmpNo must be a positive number.")]
    public int EmpNo { get; set; }

    [Required(ErrorMessage = "EmpName is required.")]
    [StringLength(100, MinimumLength = 2, ErrorMessage = "EmpName must be between 2 and 100 characters.")]
    public string EmpName { get; set; } = string.Empty;

    [Range(1, 10_000_000, ErrorMessage = "Salary must be greater than zero.")]
    public int Salary { get; set; }

    [Required(ErrorMessage = "DeptName is required.")]
    [StringLength(100, MinimumLength = 2, ErrorMessage = "DeptName must be between 2 and 100 characters.")]
    public string DeptName { get; set; } = string.Empty;

    [Required(ErrorMessage = "Designation is required.")]
    [StringLength(100, MinimumLength = 2, ErrorMessage = "Designation must be between 2 and 100 characters.")]
    public string Designation { get; set; } = string.Empty;

    public EmployeeDto Clone() => new()
    {
        EmpNo = EmpNo,
        EmpName = EmpName,
        Salary = Salary,
        DeptName = DeptName,
        Designation = Designation
    };
}

/// <summary>
/// Client-side copy of <c>MVC_App/Diagnostics/ApiProblemDetails.cs</c>, the single JSON error
/// shape every <c>/api/...</c> endpoint returns. <c>TraceId</c> is the useful part: it matches the
/// <c>trace_id</c> on the server's log documents, so a failure shown in this UI can be pasted
/// straight into Kibana.
/// </summary>
public sealed class ApiProblemDetails
{
    public int Status { get; set; }

    public string Title { get; set; } = string.Empty;

    public string? Detail { get; set; }

    public string? Instance { get; set; }

    public string? ErrorCategory { get; set; }

    public string? ErrorType { get; set; }

    public string? TraceId { get; set; }

    public DateTimeOffset? Timestamp { get; set; }
}

MVC_App.BlazorClient/Api/ApiException.cs

ApiException carries the failed status, parsed problem, category and server trace ID together. Components can give the user a useful message and a diagnostic reference without exposing a stack trace.

using MVC_App.BlazorClient.Diagnostics;

namespace MVC_App.BlazorClient.Api;

/// <summary>
/// Thrown when the API answers with a failure status. It carries the status code and the server's
/// own <c>ApiProblemDetails</c> so the UI can show the real reason ("Department name 'HRD-20'
/// already exists") instead of a generic message, and so the exception can be logged under the
/// right <c>error.category</c> rather than being flattened to a 500.
///
/// <see cref="TraceId"/> is the server's trace id for the failed call. Showing it in the UI means
/// a user can report a failure by quoting an id that finds the server-side record in Kibana.
/// </summary>
public sealed class ApiException : Exception
{
    public ApiException(int statusCode, string message, ApiProblemDetails? problem, string? requestUri)
        : base(message)
    {
        StatusCode = statusCode;
        Problem = problem;
        RequestUri = requestUri;
    }

    public int StatusCode { get; }

    public ApiProblemDetails? Problem { get; }

    public string? RequestUri { get; }

    /// <summary>The slug this failure is filed under in ELK, e.g. "conflict", "not_found".</summary>
    public string ErrorCategory => Problem?.ErrorCategory ?? ErrorConditions.ForStatusCode(StatusCode);

    /// <summary>The server-side exception type, when the failure came from a thrown exception.</summary>
    public string? ServerErrorType => Problem?.ErrorType;

    /// <summary>Correlates with <c>trace_id</c> on the server's log and trace documents.</summary>
    public string? TraceId => Problem?.TraceId;

    /// <summary>
    /// The sentence to put in front of a user. Prefers the server's own detail, which is written
    /// for exactly this purpose, and falls back to the title and then the status code.
    /// </summary>
    public string DisplayMessage =>
        Problem?.Detail
        ?? (string.IsNullOrWhiteSpace(Problem?.Title) ? null : Problem!.Title)
        ?? $"The API returned {StatusCode}.";
}

MVC_App.BlazorClient/Api/MvcAppApiClient.cs

The component code doesn't need to repeat JSON serialization and problem-response handling for every operation. This typed client does that work and raises ApiException on failure. It also treats an unexpectedly empty success body as a problem when the caller expects a value.

using System.Net;
using System.Net.Http.Json;
using System.Text.Json;

namespace MVC_App.BlazorClient.Api;

/// <summary>
/// Typed client for the MVC_App JSON API. Every method either returns the deserialized result or
/// throws <see cref="ApiException"/> carrying the server's own problem document, so the pages can
/// render a real error message and the error boundary can log it under the right category.
///
/// The API key is attached by the handler registered in <c>Program.cs</c>, never here, so it stays
/// a configuration concern. Because this is Blazor Server the key lives in server-side
/// configuration and is never sent to the browser.
/// </summary>
public sealed class MvcAppApiClient
{
    private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web);

    private readonly HttpClient _http;
    private readonly ILogger<MvcAppApiClient> _logger;

    public MvcAppApiClient(HttpClient http, ILogger<MvcAppApiClient> logger)
    {
        _http = http;
        _logger = logger;
    }

    // -----------------------------------------------------------------------
    // Departments
    // -----------------------------------------------------------------------

    public Task<IReadOnlyList<DepartmentDto>> GetDepartmentsAsync(string? simulate = null, CancellationToken ct = default)
        => GetListAsync<DepartmentDto>(WithSimulate("api/departments", simulate), ct);

    public Task<DepartmentDto> GetDepartmentAsync(int deptNo, CancellationToken ct = default)
        => GetAsync<DepartmentDto>($"api/departments/{deptNo}", ct);

    public Task<DepartmentDto> CreateDepartmentAsync(DepartmentDto department, CancellationToken ct = default)
        => SendAsync<DepartmentDto>(HttpMethod.Post, "api/departments", department, ct);

    public Task<DepartmentDto> UpdateDepartmentAsync(DepartmentDto department, CancellationToken ct = default)
        => SendAsync<DepartmentDto>(HttpMethod.Put, $"api/departments/{department.DeptNo}", department, ct);

    public Task DeleteDepartmentAsync(int deptNo, CancellationToken ct = default)
        => SendAsync($"api/departments/{deptNo}", ct);

    // -----------------------------------------------------------------------
    // Employees
    // -----------------------------------------------------------------------

    public Task<IReadOnlyList<EmployeeDto>> GetEmployeesAsync(
        string? deptName = null, string? simulate = null, CancellationToken ct = default)
    {
        var uri = "api/employees";
        var query = new List<string>();

        if (!string.IsNullOrWhiteSpace(deptName))
        {
            query.Add($"deptName={Uri.EscapeDataString(deptName)}");
        }

        if (!string.IsNullOrWhiteSpace(simulate))
        {
            query.Add($"simulate={Uri.EscapeDataString(simulate)}");
        }

        if (query.Count > 0)
        {
            uri += "?" + string.Join("&", query);
        }

        return GetListAsync<EmployeeDto>(uri, ct);
    }

    public Task<EmployeeDto> GetEmployeeAsync(int empNo, CancellationToken ct = default)
        => GetAsync<EmployeeDto>($"api/employees/{empNo}", ct);

    public Task<EmployeeDto> CreateEmployeeAsync(EmployeeDto employee, CancellationToken ct = default)
        => SendAsync<EmployeeDto>(HttpMethod.Post, "api/employees", employee, ct);

    public Task<EmployeeDto> UpdateEmployeeAsync(EmployeeDto employee, CancellationToken ct = default)
        => SendAsync<EmployeeDto>(HttpMethod.Put, $"api/employees/{employee.EmpNo}", employee, ct);

    public Task DeleteEmployeeAsync(int empNo, CancellationToken ct = default)
        => SendAsync($"api/employees/{empNo}", ct);

    // -----------------------------------------------------------------------
    // Plumbing
    // -----------------------------------------------------------------------

    private static string WithSimulate(string uri, string? simulate)
        => string.IsNullOrWhiteSpace(simulate) ? uri : $"{uri}?simulate={Uri.EscapeDataString(simulate)}";

    private async Task<IReadOnlyList<T>> GetListAsync<T>(string uri, CancellationToken ct)
    {
        using var response = await _http.GetAsync(uri, ct);
        await ThrowIfFailedAsync(response, uri, ct);

        return await response.Content.ReadFromJsonAsync<List<T>>(JsonOptions, ct) ?? [];
    }

    private async Task<T> GetAsync<T>(string uri, CancellationToken ct)
    {
        using var response = await _http.GetAsync(uri, ct);
        await ThrowIfFailedAsync(response, uri, ct);

        return await ReadRequiredAsync<T>(response, uri, ct);
    }

    private async Task<T> SendAsync<T>(HttpMethod method, string uri, object body, CancellationToken ct)
    {
        using var request = new HttpRequestMessage(method, uri)
        {
            Content = JsonContent.Create(body, options: JsonOptions)
        };

        using var response = await _http.SendAsync(request, ct);
        await ThrowIfFailedAsync(response, uri, ct);

        return await ReadRequiredAsync<T>(response, uri, ct);
    }

    /// <summary>DELETE returns 204 with no body, so there is nothing to deserialize.</summary>
    private async Task SendAsync(string uri, CancellationToken ct)
    {
        using var response = await _http.DeleteAsync(uri, ct);
        await ThrowIfFailedAsync(response, uri, ct);
    }

    private async Task<T> ReadRequiredAsync<T>(HttpResponseMessage response, string uri, CancellationToken ct)
    {
        var value = await response.Content.ReadFromJsonAsync<T>(JsonOptions, ct);

        if (value is null)
        {
            // A 2xx with an empty or unparseable body is still a broken contract, and silently
            // returning null here would surface later as a NullReferenceException with no context.
            throw new ApiException(
                (int)response.StatusCode,
                $"The API returned {(int)response.StatusCode} for {uri} but the body could not be read as {typeof(T).Name}.",
                problem: null,
                requestUri: uri);
        }

        return value;
    }

    /// <summary>
    /// Turns a failure status into an <see cref="ApiException"/>, reading the server's problem
    /// document for the real reason. The document is written by the MVC app's
    /// ExceptionHandlingMiddleware and carries the same error.category and trace id that its log
    /// entry does.
    /// </summary>
    private async Task ThrowIfFailedAsync(HttpResponseMessage response, string uri, CancellationToken ct)
    {
        if (response.IsSuccessStatusCode)
        {
            return;
        }

        var statusCode = (int)response.StatusCode;
        ApiProblemDetails? problem = null;

        try
        {
            problem = await response.Content.ReadFromJsonAsync<ApiProblemDetails>(JsonOptions, ct);
        }
        catch (Exception readFailure) when (readFailure is JsonException or NotSupportedException)
        {
            // Not every failure carries the JSON shape, e.g. a proxy error page or an HTML 502.
            // The status code alone is still enough to report and categorise the failure.
            _logger.LogDebug(readFailure,
                "Response body for the failed call to {Uri} was not an ApiProblemDetails document.", uri);
        }

        var message = problem?.Detail
            ?? problem?.Title
            ?? $"{response.RequestMessage?.Method.Method} {uri} failed with {statusCode} {response.ReasonPhrase}.";

        throw new ApiException(statusCode, message, problem, uri);
    }
}

MVC_App.BlazorClient/appsettings.Development.json

The development settings point to the local API and Collector. Keep the Blazor service name distinct from MVC_App so shared-index searches can separate them. Set SDK export-interval options in the process environment when you want to control their effect.

{
  "DetailedErrors": true,
  "OTEL_SERVICE_NAME": "MVC_App.BlazorClient",
  "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4318",
  "OTEL_METRIC_EXPORT_INTERVAL": 10000,
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "Api": {
    "BaseUrl": "https://localhost:7119",
    "HeaderName": "X-Api-Key",
    "Key": "<REDACTED_API_KEY>"
  },
  "ConnectionStrings": {
    "IdentityConnection": "<REDACTED_CONNECTION_STRING>"
  }
}

MVC_App.BlazorClient/appsettings.json

These settings tell Blazor where to find the API, which header/key to send and which Identity database to use. Replace the redacted connection values privately before running it.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "Api": {
    "BaseUrl": "https://localhost:7119",
    "HeaderName": "X-Api-Key",
    "Key": "<REDACTED_API_KEY>"
  },
  "ConnectionStrings": {
    "IdentityConnection": "<REDACTED_CONNECTION_STRING>"
  }
}

MVC_App.BlazorClient/Components/Account/Login.razor

Sign-in stays on the static-server-rendered path so Identity can set the authentication cookie. The form checks local return URLs, reports a generic failure and logs the outcome. The current flow does not enable lockout on failure, which is worth revisiting for production use.

@page "/account/login"
@using System.ComponentModel.DataAnnotations
@using Microsoft.AspNetCore.Authentication
@using Microsoft.AspNetCore.Components.Endpoints
@using Microsoft.AspNetCore.Identity
@using MVC_App.BlazorClient.Identity

@* Static SSR, so SignInManager can still write the authentication cookie. See App.razor. *@
@attribute [ExcludeFromInteractiveRouting]

@inject SignInManager<ApplicationUser> SignInManager
@inject AuthEventLogger AuthLog
@inject NavigationManager Navigation

<PageTitle>Sign in</PageTitle>

<div class="login-wrap">
    <div class="card login-card">
        <h1>Sign in</h1>
        <p class="muted">
            Use your MVC_App account. Both applications share the one
            <code>MVC_AppIdentity</code> database, so the same credentials work in each.
        </p>

        @if (_errorMessage is { Length: > 0 })
        {
            <div class="alert alert-error">@_errorMessage</div>
        }

        <EditForm Model="Input" method="post" OnValidSubmit="SignInAsync" FormName="login">
            <DataAnnotationsValidator />

            <div class="field">
                <label for="email">Email</label>
                <InputText id="email" @bind-Value="Input.Email" autocomplete="username" />
                <ValidationMessage For="() => Input.Email" />
            </div>

            <div class="field">
                <label for="password">Password</label>
                <InputText id="password" type="password" @bind-Value="Input.Password"
                           autocomplete="current-password" />
                <ValidationMessage For="() => Input.Password" />
            </div>

            <div class="field checkbox">
                <InputCheckbox id="remember" @bind-Value="Input.RememberMe" />
                <label for="remember">Stay signed in</label>
            </div>

            <button type="submit" class="btn btn-primary">Sign in</button>
        </EditForm>
    </div>
</div>

@code {
    [CascadingParameter]
    private HttpContext HttpContext { get; set; } = default!;

    // The form posts back to this same page, so the model has to be rehydrated from the post
    // rather than held in a field across requests. Static SSR keeps no state between them.
    [SupplyParameterFromForm]
    private LoginInput Input { get; set; } = new();

    [SupplyParameterFromQuery]
    private string? ReturnUrl { get; set; }

    private string? _errorMessage;

    protected override async Task OnInitializedAsync()
    {
        if (HttpMethods.IsGet(HttpContext.Request.Method))
        {
            // Clears any half-finished sign-in left over from an earlier attempt, so a fresh visit
            // to the login page always starts from a known state.
            await HttpContext.SignOutAsync(IdentityConstants.ExternalScheme);
        }
    }

    private async Task SignInAsync()
    {
        // lockoutOnFailure stays false to match the MVC app's AccountController, so the two do not
        // disagree about when an account becomes locked.
        var result = await SignInManager.PasswordSignInAsync(
            Input.Email, Input.Password, Input.RememberMe, lockoutOnFailure: false);

        if (result.Succeeded)
        {
            AuthLog.SignInSucceeded(Input.Email);
            Navigation.NavigateTo(SafeReturnUrl(), forceLoad: true);
            return;
        }

        var reason = result switch
        {
            { IsLockedOut: true } => "locked_out",
            { IsNotAllowed: true } => "not_allowed",
            { RequiresTwoFactor: true } => "requires_two_factor",
            _ => "invalid_credentials"
        };

        AuthLog.SignInFailed(Input.Email, reason);

        // Deliberately the same sentence whatever the reason, so the page never reveals whether an
        // account exists. The specific reason is in the log entry for whoever is investigating.
        _errorMessage = "Invalid sign-in attempt.";
    }

    /// <summary>
    /// Only ever redirects within this app. An unchecked returnUrl is an open redirect: a crafted
    /// link could send someone to an external page after a successful sign-in.
    /// </summary>
    private string SafeReturnUrl()
    {
        if (string.IsNullOrWhiteSpace(ReturnUrl))
        {
            return "/";
        }

        // Rejects absolute URLs and protocol-relative ones such as "//evil.example".
        if (Uri.TryCreate(ReturnUrl, UriKind.Absolute, out _) || ReturnUrl.StartsWith("//", StringComparison.Ordinal))
        {
            return "/";
        }

        return ReturnUrl.StartsWith('/') ? ReturnUrl : "/" + ReturnUrl;
    }

    private sealed class LoginInput
    {
        [Required(ErrorMessage = "Email is required.")]
        [EmailAddress(ErrorMessage = "Enter a valid email address.")]
        public string Email { get; set; } = string.Empty;

        [Required(ErrorMessage = "Password is required.")]
        [DataType(DataType.Password)]
        public string Password { get; set; } = string.Empty;

        public bool RememberMe { get; set; }
    }
}

MVC_App.BlazorClient/Components/Account/Logout.razor

Signing out also needs an HTTP response to clear the cookie. This POST-based page clears the session, logs the action and redirects the browser.

@page "/account/logout"
@using Microsoft.AspNetCore.Components.Endpoints
@using Microsoft.AspNetCore.Identity
@using MVC_App.BlazorClient.Identity

@* Static SSR for the same reason as Login: clearing the cookie is a response header. *@
@attribute [ExcludeFromInteractiveRouting]

@inject SignInManager<ApplicationUser> SignInManager
@inject AuthEventLogger AuthLog
@inject NavigationManager Navigation

<PageTitle>Sign out</PageTitle>

<div class="login-wrap">
    <div class="card login-card">
        <h1>Sign out</h1>
        <p>You are signing out of the Blazor client.</p>
        <p class="muted">
            This does not sign you out of the MVC app. The two share a user database but issue
            their own cookies, so each session ends separately.
        </p>

        @*
            A POST, not a link. A GET that changes state can be triggered by anything that fetches
            the URL, including a prefetch or an image tag on another site, which would sign people
            out unexpectedly. The antiforgery token is added automatically by the framework.
        *@
        <EditForm Model="this" method="post" OnValidSubmit="SignOutAsync" FormName="logout">
            <button type="submit" class="btn btn-primary">Sign out</button>
            <a class="btn" href="">Cancel</a>
        </EditForm>
    </div>
</div>

@code {
    [CascadingParameter]
    private HttpContext HttpContext { get; set; } = default!;

    private async Task SignOutAsync()
    {
        var userName = HttpContext.User.Identity?.Name ?? "(unknown)";

        await SignInManager.SignOutAsync();
        AuthLog.SignedOut(userName);

        Navigation.NavigateTo("account/login", forceLoad: true);
    }
}

MVC_App.BlazorClient/Components/Account/RedirectToLogin.razor

An anonymous visitor is sent to sign-in with a return URL, so the application can bring them back to the page they originally requested.

@inject NavigationManager Navigation

@*
    Sends an anonymous visitor to the login page, carrying where they were trying to go so they
    land back there after signing in.

    forceLoad is required: the login page is static SSR (see App.razor), so it has to be fetched
    as a real navigation rather than patched in over the existing circuit.
*@

@code {
    protected override void OnInitialized()
    {
        var returnUrl = Navigation.ToBaseRelativePath(Navigation.Uri);

        Navigation.NavigateTo(
            $"account/login?returnUrl={Uri.EscapeDataString(returnUrl)}",
            forceLoad: true);
    }
}

MVC_App.BlazorClient/Components/App.razor

The application shell loads the assets and Blazor scripts and chooses the render mode. Most pages become interactive on the server, while excluded account pages stay on the normal HTTP path so sign-in and sign-out can change cookies.

@using Microsoft.AspNetCore.Components.Endpoints

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <base href="/" />
    <ResourcePreloader />
    <link rel="stylesheet" href="@Assets["app.css"]" />
    <link rel="stylesheet" href="@Assets["MVC_App.BlazorClient.styles.css"]" />
    <ImportMap />
    <HeadOutlet @rendermode="RenderModeForPage" />
</head>

<body>
    <Routes @rendermode="RenderModeForPage" />
    <ReconnectModal />
    <script src="@Assets["_framework/blazor.web.js"]"></script>
</body>

</html>

@code {
    [CascadingParameter]
    private HttpContext HttpContext { get; set; } = default!;

    /// <summary>
    /// Interactive for every page except the ones marked <c>[ExcludeFromInteractiveRouting]</c>,
    /// which render as static SSR instead.
    ///
    /// Signing in has to write an authentication cookie, and a cookie is a response header. An
    /// interactive component renders over the SignalR circuit, long after the response headers
    /// were sent, so SignInManager cannot set the cookie from one. The sign-in pages therefore
    /// opt out of interactivity and post a real form, which is the only point in the request where
    /// the response is still writable.
    /// </summary>
    private IComponentRenderMode? RenderModeForPage =>
        HttpContext.AcceptsInteractiveRouting() ? RenderMode.InteractiveServer : null;
}

MVC_App.BlazorClient/Components/Layout/MainLayout.razor

Alongside the navigation and account controls, the layout wraps page content in an ErrorBoundary. It can show diagnostic information and let the user recover after a component failure. That matters in Blazor because the original page request may be long finished when a later event throws.

@inherits LayoutComponentBase

<div class="page">
    <nav class="sidebar">
        <div class="brand">
            <span class="brand-name">MVC_App</span>
            <span class="brand-sub">Blazor client</span>
        </div>
        <NavLink class="nav-item" href="" Match="NavLinkMatch.All">Home</NavLink>
        <NavLink class="nav-item" href="departments">Departments</NavLink>
        <NavLink class="nav-item" href="employees">Employees</NavLink>
        <NavLink class="nav-item" href="diagnostics">Diagnostics</NavLink>

        @*
            AuthorizeView reads the cascading AuthenticationState, so this re-renders by itself
            when the user signs in or out. No manual state handling needed.
        *@
        <div class="nav-account">
            <AuthorizeView>
                <Authorized>
                    <span class="nav-user" title="@context.User.Identity?.Name">
                        @context.User.Identity?.Name
                    </span>
                    <a class="nav-item" href="account/logout">Sign out</a>
                </Authorized>
                <NotAuthorized>
                    <a class="nav-item" href="account/login">Sign in</a>
                </NotAuthorized>
            </AuthorizeView>
        </div>
    </nav>

    <main>
        @*
            Catches anything a component throws while rendering, including from OnInitializedAsync
            and from a button handler. LoggingErrorBoundaryLogger writes it into mvcapp-logs-* with
            the ELK field set before this markup is shown. Without the boundary an exception tears
            down the SignalR circuit and the user just sees the reconnect modal.
        *@
        <ErrorBoundary @ref="_errorBoundary" @key="@_boundaryKey">
            <ChildContent>
                @Body
            </ChildContent>
            <ErrorContent Context="exception">
                <div class="alert alert-error">
                    <h2>Something went wrong</h2>
                    <p>@DescribeFor(exception)</p>
                    @if (TraceIdFor(exception) is { Length: > 0 } traceId)
                    {
                        <p class="trace">
                            Trace id <code>@traceId</code> &mdash; paste this into Kibana as
                            <code>trace_id : "@traceId"</code> to see the server-side record.
                        </p>
                    }
                    <button class="btn" @onclick="Recover">Try again</button>
                </div>
            </ErrorContent>
        </ErrorBoundary>
    </main>
</div>

<div id="blazor-error-ui" data-nosnippet>
    An unhandled error has occurred.
    <a href="." class="reload">Reload</a>
    <span class="dismiss">🗙</span>
</div>

@code {
    private ErrorBoundary? _errorBoundary;

    // Changing the key forces a fresh boundary, which is what actually clears a latched error.
    // Recover() alone resets the boundary but leaves the failed component tree in place.
    private int _boundaryKey;

    private static string DescribeFor(Exception exception) => exception switch
    {
        MVC_App.BlazorClient.Api.ApiException apiException =>
            $"{apiException.DisplayMessage} (HTTP {apiException.StatusCode}, {apiException.ErrorCategory})",
        _ => exception.Message
    };

    private static string? TraceIdFor(Exception exception) =>
        exception is MVC_App.BlazorClient.Api.ApiException apiException ? apiException.TraceId : null;

    private void Recover()
    {
        _boundaryKey++;
        _errorBoundary?.Recover();
    }
}

 

MVC_App.BlazorClient/Components/Pages/Departments.razor

The department grid uses the typed API client for its CRUD operations. Expected API errors are caught and displayed in place; unexpected exceptions can reach the layout boundary. Follow the outgoing call log and backend trace to see what happened beyond the component.

@page "/departments"
@using MVC_App.BlazorClient.Api
@attribute [Microsoft.AspNetCore.Authorization.Authorize]
@using MVC_App.BlazorClient.Components.Shared
@inject MvcAppApiClient Api
@inject ILogger<Departments> Logger

<PageTitle>Departments</PageTitle>

<h1>Departments</h1>
<p class="lede">
    Every button on this page calls <code>/api/departments</code> on MVC_App. Each call is logged
    into <code>mvcapp-logs-*</code> by <code>ApiLoggingHandler</code>, and the API logs its own
    side of the same call under the same <code>trace_id</code>.
</p>

<ApiErrorAlert Exception="_error" OnDismiss="() => _error = null" />

<div class="toolbar">
    <button class="btn btn-primary" @onclick="StartCreate" disabled="@_busy">New department</button>
    <button class="btn" @onclick="() => LoadAsync()" disabled="@_busy">Reload</button>

    @* Drives the API's own ?simulate= switch, so each error condition can be produced on demand
       and watched arriving in Kibana. Values come from ApiControllerBase.ApplySimulatedFault. *@
    <span class="spacer"></span>
    <label class="simulate">
        Simulate fault
        <select @bind="_simulate" @bind:after="() => LoadAsync()" disabled="@_busy">
            <option value="">none</option>
            <option value="400">400 validation</option>
            <option value="401">401 unauthorized</option>
            <option value="403">403 forbidden</option>
            <option value="404">404 not found</option>
            <option value="409">409 conflict</option>
            <option value="500">500 server error</option>
            <option value="503">503 unavailable</option>
        </select>
    </label>
</div>

@if (_editing is not null)
{
    <div class="card">
        <h2>@(_isNew ? "New department" : $"Edit department {_editing.DeptNo}")</h2>
        <EditForm Model="_editing" OnValidSubmit="SaveAsync" FormName="department">
            <DataAnnotationsValidator />
            <div class="field">
                <label>DeptNo</label>
                @* The primary key is the route on update, so it is fixed once the row exists. *@
                <InputNumber @bind-Value="_editing.DeptNo" disabled="@(!_isNew)" />
                <ValidationMessage For="() => _editing.DeptNo" />
            </div>
            <div class="field">
                <label>DeptName</label>
                <InputText @bind-Value="_editing.DeptName" />
                <ValidationMessage For="() => _editing.DeptName" />
            </div>
            <div class="field">
                <label>Location</label>
                <InputText @bind-Value="_editing.Location" />
                <ValidationMessage For="() => _editing.Location" />
            </div>
            <div class="actions">
                <button type="submit" class="btn btn-primary" disabled="@_busy">Save</button>
                <button type="button" class="btn" @onclick="CancelEdit" disabled="@_busy">Cancel</button>
            </div>
        </EditForm>
    </div>
}

@if (_departments is null)
{
    <p class="muted">Loading…</p>
}
else if (_departments.Count == 0)
{
    <p class="muted">No departments.</p>
}
else
{
    <table class="grid">
        <thead>
            <tr>
                <th>DeptNo</th>
                <th>DeptName</th>
                <th>Location</th>
                <th class="right">Actions</th>
            </tr>
        </thead>
        <tbody>
            @foreach (var department in _departments)
            {
                <tr @key="department.DeptNo">
                    <td>@department.DeptNo</td>
                    <td>@department.DeptName</td>
                    <td>@department.Location</td>
                    <td class="right">
                        <button class="btn btn-sm" @onclick="() => StartEdit(department)" disabled="@_busy">Edit</button>
                        <button class="btn btn-sm btn-danger" @onclick="() => DeleteAsync(department.DeptNo)" disabled="@_busy">Delete</button>
                    </td>
                </tr>
            }
        </tbody>
    </table>
    <p class="muted">@_departments.Count department(s).</p>
}

@code {
    private IReadOnlyList<DepartmentDto>? _departments;
    private DepartmentDto? _editing;
    private ApiException? _error;
    private bool _isNew;
    private bool _busy;
    private string _simulate = string.Empty;

    protected override Task OnInitializedAsync() => LoadAsync();

    private async Task LoadAsync()
    {
        // Caught rather than thrown so the page stays usable and the user can clear the simulated
        // fault. The failure is still logged: ApiLoggingHandler records the HTTP call, and the
        // explicit LogWarning below records it as an application-level event with its category.
        await RunAsync(async () =>
        {
            _departments = await Api.GetDepartmentsAsync(string.IsNullOrEmpty(_simulate) ? null : _simulate);
        });
    }

    private void StartCreate()
    {
        _isNew = true;
        _editing = new DepartmentDto();
        _error = null;
    }

    private void StartEdit(DepartmentDto department)
    {
        _isNew = false;
        // Edit a copy, so cancelling does not leave the grid showing half-typed values.
        _editing = department.Clone();
        _error = null;
    }

    private void CancelEdit()
    {
        _editing = null;
        _error = null;
    }

    private async Task SaveAsync()
    {
        if (_editing is null)
        {
            return;
        }

        var department = _editing;
        var created = _isNew;

        await RunAsync(async () =>
        {
            if (created)
            {
                await Api.CreateDepartmentAsync(department);
            }
            else
            {
                await Api.UpdateDepartmentAsync(department);
            }

            _editing = null;
            await LoadAsync();
        });
    }

    private async Task DeleteAsync(int deptNo)
    {
        await RunAsync(async () =>
        {
            await Api.DeleteDepartmentAsync(deptNo);

            if (_editing?.DeptNo == deptNo)
            {
                _editing = null;
            }

            await LoadAsync();
        });
    }

    /// <summary>
    /// Runs an API operation, turning a failure into a visible alert and a log entry instead of a
    /// torn-down circuit. Anything that is not an ApiException is left to propagate to the
    /// ErrorBoundary in MainLayout, because an unexpected fault is not something this page can
    /// meaningfully recover from.
    /// </summary>
    private async Task RunAsync(Func<Task> operation)
    {
        _busy = true;
        _error = null;

        try
        {
            await operation();
        }
        catch (ApiException apiException)
        {
            _error = apiException;

            Logger.LogWarning(apiException,
                "Departments page: {Method} failed with {StatusCode} [{ErrorCategory}] trace {TraceId}",
                nameof(RunAsync), apiException.StatusCode, apiException.ErrorCategory, apiException.TraceId);
        }
        finally
        {
            _busy = false;
        }
    }
}

MVC_App.BlazorClient/Components/Pages/Diagnostics.razor

The diagnostic buttons let you compare a handled API failure, an uncaught component error and an unreachable endpoint. The unreachable case creates a separate HttpClient, bypassing ApiLoggingHandler. Keep that detail in mind if you can't find the handler's usual structured event for that call.

@page "/diagnostics"
@using MVC_App.BlazorClient.Api
@attribute [Microsoft.AspNetCore.Authorization.Authorize]
@inject MvcAppApiClient Api
@inject ILogger<Diagnostics> Logger

<PageTitle>Diagnostics</PageTitle>

<h1>Diagnostics</h1>
<p class="lede">
    Produces client-side failures on demand, to prove each logging path reaches
    <code>mvcapp-logs-*</code>. The grids' <strong>Simulate fault</strong> dropdowns cover the
    server-side conditions; these cover the ones that happen in this app.
</p>

<div class="card">
    <h2>Uncaught component exception</h2>
    <p>
        Throws straight out of a button handler. Nothing catches it, so it reaches the
        <code>ErrorBoundary</code> in the layout and is logged by
        <code>LoggingErrorBoundaryLogger</code> as a 500.
    </p>
    <p class="muted">Find it with <code>log_attributes.error.boundary : true</code></p>
    <button class="btn btn-danger" @onclick="ThrowUncaught">Throw an uncaught exception</button>
</div>

<div class="card">
    <h2>Unreachable API</h2>
    <p>
        Calls a port nothing is listening on. No response ever arrives, so
        <code>ApiLoggingHandler</code> logs it as a 503 with the socket exception attached, which
        is the same <code>service_unavailable</code> category the MVC app uses for a dead
        dependency.
    </p>
    <p class="muted">Find it with <code>log_attributes.error.category : "service_unavailable"</code></p>
    <button class="btn btn-danger" @onclick="CallUnreachableAsync" disabled="@_busy">Call a dead endpoint</button>
</div>

<div class="card">
    <h2>Handled API failure</h2>
    <p>
        Requests a department that does not exist. The API answers 404 with a problem document, the
        client turns it into an <code>ApiException</code>, and the page shows it below rather than
        letting it reach the boundary.
    </p>
    <p class="muted">Find it with <code>log_attributes.error.category : "not_found"</code></p>
    <button class="btn" @onclick="CallMissingAsync" disabled="@_busy">Request a missing department</button>
</div>

@if (_result is { Length: > 0 })
{
    <div class="alert">@_result</div>
}

@code {
    private string? _result;
    private bool _busy;

    private void ThrowUncaught()
        => throw new InvalidOperationException(
            "Deliberate client-side failure raised from the Diagnostics page.");

    private async Task CallUnreachableAsync()
    {
        _busy = true;
        _result = null;

        try
        {
            using var http = new HttpClient { BaseAddress = new Uri("http://localhost:1") };
            await http.GetAsync("api/departments");
        }
        catch (Exception exception)
        {
            // Logged here as well as by the handler, because this HttpClient is a local throwaway
            // and so does not run through ApiLoggingHandler.
            Logger.LogError(exception,
                "Diagnostics: call to an unreachable endpoint failed [{ErrorCategory}]: {Message}",
                MVC_App.BlazorClient.Diagnostics.ErrorConditions.Unavailable, exception.Message);

            _result = $"Failed as expected: {exception.GetType().Name} - {exception.Message}";
        }
        finally
        {
            _busy = false;
        }
    }

    private async Task CallMissingAsync()
    {
        _busy = true;
        _result = null;

        try
        {
            await Api.GetDepartmentAsync(987654);
            _result = "Unexpected: department 987654 exists.";
        }
        catch (ApiException apiException)
        {
            _result = $"Handled {apiException.StatusCode} [{apiException.ErrorCategory}]: " +
                      $"{apiException.DisplayMessage} (trace {apiException.TraceId})";
        }
        finally
        {
            _busy = false;
        }
    }
}

MVC_App.BlazorClient/Components/Pages/Employees.razor

This grid loads department choices as well as employee data, so one screen action can result in several API calls. It displays known API failures inline and leaves unexpected exceptions to the boundary. That distinction explains why not every failed call produces a component-boundary log.

@page "/employees"
@using MVC_App.BlazorClient.Api
@attribute [Microsoft.AspNetCore.Authorization.Authorize]
@using MVC_App.BlazorClient.Components.Shared
@inject MvcAppApiClient Api
@inject ILogger<Employees> Logger

<PageTitle>Employees</PageTitle>

<h1>Employees</h1>
<p class="lede">
    CRUD against <code>/api/employees</code>. An employee must belong to a department that already
    exists: the API answers a write with an unknown <code>DeptName</code> with a 404
    (<code>DeptNameNotFoundException</code>), so both the filter and the form offer only real
    department names. Use <strong>Simulate fault</strong> to produce error conditions on demand.
</p>

<ApiErrorAlert Exception="_error" OnDismiss="() => _error = null" />

<div class="toolbar">
    <button class="btn btn-primary" @onclick="StartCreate" disabled="@_busy">New employee</button>
    <button class="btn" @onclick="ReloadAllAsync" disabled="@_busy">Reload</button>

    <label class="filter">
        Department
        @* Filtering by a name that does not exist is also a 404 from the API, not an empty list. *@
        <select @bind="_deptFilter" @bind:after="() => LoadAsync()" disabled="@_busy">
            <option value="">all departments</option>
            @foreach (var department in _departmentNames)
            {
                <option value="@department">@department</option>
            }
        </select>
    </label>

    <span class="spacer"></span>
    <label class="simulate">
        Simulate fault
        <select @bind="_simulate" @bind:after="() => LoadAsync()" disabled="@_busy">
            <option value="">none</option>
            <option value="400">400 validation</option>
            <option value="401">401 unauthorized</option>
            <option value="403">403 forbidden</option>
            <option value="404">404 not found</option>
            <option value="409">409 conflict</option>
            <option value="500">500 server error</option>
            <option value="503">503 unavailable</option>
        </select>
    </label>
</div>

@if (_editing is not null)
{
    <div class="card">
        <h2>@(_isNew ? "New employee" : $"Edit employee {_editing.EmpNo}")</h2>
        <EditForm Model="_editing" OnValidSubmit="SaveAsync" FormName="employee">
            <DataAnnotationsValidator />
            <div class="field">
                <label>EmpNo</label>
                <InputNumber @bind-Value="_editing.EmpNo" disabled="@(!_isNew)" />
                <ValidationMessage For="() => _editing.EmpNo" />
            </div>
            <div class="field">
                <label>EmpName</label>
                <InputText @bind-Value="_editing.EmpName" />
                <ValidationMessage For="() => _editing.EmpName" />
            </div>
            <div class="field">
                <label>Salary</label>
                <InputNumber @bind-Value="_editing.Salary" />
                <ValidationMessage For="() => _editing.Salary" />
            </div>
            <div class="field">
                <label>DeptName</label>
                @*
                    A select, not a text box. The API rejects an employee whose DeptName is not an
                    existing department row with a 404 DeptNameNotFoundException
                    (EmployeesApiController.cs:99 on create, :139 on update), so a free-text field
                    turns every typo into a failed save. Offering only real names removes the
                    whole class of error.
                *@
                <InputSelect @bind-Value="_editing.DeptName">
                    <option value="">-- select a department --</option>
                    @foreach (var department in _departmentNames)
                    {
                        <option value="@department">@department</option>
                    }
                </InputSelect>
                <ValidationMessage For="() => _editing.DeptName" />
                @if (_departmentNames.Count == 0)
                {
                    <p class="muted">No departments loaded, so an employee cannot be saved yet. Create a department first.</p>
                }
            </div>
            <div class="field">
                <label>Designation</label>
                <InputText @bind-Value="_editing.Designation" />
                <ValidationMessage For="() => _editing.Designation" />
            </div>
            <div class="actions">
                <button type="submit" class="btn btn-primary" disabled="@_busy">Save</button>
                <button type="button" class="btn" @onclick="CancelEdit" disabled="@_busy">Cancel</button>
            </div>
        </EditForm>
    </div>
}

@if (_employees is null)
{
    <p class="muted">Loading…</p>
}
else if (_employees.Count == 0)
{
    <p class="muted">No employees.</p>
}
else
{
    <table class="grid">
        <thead>
            <tr>
                <th>EmpNo</th>
                <th>EmpName</th>
                <th>Designation</th>
                <th>DeptName</th>
                <th class="right">Salary</th>
                <th class="right">Actions</th>
            </tr>
        </thead>
        <tbody>
            @foreach (var employee in _employees)
            {
                <tr @key="employee.EmpNo">
                    <td>@employee.EmpNo</td>
                    <td>@employee.EmpName</td>
                    <td>@employee.Designation</td>
                    <td>@employee.DeptName</td>
                    <td class="right">@employee.Salary.ToString("N0")</td>
                    <td class="right">
                        <button class="btn btn-sm" @onclick="() => StartEdit(employee)" disabled="@_busy">Edit</button>
                        <button class="btn btn-sm btn-danger" @onclick="() => DeleteAsync(employee.EmpNo)" disabled="@_busy">Delete</button>
                    </td>
                </tr>
            }
        </tbody>
    </table>
    <p class="muted">@_employees.Count employee(s).</p>
}

@code {
    private IReadOnlyList<EmployeeDto>? _employees;
    private IReadOnlyList<string> _departmentNames = [];
    private EmployeeDto? _editing;
    private ApiException? _error;
    private bool _isNew;
    private bool _busy;
    private string _deptFilter = string.Empty;
    private string _simulate = string.Empty;

    protected override async Task OnInitializedAsync()
    {
        await LoadDepartmentNamesAsync();
        await LoadAsync();
    }

    /// <summary>
    /// The set of DeptName values the API will accept on a write. Loaded once on init and after a
    /// save, rather than per keystroke, because it changes only when a department is added.
    /// </summary>
    private async Task LoadDepartmentNamesAsync()
    {
        await RunAsync(async () =>
        {
            var departments = await Api.GetDepartmentsAsync();
            _departmentNames = departments
                .Select(department => department.DeptName)
                .OrderBy(name => name, StringComparer.OrdinalIgnoreCase)
                .ToList();
        });
    }

    /// <summary>Refreshes the department list as well, so one added elsewhere becomes selectable.</summary>
    private async Task ReloadAllAsync()
    {
        await LoadDepartmentNamesAsync();
        await LoadAsync();
    }

    private async Task LoadAsync()
    {
        await RunAsync(async () =>
        {
            _employees = await Api.GetEmployeesAsync(
                string.IsNullOrWhiteSpace(_deptFilter) ? null : _deptFilter,
                string.IsNullOrEmpty(_simulate) ? null : _simulate);
        });
    }

    private void StartCreate()
    {
        _isNew = true;
        _editing = new EmployeeDto();
        _error = null;
    }

    private void StartEdit(EmployeeDto employee)
    {
        _isNew = false;
        _editing = employee.Clone();
        _error = null;
    }

    private void CancelEdit()
    {
        _editing = null;
        _error = null;
    }

    private async Task SaveAsync()
    {
        if (_editing is null)
        {
            return;
        }

        var employee = _editing;
        var created = _isNew;

        await RunAsync(async () =>
        {
            if (created)
            {
                await Api.CreateEmployeeAsync(employee);
            }
            else
            {
                await Api.UpdateEmployeeAsync(employee);
            }

            _editing = null;
            await LoadAsync();
        });
    }

    private async Task DeleteAsync(int empNo)
    {
        await RunAsync(async () =>
        {
            await Api.DeleteEmployeeAsync(empNo);

            if (_editing?.EmpNo == empNo)
            {
                _editing = null;
            }

            await LoadAsync();
        });
    }

    private async Task RunAsync(Func<Task> operation)
    {
        _busy = true;
        _error = null;

        try
        {
            await operation();
        }
        catch (ApiException apiException)
        {
            _error = apiException;

            Logger.LogWarning(apiException,
                "Employees page: call failed with {StatusCode} [{ErrorCategory}] trace {TraceId}",
                apiException.StatusCode, apiException.ErrorCategory, apiException.TraceId);
        }
        finally
        {
            _busy = false;
        }
    }
}

MVC_App.BlazorClient/Components/Pages/Error.razor

The error page shows the available Activity or HTTP request identifier. Use it as a starting point for investigation, while remembering that a later circuit event may not share the original page request's trace.

@page "/Error"
@using System.Diagnostics

<PageTitle>Error</PageTitle>

<h1 class="text-danger">Error.</h1>
<h2 class="text-danger">An error occurred while processing your request.</h2>

@if (ShowRequestId)
{
    <p>
        <strong>Request ID:</strong> <code>@RequestId</code>
    </p>
}

<h3>Development Mode</h3>
<p>
    Swapping to <strong>Development</strong> environment will display more detailed information about the error that occurred.
</p>
<p>
    <strong>The Development environment shouldn't be enabled for deployed applications.</strong>
    It can result in displaying sensitive information from exceptions to end users.
    For local debugging, enable the <strong>Development</strong> environment by setting the <strong>ASPNETCORE_ENVIRONMENT</strong> environment variable to <strong>Development</strong>
    and restarting the app.
</p>

@code{
    [CascadingParameter]
    private HttpContext? HttpContext { get; set; }

    private string? RequestId { get; set; }
    private bool ShowRequestId => !string.IsNullOrEmpty(RequestId);

    protected override void OnInitialized() =>
        RequestId = Activity.Current?.Id ?? HttpContext?.TraceIdentifier;
}

MVC_App.BlazorClient/Components/Pages/Home.razor

The home page introduces the sample's logging paths and trace correlation before the user starts exercising the CRUD and diagnostic screens.

@page "/"

<PageTitle>MVC_App Blazor client</PageTitle>

<h1>MVC_App Blazor client</h1>

<p class="lede">
    A Blazor Server front end for the MVC_App JSON API. Every CRUD action here goes out over HTTP
    to <code>/api/departments</code> or <code>/api/employees</code>, and both sides of each call
    are logged into the same <code>mvcapp-logs-*</code> indices.
</p>

<div class="card">
    <h2>What gets logged</h2>
    <p>Three separate paths, because in Blazor Server a single middleware cannot see all of them.</p>
    <table class="grid">
        <thead>
            <tr><th>What</th><th>Written by</th><th>Find it with</th></tr>
        </thead>
        <tbody>
            <tr>
                <td>Outbound API calls</td>
                <td><code>ApiLoggingHandler</code></td>
                <td><code>log_attributes.http.request.direction : "outbound"</code></td>
            </tr>
            <tr>
                <td>Inbound page requests</td>
                <td><code>RequestLoggingMiddleware</code></td>
                <td><code>log_attributes.url.path : *</code></td>
            </tr>
            <tr>
                <td>Component exceptions</td>
                <td><code>LoggingErrorBoundaryLogger</code></td>
                <td><code>log_attributes.error.boundary : true</code></td>
            </tr>
        </tbody>
    </table>
    <p class="muted">
        Everything from this app carries <code>log_attributes.app.component : "blazor-client"</code>
        and <code>service_name : "MVC_App.BlazorClient"</code>, so it can be separated from the MVC
        app's own entries in the shared index.
    </p>
</div>

<div class="card">
    <h2>Correlating a call across both apps</h2>
    <p>
        A failed call shows its trace id in the error alert. One id in Kibana returns the click,
        the outbound request, the API's own request log, its SQL, and any exception it threw:
    </p>
    <pre><code>trace_id : "paste-the-id-here"</code></pre>
    <p class="muted">
        Allow 10 to 30 seconds. The Collector batches on a 5 second timeout before Logstash picks
        the file up and Elasticsearch refreshes.
    </p>
</div>

<div class="card">
    <h2>Producing errors on purpose</h2>
    <p>
        Both grids have a <strong>Simulate fault</strong> dropdown wired to the API's own
        <code>?simulate=</code> switch, which covers 400, 401, 403, 404, 409, 500 and 503. The
        <a href="diagnostics">Diagnostics</a> page throws client-side failures instead, so you can
        see the difference between an API error and an exception raised inside a component.
    </p>
</div>

 MVC_App.BlazorClient/Components/Shared/ApiErrorAlert.razor

This dismissible alert keeps API failures consistent across the CRUD screens. Showing the category and server trace ID gives a user something useful to report without displaying the whole exception.

@*
    Renders a failed API call in the shape the user needs: the server's own explanation, the
    condition slug it was filed under, and the trace id that finds the server-side log entry.

    This is for errors a page chose to catch and show in place (a 409 on save, a 404 on load).
    Anything a page does not catch reaches the ErrorBoundary in MainLayout instead. Both end up in
    mvcapp-logs-*; the difference is only whether the page stays usable.
*@

@if (Exception is not null)
{
    <div class="alert alert-error">
        <strong>@Title</strong>
        <p>@Exception.DisplayMessage</p>
        <p class="meta">
            HTTP @Exception.StatusCode
            &middot; <code>@Exception.ErrorCategory</code>
            @if (Exception.ServerErrorType is { Length: > 0 } errorType)
            {
                <text> &middot; <code>@errorType</code></text>
            }
        </p>
        @if (Exception.TraceId is { Length: > 0 } traceId)
        {
            <p class="trace">
                Kibana: <code>trace_id : "@traceId"</code>
            </p>
        }
        @if (OnDismiss.HasDelegate)
        {
            <button class="btn btn-sm" @onclick="OnDismiss">Dismiss</button>
        }
    </div>
}

@code {
    [Parameter] public MVC_App.BlazorClient.Api.ApiException? Exception { get; set; }

    [Parameter] public string Title { get; set; } = "The API rejected that request";

    [Parameter] public EventCallback OnDismiss { get; set; }
}

MVC_App.BlazorClient/Diagnostics/ErrorConditions.cs

The client uses the same status-to-category vocabulary as the server. Keep the two definitions aligned when adding a new error case; otherwise a shared dashboard filter can quietly miss one side of the call.

namespace MVC_App.BlazorClient.Diagnostics;

/// <summary>
/// The canonical error-condition slugs stamped onto every log entry as <c>error.category</c>.
///
/// This is a deliberate copy of <c>MVC_App/Diagnostics/ErrorConditions.cs</c> rather than a project
/// reference. Referencing the MVC project would drag EF Core, Identity and the whole data layer
/// into this client for the sake of nine string constants. The values must stay identical to the
/// MVC app's, because both write into the same <c>mvcapp-logs-*</c> index and the saved Kibana
/// queries filter on these exact slugs.
/// </summary>
public static class ErrorConditions
{
    /// <summary>2xx/3xx: the request completed as intended.</summary>
    public const string None = "none";

    /// <summary>400: the request itself was malformed or failed model validation.</summary>
    public const string Validation = "validation_error";

    /// <summary>401: the caller is unauthenticated (missing/invalid credentials or API key).</summary>
    public const string Unauthorized = "unauthorized";

    /// <summary>403: the caller is authenticated but not allowed to perform this operation.</summary>
    public const string Forbidden = "forbidden";

    /// <summary>404: the requested route, or the entity behind it, does not exist.</summary>
    public const string NotFound = "not_found";

    /// <summary>409: the write conflicts with data that already exists.</summary>
    public const string Conflict = "conflict";

    /// <summary>Any other 4xx that isn't one of the named conditions above.</summary>
    public const string ClientError = "client_error";

    /// <summary>500: an unhandled or deliberately raised failure inside the app.</summary>
    public const string ServerError = "server_error";

    /// <summary>502/503/504: a dependency the app relies on was unavailable.</summary>
    public const string Unavailable = "service_unavailable";

    /// <summary>Maps an HTTP status code onto the condition slug it should be reported as.</summary>
    public static string ForStatusCode(int statusCode) => statusCode switch
    {
        StatusCodes.Status400BadRequest or StatusCodes.Status422UnprocessableEntity => Validation,
        StatusCodes.Status401Unauthorized => Unauthorized,
        StatusCodes.Status403Forbidden => Forbidden,
        StatusCodes.Status404NotFound or StatusCodes.Status410Gone => NotFound,
        StatusCodes.Status409Conflict => Conflict,
        StatusCodes.Status502BadGateway
            or StatusCodes.Status503ServiceUnavailable
            or StatusCodes.Status504GatewayTimeout => Unavailable,
        >= 500 => ServerError,
        >= 400 => ClientError,
        _ => None
    };

    /// <summary>
    /// True for the status codes that represent a failed request. 3xx redirects and 2xx
    /// successes are not failures, so <c>event.outcome</c> stays "success" for them.
    /// </summary>
    public static bool IsFailure(int statusCode) => statusCode >= 400;
}

MVC_App.BlazorClient/Program.cs

The Blazor host combines interactive-server components, the shared account database and a typed client for the MVC API. Its own cookie keeps the two web sessions separate. Request logging wraps the pipeline, while the HTTP handler and error-boundary logger cover outgoing calls and component failures. The development-only certificate bypass is worth noticing before you reuse this configuration elsewhere.

using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.AspNetCore.Components.Server;
using Microsoft.AspNetCore.Components.Web;
using Microsoft.AspNetCore.Identity;
using Microsoft.EntityFrameworkCore;
using MVC_App.BlazorClient.Api;
using MVC_App.BlazorClient.Components;
using MVC_App.BlazorClient.Identity;
using MVC_App.BlazorClient.Telemetry;

var builder = WebApplication.CreateBuilder(args);

// Exports logs, traces and metrics over OTLP to the same Collector as MVC_App, so this app's
// entries land in the same mvcapp-logs-* indices. See Telemetry/TelemetryExtensions.cs.
builder.AddApplicationTelemetry();

builder.Services.AddRazorComponents()
    .AddInteractiveServerComponents();

// -----------------------------------------------------------------------------
// Authentication
//
// Points at the same MVC_AppIdentity database as the MVC app, so one account works in both.
// Nothing here creates or migrates the schema: MVC_App owns the migrations, this app only reads
// and writes the existing tables.
// -----------------------------------------------------------------------------
builder.Services.AddDbContext<IdentityContext>(options =>
    options.UseSqlServer(builder.Configuration.GetConnectionString("IdentityConnection")));

builder.Services.AddIdentity<ApplicationUser, IdentityRole>(options =>
    {
        // Matched to the MVC app's rules. If the two disagreed, a password acceptable to one app
        // would be rejected by the other against the same user row.
        options.Password.RequireNonAlphanumeric = false;
        options.SignIn.RequireConfirmedAccount = false;
    })
    .AddEntityFrameworkStores<IdentityContext>()
    .AddDefaultTokenProviders();

builder.Services.ConfigureApplicationCookie(options =>
{
    options.LoginPath = "/account/login";
    options.LogoutPath = "/account/logout";
    options.AccessDeniedPath = "/account/login";

    // Named distinctly from the MVC app's cookie. They are separate sessions on the same host, and
    // sharing a name would have each app overwrite the other's cookie on the localhost origin.
    options.Cookie.Name = "MVC_App.BlazorClient.Auth";
});

// Supplies the AuthenticationState that <AuthorizeView> and [Authorize] read, and flows it into
// the SignalR circuit when an interactive page takes over from the server-rendered one.
builder.Services.AddCascadingAuthenticationState();
builder.Services.AddScoped<AuthenticationStateProvider, ServerAuthenticationStateProvider>();

builder.Services.AddScoped<AuthEventLogger>();

// Replaces the framework's default, which writes a component exception to the local logger only.
// Ours stamps the ELK field set on it first, so a failure inside a component is queryable in
// Kibana the same way an API failure is. Registered before AddRazorComponents' own default wins.
builder.Services.AddScoped<IErrorBoundaryLogger, LoggingErrorBoundaryLogger>();

builder.Services.AddSingleton(TimeProvider.System);
builder.Services.AddTransient<ApiLoggingHandler>();

// Base address and key default to the MVC app's https launch profile and its Api:Keys entry, so
// running both projects needs no configuration. Override in appsettings.Development.json.
var apiBaseUrl = builder.Configuration["Api:BaseUrl"] ?? "https://localhost:7119";
var apiKey = builder.Configuration["Api:Key"] ?? "<REDACTED_API_KEY>";
var apiHeaderName = builder.Configuration["Api:HeaderName"] ?? "X-Api-Key";

builder.Services.AddHttpClient<MvcAppApiClient>(client =>
    {
        client.BaseAddress = new Uri(apiBaseUrl.TrimEnd('/') + "/");
        client.DefaultRequestHeaders.Add(apiHeaderName, apiKey);
        client.Timeout = TimeSpan.FromSeconds(30);
    })
    // Logs every outbound call into mvcapp-logs-*. The OTel HttpClient instrumentation traces
    // these calls too, but a trace is not a log and never reaches the log index.
    .AddHttpMessageHandler<ApiLoggingHandler>()
    .ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
    {
        // The MVC app runs on the ASP.NET Core development certificate, which this process does
        // not trust by default. Scoped to Development only; in any other environment a bad
        // certificate stays a hard failure.
        ServerCertificateCustomValidationCallback = builder.Environment.IsDevelopment()
            ? HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
            : null
    });

var app = builder.Build();

// Outermost, so it sees the status code that was ultimately written and is the last place an
// escaping exception can still be recorded. Mirrors MVC_App's own pipeline order.
app.UseAppRequestLogging();

if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Error", createScopeForErrors: true);
    app.UseHsts();
}

app.UseStatusCodePagesWithReExecute("/not-found", createScopeForStatusCodePages: true);
app.UseHttpsRedirection();

// Before UseAntiforgery and the endpoints, so an authenticated identity exists by the time a
// component is authorized or a sign-in form is handled.
app.UseAuthentication();
app.UseAuthorization();

app.UseAntiforgery();

app.MapStaticAssets();
app.MapRazorComponents<App>()
    .AddInteractiveServerRenderMode();

app.Run();

 MVC_App.BlazorClient/Telemetry/ApiLoggingHandler.cs

An HTTP span won't appear in the log index by itself. This handler therefore writes a structured log for each typed-client API call, including its duration. Transport exceptions are classified as 503 and rethrown; the catch also includes cancellation paths. These records complement the separate HttpClient spans.

using System.Diagnostics;
using MVC_App.BlazorClient.Diagnostics;

namespace MVC_App.BlazorClient.Telemetry;

/// <summary>
/// Writes one structured log entry for every HTTP call this app makes to the MVC API, including
/// the ones that fail before a response exists (connection refused, TLS failure, timeout).
///
/// This is deliberately separate from the OpenTelemetry HttpClient instrumentation registered in
/// <see cref="TelemetryExtensions"/>. That instrumentation produces <em>traces</em>, which go
/// straight to Elasticsearch as <c>traces-generic.otel-default</c> and never reach
/// <c>mvcapp-logs-*</c>. Outbound calls therefore have to be logged explicitly to be queryable
/// alongside everything else in the log index.
///
/// Each entry carries the same <c>trace_id</c> as the MVC app's server-side entry for the same
/// call, so one trace id in Kibana shows the click, the outbound request, the API's own request
/// log, its SQL, and any exception it threw.
/// </summary>
public sealed class ApiLoggingHandler : DelegatingHandler
{
    private readonly ILogger<ApiLoggingHandler> _logger;
    private readonly TimeProvider _timeProvider;

    public ApiLoggingHandler(ILogger<ApiLoggingHandler> logger, TimeProvider timeProvider)
    {
        _logger = logger;
        _timeProvider = timeProvider;
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var started = _timeProvider.GetTimestamp();
        var method = request.Method.Method;
        var url = request.RequestUri?.ToString();
        var path = request.RequestUri?.AbsolutePath;

        try
        {
            var response = await base.SendAsync(request, cancellationToken);

            var elapsedMs = _timeProvider.GetElapsedTime(started).TotalMilliseconds;
            var statusCode = (int)response.StatusCode;
            var condition = ErrorConditions.ForStatusCode(statusCode);

            using var scope = _logger.BeginScope(AppLogging.Fields(statusCode, exception: null, Extra(
                method, url, path, elapsedMs)));

            _logger.Log(AppLogging.LevelForStatusCode(statusCode),
                "API call {Method} {Path} responded {StatusCode} [{ErrorCategory}] in {DurationMs:F1}ms",
                method, path, statusCode, condition, elapsedMs);

            return response;
        }
        catch (Exception exception)
        {
            // No response came back at all. Reported as 503 rather than 500: the failure is that a
            // dependency (the MVC API) could not be reached, which is the same condition the MVC
            // app reports with DependencyUnavailableException, so both show up under the one
            // error.category : "service_unavailable" filter.
            var elapsedMs = _timeProvider.GetElapsedTime(started).TotalMilliseconds;
            const int statusCode = StatusCodes.Status503ServiceUnavailable;

            using var scope = _logger.BeginScope(AppLogging.Fields(statusCode, exception, Extra(
                method, url, path, elapsedMs)));

            _logger.LogError(exception,
                "API call {Method} {Path} failed after {DurationMs:F1}ms without a response [{ErrorCategory}]: {Message}",
                method, path, elapsedMs, ErrorConditions.Unavailable, exception.Message);

            throw;
        }
    }

    private static Dictionary<string, object?> Extra(
        string method, string? url, string? path, double elapsedMs) => new()
    {
        ["http.request.method"] = method,
        ["url.full"] = url,
        ["url.path"] = path,
        ["http.client.duration_ms"] = elapsedMs,

        // Marks this as a call leaving the app, as opposed to one arriving at it. Both shapes
        // carry http.request.method and url.path, so without this they are hard to tell apart.
        ["http.request.direction"] = "outbound",
        ["trace.id"] = Activity.Current?.TraceId.ToString()
    };
}

MVC_App.BlazorClient/Telemetry/AppLogging.cs

Keeping the status and error fields consistent with MVC means the same queries can work for both applications. This helper adds the Blazor component label, lets callers supply their own context and drops null fields such as a missing exception type.

using MVC_App.BlazorClient.Diagnostics;

namespace MVC_App.BlazorClient.Telemetry;

/// <summary>
/// Builds the structured field set that every log entry from this app carries, so a document
/// written here is shaped exactly like one written by MVC_App and the same Kibana queries find
/// both (see docs/ELK-KQL-Examples.txt).
///
/// Three callers use it, covering the three ways something can happen in a Blazor Server app:
/// <list type="bullet">
/// <item>inbound page and SignalR requests, via <see cref="RequestLoggingMiddleware"/>;</item>
/// <item>outbound API calls, via <see cref="ApiLoggingHandler"/>;</item>
/// <item>exceptions raised while a component renders, via <see cref="LoggingErrorBoundaryLogger"/>.</item>
/// </list>
/// </summary>
internal static class AppLogging
{
    /// <summary>
    /// Maps an HTTP status code to the level ELK should surface it at. Identical to the MVC app's
    /// rule: 5xx is Error, every 4xx is Warning, everything else is Information. Note that this
    /// puts a 404 or a 409 at Warning, so a <c>severity_text : "Error"</c> filter in Kibana will
    /// not show them. Filter on <c>error.type : *</c> instead.
    /// </summary>
    public static LogLevel LevelForStatusCode(int statusCode) => statusCode switch
    {
        >= 500 => LogLevel.Error,
        401 or 402 or 403 => LogLevel.Warning,
        >= 400 => LogLevel.Warning,
        _ => LogLevel.Information
    };

    /// <summary>
    /// The common field set. <paramref name="extra"/> adds call-site specific fields, e.g. the
    /// target URL on an outbound call. Nulls are stripped, because an absent field and a null
    /// behave identically for an <c>exists</c> query but an absent one keeps Discover readable.
    /// </summary>
    public static Dictionary<string, object?> Fields(
        int statusCode,
        Exception? exception,
        IReadOnlyDictionary<string, object?>? extra = null)
    {
        var condition = ErrorConditions.ForStatusCode(statusCode);
        var isFailure = ErrorConditions.IsFailure(statusCode);

        var fields = new Dictionary<string, object?>
        {
            ["http.response.status_code"] = statusCode,

            // Error-condition fields, present on every entry including the successful ones
            // (error.category = "none"), so a Kibana filter never silently drops documents
            // just because a field is missing on the healthy path.
            ["error.category"] = condition,
            ["error.is_unauthorized"] = condition == ErrorConditions.Unauthorized,
            ["error.is_not_found"] = condition == ErrorConditions.NotFound,
            ["error.is_server_error"] = condition == ErrorConditions.ServerError,
            ["error.type"] = exception?.GetType().Name,
            ["event.outcome"] = isFailure ? "failure" : "success",
            ["event.kind"] = isFailure ? "error" : "event",

            // Distinguishes this app's documents from MVC_App's inside the shared index. The MVC
            // app stamps http.request.is_api; this is the Blazor equivalent dimension.
            ["app.component"] = "blazor-client"
        };

        if (extra is not null)
        {
            foreach (var (key, value) in extra)
            {
                fields[key] = value;
            }
        }

        foreach (var key in fields.Where(f => f.Value is null).Select(f => f.Key).ToList())
        {
            fields.Remove(key);
        }

        return fields;
    }
}

MVC_App.BlazorClient/Telemetry/LoggingErrorBoundaryLogger.cs

When a component failure reaches ErrorBoundary, this logger gives it the same searchable fields as an API error. An ApiException keeps its original status instead of becoming a blanket 500. Errors already caught and handled by a page don't reach this boundary.

using Microsoft.AspNetCore.Components.Web;
using MVC_App.BlazorClient.Api;
using MVC_App.BlazorClient.Diagnostics;

namespace MVC_App.BlazorClient.Telemetry;

/// <summary>
/// Logs exceptions caught by an <see cref="ErrorBoundary"/> in the component tree.
///
/// This is the third of the three ways something can go wrong here, and the one a plain
/// middleware cannot see. A Blazor Server component renders over the SignalR circuit, long after
/// the HTTP request that opened the page has completed, so an exception thrown in
/// <c>OnInitializedAsync</c> or in a button handler never passes through
/// <see cref="RequestLoggingMiddleware"/>. The framework's default implementation writes it to the
/// local logger only; replacing the service routes it through the same field set as everything
/// else and therefore into <c>mvcapp-logs-*</c>.
///
/// An <see cref="ApiException"/> keeps the status code the API actually returned, so a 409 from
/// the server is logged as a conflict here too rather than being flattened into a 500.
/// </summary>
public sealed class LoggingErrorBoundaryLogger : IErrorBoundaryLogger
{
    private readonly ILogger<LoggingErrorBoundaryLogger> _logger;

    public LoggingErrorBoundaryLogger(ILogger<LoggingErrorBoundaryLogger> logger)
    {
        _logger = logger;
    }

    public ValueTask LogErrorAsync(Exception exception)
    {
        var statusCode = exception is ApiException apiException
            ? apiException.StatusCode
            : StatusCodes.Status500InternalServerError;

        var condition = ErrorConditions.ForStatusCode(statusCode);

        var fields = AppLogging.Fields(statusCode, exception, new Dictionary<string, object?>
        {
            // Names where in the app the failure surfaced, since there is no URL path to key on.
            ["event.module"] = "blazor-component",
            ["error.boundary"] = true
        });

        using var scope = _logger.BeginScope(fields);

        _logger.Log(AppLogging.LevelForStatusCode(statusCode), exception,
            "{ExceptionType} while rendering a component -> {StatusCode} [{ErrorCategory}]: {Message}",
            exception.GetType().Name, statusCode, condition, exception.Message);

        return ValueTask.CompletedTask;
    }
}

MVC_App.BlazorClient/Telemetry/RequestLoggingMiddleware.cs

The Blazor host records its incoming HTTP outcomes here, but skips /_blazor circuit traffic to avoid a noisy log stream. For failures inside a later component event, look to the error boundary or the outgoing API handler rather than expecting this middleware to describe every click.

using MVC_App.BlazorClient.Diagnostics;

namespace MVC_App.BlazorClient.Telemetry;

/// <summary>
/// Outermost middleware: logs the final outcome of every inbound request, and is the last place
/// an exception can be recorded before it leaves the process.
///
/// Mirrors <c>MVC_App/Middleware/RequestLoggingMiddleware.cs</c>, including its rule that an
/// exception reaching here unlogged must be written out with its stack trace rather than reduced
/// to a bare status code.
/// </summary>
public sealed class RequestLoggingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<RequestLoggingMiddleware> _logger;

    public RequestLoggingMiddleware(RequestDelegate next, ILogger<RequestLoggingMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        try
        {
            await _next(context);
            Log(context, context.Response.StatusCode, exception: null);
        }
        catch (Exception exception)
        {
            Log(context, StatusCodes.Status500InternalServerError, exception);
            throw;
        }
    }

    private void Log(HttpContext context, int statusCode, Exception? exception)
    {
        var method = context.Request.Method;
        var path = context.Request.Path.Value;

        // The Blazor SignalR circuit is a single long-lived request that would otherwise be logged
        // once per connection with no useful outcome. Component activity inside the circuit is
        // logged by LoggingErrorBoundaryLogger and ApiLoggingHandler instead.
        if (path is not null && path.StartsWith("/_blazor", StringComparison.OrdinalIgnoreCase))
        {
            return;
        }

        var condition = ErrorConditions.ForStatusCode(statusCode);

        var fields = AppLogging.Fields(statusCode, exception, new Dictionary<string, object?>
        {
            ["http.request.method"] = method,
            ["url.path"] = path,
            ["url.query"] = context.Request.QueryString.HasValue ? context.Request.QueryString.Value : null
        });

        using var scope = _logger.BeginScope(fields);

        if (exception is not null)
        {
            _logger.Log(AppLogging.LevelForStatusCode(statusCode), exception,
                "{ExceptionType} while handling {Method} {Path} -> {StatusCode} [{ErrorCategory}]: {Message}",
                exception.GetType().Name, method, path, statusCode, condition, exception.Message);
        }
        else
        {
            _logger.Log(AppLogging.LevelForStatusCode(statusCode),
                "{Method} {Path} responded {StatusCode} [{ErrorCategory}]",
                method, path, statusCode, condition);
        }
    }
}

public static class RequestLoggingMiddlewareExtensions
{
    public static IApplicationBuilder UseAppRequestLogging(this IApplicationBuilder app)
        => app.UseMiddleware<RequestLoggingMiddleware>();
}

MVC_App.BlazorClient/Telemetry/TelemetryExtensions.cs

Blazor sends its logs, HTTP traces and runtime/HTTP metrics to the same Collector as MVC_App, but uses a distinct service name. That lets you search both services together or separate them in Kibana. Unlike the MVC extension, this one does not add SqlClient instrumentation for its direct Identity database calls.

using OpenTelemetry;
using OpenTelemetry.Exporter;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

namespace MVC_App.BlazorClient.Telemetry;

/// <summary>
/// Wires this app into the same observability stack as MVC_App: OTLP over HTTP to the
/// OpenTelemetry Collector in compose.yaml, which hands logs to Logstash and traces/metrics
/// straight to Elasticsearch.
///
/// Logs therefore land in the same <c>mvcapp-logs-*</c> indices as the MVC app's, because the
/// Logstash pipeline indexes everything it reads from the Collector's log file under that name
/// regardless of which service produced it. <c>service.name</c> is the field that tells them
/// apart, so this app reports itself as MVC_App.BlazorClient rather than borrowing MVC_App's name.
/// </summary>
public static class TelemetryExtensions
{
    public static WebApplicationBuilder AddApplicationTelemetry(this WebApplicationBuilder builder)
    {
        var endpoint = builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"];
        if (!builder.Configuration.GetValue("Telemetry:Enabled", true) || string.IsNullOrWhiteSpace(endpoint))
        {
            return builder;
        }

        if (!Uri.TryCreate(endpoint, UriKind.Absolute, out var endpointUri) ||
            (endpointUri.Scheme != Uri.UriSchemeHttp && endpointUri.Scheme != Uri.UriSchemeHttps))
        {
            throw new InvalidOperationException("OTEL_EXPORTER_OTLP_ENDPOINT must be an absolute HTTP or HTTPS base URL.");
        }

        builder.Logging.AddOpenTelemetry(options =>
        {
            // IncludeScopes is what carries the AppLogging field set onto the exported record;
            // without it every error.category / error.type field would be dropped at the exporter.
            options.IncludeFormattedMessage = true;
            options.IncludeScopes = true;
        });

        builder.Services.AddOpenTelemetry()
            .ConfigureResource(resource => resource
                .AddService(builder.Configuration["OTEL_SERVICE_NAME"] ?? builder.Environment.ApplicationName)
                .AddAttributes(new Dictionary<string, object>
                {
                    ["deployment.environment.name"] = builder.Environment.EnvironmentName
                }))
            .WithTracing(tracing => tracing
                .AddAspNetCoreInstrumentation()
                // Produces a client span for every call this app makes to the MVC API. The span
                // carries the same trace_id as the server-side spans and log entries the MVC app
                // writes for that call, so one trace_id in Kibana shows the whole hop.
                .AddHttpClientInstrumentation(options =>
                {
                    options.RecordException = true;
                }))
            .WithMetrics(metrics => metrics
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation())
            .UseOtlpExporter(OtlpExportProtocol.HttpProtobuf, endpointUri);

        return builder;
    }
}

MVC_App.ApiClient

The API console gives us repeatable traffic to investigate. Its local reports describe the calls it made; the backend supplies the corresponding logs, traces and metrics in ELK.

MVC_App.ApiClient/ApiCallResult.cs

Each result holds the status and elapsed time together with any problem-response metadata. In particular, the returned traceId gives you a direct starting point for finding the backend events behind a console failure.

using System.Net;
using System.Text.Json;

namespace MVC_App.ApiClient;

/// <summary>
/// One call's outcome: what was expected, what came back, and the correlation id the server put
/// in the problem document. The <see cref="TraceId"/> is the value to paste into Kibana
/// (<c>trace_id : "..."</c>) to find the server-side log entry for this exact call.
/// </summary>
public sealed record ApiCallResult(
    string Scenario,
    HttpMethod Method,
    string Path,
    int ExpectedStatus,
    int ActualStatus,
    string? ErrorCategory,
    string? ErrorType,
    string? TraceId,
    string Body,
    TimeSpan Elapsed,
    string? TransportError = null)
{
    public bool Matched => TransportError is null && ActualStatus == ExpectedStatus;

    public static ApiCallResult Transport(
        string scenario, HttpMethod method, string path, int expectedStatus, string error, TimeSpan elapsed)
        => new(scenario, method, path, expectedStatus, 0, null, null, null, string.Empty, elapsed, error);

    /// <summary>
    /// Pulls <c>errorCategory</c>, <c>errorType</c> and <c>traceId</c> out of an
    /// <c>application/problem+json</c> body. Returns nulls for a success response or any body that
    /// isn't the expected shape; a malformed body is never a reason to fail the run.
    /// </summary>
    public static (string? Category, string? Type, string? TraceId) ReadProblemFields(string body)
    {
        if (string.IsNullOrWhiteSpace(body) || body.TrimStart()[0] != '{')
        {
            return (null, null, null);
        }

        try
        {
            using var document = JsonDocument.Parse(body);
            var root = document.RootElement;
            return (
                root.TryGetProperty("errorCategory", out var category) ? category.GetString() : null,
                root.TryGetProperty("errorType", out var type) ? type.GetString() : null,
                root.TryGetProperty("traceId", out var traceId) ? traceId.GetString() : null);
        }
        catch (JsonException)
        {
            return (null, null, null);
        }
    }

    public string StatusText => TransportError is not null
        ? "ERR"
        : $"{ActualStatus} {(HttpStatusCode)ActualStatus}";
}

MVC_App.ApiClient/ClientOptions.cs

Use these options to choose the API address, key, iteration count, delays and load concurrency. Environment variables can supply the URL/key without putting credentials in command-line arguments. The defaults target MVC's HTTPS development profile; the insecure TLS option should stay limited to local testing.

namespace MVC_App.ApiClient;

/// <summary>
/// Command-line and environment configuration for the console client. Every value has a default
/// that matches the <c>https</c> launch profile in <c>MVC_App/Properties/launchSettings.json</c>
/// and the <c>Api:Keys</c> entry in <c>appsettings.Development.json</c>, so the common case is
/// just <c>dotnet run</c> with no arguments.
/// </summary>
public sealed class ClientOptions
{
    public Uri BaseAddress { get; private set; } = new("https://localhost:7119");

    public string ApiKey { get; private set; } = "<REDACTED_API_KEY>";

    /// <summary>Accept the local development certificate without validating it.</summary>
    public bool Insecure { get; private set; }

    /// <summary>How many times to run the whole scenario suite. Useful for filling a dashboard.</summary>
    public int Iterations { get; private set; } = 1;

    /// <summary>Pause between individual calls, so a time-series panel has visible spread.</summary>
    public TimeSpan Delay { get; private set; } = TimeSpan.Zero;

    /// <summary>Print the response body of every call, not just the failures.</summary>
    public bool Verbose { get; private set; }

    // -----------------------------------------------------------------------
    // Load mode: a high-volume run with an exact request count per HTTP verb,
    // instead of the 44-call correctness suite.
    // -----------------------------------------------------------------------

    /// <summary>Run the bulk load instead of the scenario suite.</summary>
    public bool Load { get; private set; }

    public int GetCount { get; private set; } = 3000;

    public int PostCount { get; private set; } = 1000;

    public int PutCount { get; private set; } = 1000;

    public int DeleteCount { get; private set; } = 1000;

    /// <summary>Requests in flight at once. Each worker owns a disjoint range of entity ids.</summary>
    public int Concurrency { get; private set; } = 8;

    /// <summary>Print a progress line every N completed requests. 0 disables progress output.</summary>
    public int ProgressEvery { get; private set; } = 250;

    public int TotalLoadRequests => GetCount + PostCount + PutCount + DeleteCount;

    public static bool TryParse(string[] args, out ClientOptions options, out string? error)
    {
        options = new ClientOptions();
        error = null;

        if (Environment.GetEnvironmentVariable("MVCAPP_API_BASEURL") is { Length: > 0 } envUrl &&
            Uri.TryCreate(envUrl, UriKind.Absolute, out var parsedEnvUrl))
        {
            options.BaseAddress = parsedEnvUrl;
        }

        if (Environment.GetEnvironmentVariable("MVCAPP_API_KEY") is { Length: > 0 } envKey)
        {
            options.ApiKey = envKey;
        }

        for (var i = 0; i < args.Length; i++)
        {
            var arg = args[i];
            switch (arg)
            {
                case "--url" or "-u":
                    if (!TryTakeValue(args, ref i, out var url) ||
                        !Uri.TryCreate(url, UriKind.Absolute, out var parsedUrl))
                    {
                        error = $"'{arg}' needs an absolute URL, e.g. --url https://localhost:7119";
                        return false;
                    }
                    options.BaseAddress = parsedUrl;
                    break;

                case "--key" or "-k":
                    if (!TryTakeValue(args, ref i, out var key))
                    {
                        error = $"'{arg}' needs a value, e.g. --key <REDACTED_API_KEY>";
                        return false;
                    }
                    options.ApiKey = key;
                    break;

                case "--iterations" or "-n":
                    if (!TryTakeValue(args, ref i, out var rawIterations) ||
                        !int.TryParse(rawIterations, out var iterations) || iterations < 1)
                    {
                        error = $"'{arg}' needs a positive whole number, e.g. --iterations 5";
                        return false;
                    }
                    options.Iterations = iterations;
                    break;

                case "--delay" or "-d":
                    if (!TryTakeValue(args, ref i, out var rawDelay) ||
                        !int.TryParse(rawDelay, out var delayMs) || delayMs < 0)
                    {
                        error = $"'{arg}' needs a millisecond count, e.g. --delay 250";
                        return false;
                    }
                    options.Delay = TimeSpan.FromMilliseconds(delayMs);
                    break;

                case "--load" or "-l":
                    options.Load = true;
                    break;

                case "--gets":
                case "--posts":
                case "--puts":
                case "--deletes":
                case "--concurrency" or "-c":
                case "--progress":
                {
                    if (!TryTakeValue(args, ref i, out var rawCount) ||
                        !int.TryParse(rawCount, out var count) || count < 0)
                    {
                        error = $"'{arg}' needs a whole number of 0 or more, e.g. {arg} 1000";
                        return false;
                    }

                    switch (arg)
                    {
                        case "--gets": options.GetCount = count; break;
                        case "--posts": options.PostCount = count; break;
                        case "--puts": options.PutCount = count; break;
                        case "--deletes": options.DeleteCount = count; break;
                        case "--progress": options.ProgressEvery = count; break;
                        default:
                            if (count < 1)
                            {
                                error = "'--concurrency' needs at least 1.";
                                return false;
                            }
                            options.Concurrency = count;
                            break;
                    }

                    // Any of these implies the bulk run; nobody passes --gets 3000 for a 44-call suite.
                    if (arg is not ("--concurrency" or "-c" or "--progress"))
                    {
                        options.Load = true;
                    }

                    break;
                }

                case "--insecure":
                    options.Insecure = true;
                    break;

                case "--verbose" or "-v":
                    options.Verbose = true;
                    break;

                case "--help" or "-h" or "-?":
                    error = string.Empty;
                    return false;

                default:
                    error = $"Unrecognised argument '{arg}'.";
                    return false;
            }
        }

        return true;
    }

    private static bool TryTakeValue(string[] args, ref int index, out string value)
    {
        if (index + 1 >= args.Length)
        {
            value = string.Empty;
            return false;
        }

        value = args[++index];
        return true;
    }

    public const string Usage = """
        MVC_App.ApiClient - exercises the MVC_App JSON API and prints the status code each call returned.

        Two modes. By default it runs the 44-call correctness suite, which produces every status
        code once. With --load it runs a bulk load with an exact request count per HTTP verb.

        Usage:
          dotnet run --project MVC_App.ApiClient [options]

        Options:
          -u, --url <url>          Base address of the running app.   Default: https://localhost:7119
                                   Also read from MVCAPP_API_BASEURL.
          -k, --key <key>          API key sent as the X-Api-Key header.
                                   Default: <REDACTED_API_KEY>. Also read from MVCAPP_API_KEY.
          -n, --iterations <n>     Run the whole suite n times.       Default: 1
          -d, --delay <ms>         Pause between calls, in ms.        Default: 0
              --insecure           Skip TLS validation (local dev certificate only).
          -v, --verbose            Print every response body, not just the unexpected ones.
          -h, --help               Show this help.

        Load mode:
          -l, --load               Run the bulk load instead of the scenario suite.
              --gets <n>           GET requests to issue.             Default: 3000
              --posts <n>          POST requests to issue.            Default: 1000
              --puts <n>           PUT requests to issue.             Default: 1000
              --deletes <n>        DELETE requests to issue.          Default: 1000
          -c, --concurrency <n>    Requests in flight at once.        Default: 8
              --progress <n>       Progress line every n requests.    Default: 250 (0 = off)

          Passing any of --gets/--posts/--puts/--deletes turns on load mode by itself.

        Examples:
          dotnet run --project MVC_App.ApiClient
          dotnet run --project MVC_App.ApiClient -- --url http://localhost:5121
          dotnet run --project MVC_App.ApiClient -- --iterations 20 --delay 300

          # 6000 requests: 3000 GET, 1000 POST, 1000 PUT, 1000 DELETE
          dotnet run --project MVC_App.ApiClient -- --url http://localhost:5121 --load

          # 5000 requests: 2500 GET, 1000 POST, 750 PUT, 750 DELETE
          dotnet run --project MVC_App.ApiClient -- --url http://localhost:5121 \
            --gets 2500 --posts 1000 --puts 750 --deletes 750 --concurrency 16
        """;
}

MVC_App.ApiClient/ConsoleReport.cs

This report puts the per-call results and final summary in a readable console format, including diagnostic fields on failures. It stays local; there is no exporter forwarding these console lines to ELK.

namespace MVC_App.ApiClient;

/// <summary>Console formatting: one line per call, then a summary grouped by status code.</summary>
public static class ConsoleReport
{
    public static void WriteCallLine(ApiCallResult result, bool verbose)
    {
        var original = Console.ForegroundColor;

        Console.ForegroundColor = result.Matched ? ConsoleColor.Green : ConsoleColor.Red;
        Console.Write(result.Matched ? "  ok   " : "  FAIL ");
        Console.ForegroundColor = original;

        Console.Write($"{result.Method.Method,-6} {Truncate(result.Path, 46),-46} ");

        Console.ForegroundColor = ColourForStatus(result.ActualStatus);
        Console.Write($"{(result.TransportError is null ? result.ActualStatus.ToString() : "---"),-4}");
        Console.ForegroundColor = original;

        Console.Write($" {Truncate(result.Scenario, 42),-42} {result.Elapsed.TotalMilliseconds,6:F0} ms");

        if (result.ErrorCategory is { Length: > 0 } category && category != "none")
        {
            Console.ForegroundColor = ConsoleColor.DarkGray;
            Console.Write($"  {category}");
            Console.ForegroundColor = original;
        }

        Console.WriteLine();

        if (result.TransportError is not null)
        {
            WriteDetail($"transport: {result.TransportError}", ConsoleColor.Red);
            return;
        }

        if (!result.Matched)
        {
            WriteDetail($"expected {result.ExpectedStatus}, got {result.ActualStatus}", ConsoleColor.Red);
        }

        if (result.TraceId is { Length: > 0 } traceId)
        {
            WriteDetail($"trace_id : \"{traceId}\"", ConsoleColor.DarkGray);
        }

        if (verbose && result.Body.Length > 0)
        {
            WriteDetail(Truncate(result.Body.ReplaceLineEndings(" "), 160), ConsoleColor.DarkGray);
        }
    }

    public static void WriteSummary(IReadOnlyList<ApiCallResult> results, ClientOptions options)
    {
        var original = Console.ForegroundColor;
        var matched = results.Count(r => r.Matched);
        var transportFailures = results.Count(r => r.TransportError is not null);

        Console.WriteLine();
        Console.WriteLine(new string('=', 92));
        Console.WriteLine("Summary");
        Console.WriteLine(new string('=', 92));

        Console.WriteLine();
        Console.WriteLine($"  Target          {options.BaseAddress}");
        Console.WriteLine($"  Calls           {results.Count}");

        Console.ForegroundColor = matched == results.Count ? ConsoleColor.Green : ConsoleColor.Yellow;
        Console.WriteLine($"  Matched         {matched} of {results.Count}");
        Console.ForegroundColor = original;

        if (transportFailures > 0)
        {
            Console.ForegroundColor = ConsoleColor.Red;
            Console.WriteLine($"  Unreachable     {transportFailures} (the app did not answer)");
            Console.ForegroundColor = original;
        }

        Console.WriteLine();
        Console.WriteLine("  Responses by status code");
        foreach (var group in results
                     .Where(r => r.TransportError is null)
                     .GroupBy(r => r.ActualStatus)
                     .OrderBy(g => g.Key))
        {
            Console.Write("    ");
            Console.ForegroundColor = ColourForStatus(group.Key);
            Console.Write($"{group.Key,-5}");
            Console.ForegroundColor = original;
            Console.WriteLine($"{group.Count(),4} call(s)");
        }

        var categories = results
            .Where(r => r.ErrorCategory is { Length: > 0 } && r.ErrorCategory != "none")
            .GroupBy(r => r.ErrorCategory!)
            .OrderByDescending(g => g.Count())
            .ToList();

        if (categories.Count > 0)
        {
            Console.WriteLine();
            Console.WriteLine("  Error conditions reported by the app (error.category in Kibana)");
            foreach (var group in categories)
            {
                Console.WriteLine($"    {group.Key,-22}{group.Count(),4} call(s)");
            }
        }

        var mismatches = results.Where(r => !r.Matched).ToList();
        if (mismatches.Count > 0)
        {
            Console.WriteLine();
            Console.ForegroundColor = ConsoleColor.Red;
            Console.WriteLine("  Calls that did not return the expected status");
            Console.ForegroundColor = original;
            foreach (var mismatch in mismatches)
            {
                Console.WriteLine(
                    $"    {mismatch.Method.Method,-6} {Truncate(mismatch.Path, 44),-44} " +
                    $"expected {mismatch.ExpectedStatus}, got {(mismatch.TransportError is null ? mismatch.ActualStatus.ToString() : "no response")}");
            }
        }

        Console.WriteLine();
        Console.WriteLine("  Next: open Kibana and run one of the queries from");
        Console.WriteLine("  docs/elk-dashboard-tutorial.html, e.g.");
        Console.WriteLine("    log_attributes.error.category : \"server_error\"");
        Console.WriteLine("    log_attributes.http.request.is_api : true and log_attributes.error.is_not_found : true");
        Console.WriteLine();
    }

    private static void WriteDetail(string text, ConsoleColor colour)
    {
        var original = Console.ForegroundColor;
        Console.ForegroundColor = colour;
        Console.WriteLine($"         {text}");
        Console.ForegroundColor = original;
    }

    private static ConsoleColor ColourForStatus(int statusCode) => statusCode switch
    {
        >= 500 => ConsoleColor.Red,
        401 or 403 => ConsoleColor.Magenta,
        404 => ConsoleColor.Yellow,
        >= 400 => ConsoleColor.Yellow,
        >= 200 and < 300 => ConsoleColor.Green,
        _ => ConsoleColor.Gray
    };

    private static string Truncate(string value, int maxLength)
        => value.Length <= maxLength ? value : string.Concat(value.AsSpan(0, maxLength - 1), "…");
}

MVC_App.ApiClient/LoadPlan.cs

Before starting workers, the plan groups dependent operations into lifecycle units so creates, updates and deletes can run in a sensible order while meeting the requested verb counts. Generated IDs start in the 500000 range. Use an isolated database to avoid collisions with existing data.

namespace MVC_App.ApiClient;

/// <summary>One planned request. The plan is built in full before anything is sent.</summary>
public sealed record LoadOperation(
    HttpMethod Method,
    string Path,
    object? Body,
    int ExpectedStatus,
    string Label);

/// <summary>
/// A sequence of operations that must run in order against the same entities, and that no other
/// unit touches. Units are handed to workers whole, so the whole plan can run concurrently
/// without two workers ever racing on the same row.
/// </summary>
public sealed record LoadUnit(int Index, IReadOnlyList<LoadOperation> Operations);

/// <summary>
/// Turns the requested per-verb counts into a concrete list of requests.
///
/// The building block is a <em>pair</em>: one department and one employee taken through a full
/// lifecycle. A pair is 2 POST, 6 GET, 2 PUT and 2 DELETE, which is exactly the 1 : 3 : 1 : 1
/// ratio of the default 1000 / 3000 / 1000 / 1000 run, so 500 pairs produce those counts with
/// nothing left over. Because every pair creates what it reads and deletes what it created, a
/// completed run leaves the database exactly as it found it.
///
/// Counts that don't divide into whole pairs still come out exact: whatever is left over is
/// emitted as standalone operations after the pairs (see <see cref="AddLeftovers"/>).
/// </summary>
public static class LoadPlan
{
    private const int GetsPerPair = 6;
    private const int PostsPerPair = 2;
    private const int PutsPerPair = 2;
    private const int DeletesPerPair = 2;

    /// <summary>An id range of its own, clear of the 900000+ ids the scenario suite uses.</summary>
    private const int IdRangeStart = 500_000;

    public static IReadOnlyList<LoadUnit> Build(ClientOptions options, out int idBase)
    {
        // Offset each run inside the range so two runs started close together are unlikely to
        // collide on the unique DeptName index.
        idBase = IdRangeStart + Random.Shared.Next(0, 150) * 1_000;

        var gets = options.GetCount;
        var posts = options.PostCount;
        var puts = options.PutCount;
        var deletes = options.DeleteCount;

        // As many complete pairs as every budget can cover.
        var pairs = Math.Min(
            Math.Min(gets / GetsPerPair, posts / PostsPerPair),
            Math.Min(puts / PutsPerPair, deletes / DeletesPerPair));

        var units = new List<LoadUnit>(pairs + 16);

        for (var i = 0; i < pairs; i++)
        {
            units.Add(new LoadUnit(units.Count, BuildPair(idBase + i)));
        }

        gets -= pairs * GetsPerPair;
        posts -= pairs * PostsPerPair;
        puts -= pairs * PutsPerPair;
        deletes -= pairs * DeletesPerPair;

        AddLeftovers(units, idBase + pairs, gets, posts, puts, deletes);

        return units;
    }

    /// <summary>
    /// One department and one employee, created, read, updated, read again and removed. The
    /// employee is created after its department and deleted before it, so the department a
    /// request references always exists while it is referenced.
    /// </summary>
    private static List<LoadOperation> BuildPair(int id)
    {
        var deptName = $"LOAD-Dept-{id}";
        var encodedDeptName = Uri.EscapeDataString(deptName);

        var department = new { DeptNo = id, DeptName = deptName, Location = "Pune" };
        var departmentUpdated = new { DeptNo = id, DeptName = deptName, Location = "Mumbai" };

        var employee = new
        {
            EmpNo = id,
            EmpName = $"LOAD Employee {id}",
            Salary = 50_000,
            DeptName = deptName,
            Designation = "Engineer"
        };

        var employeeUpdated = new
        {
            EmpNo = id,
            EmpName = $"LOAD Employee {id}",
            Salary = 60_000,
            DeptName = deptName,
            Designation = "Senior Engineer"
        };

        return
        [
            // 2 POST
            new(HttpMethod.Post, "/api/departments", department, 201, "create department"),
            new(HttpMethod.Post, "/api/employees", employee, 201, "create employee"),

            // 3 GET
            new(HttpMethod.Get, $"/api/departments/{id}", null, 200, "read department"),
            new(HttpMethod.Get, $"/api/employees/{id}", null, 200, "read employee"),
            new(HttpMethod.Get, $"/api/employees?deptName={encodedDeptName}", null, 200, "list department employees"),

            // 2 PUT
            new(HttpMethod.Put, $"/api/departments/{id}", departmentUpdated, 200, "update department"),
            new(HttpMethod.Put, $"/api/employees/{id}", employeeUpdated, 200, "update employee"),

            // 3 GET
            new(HttpMethod.Get, $"/api/departments/by-name/{encodedDeptName}", null, 200, "read department by name"),
            new(HttpMethod.Get, $"/api/departments/{id}", null, 200, "re-read department"),
            new(HttpMethod.Get, $"/api/employees/{id}", null, 200, "re-read employee"),

            // 2 DELETE, employee first so its department outlives it
            new(HttpMethod.Delete, $"/api/employees/{id}", null, 204, "delete employee"),
            new(HttpMethod.Delete, $"/api/departments/{id}", null, 204, "delete department")
        ];
    }

    /// <summary>
    /// Emits whatever the whole pairs couldn't absorb, one operation per unit so they interleave
    /// freely. Surplus reads hit the collection endpoints; surplus writes target ids that were
    /// never created, which answer 404 and leave no rows behind.
    /// </summary>
    private static void AddLeftovers(
        List<LoadUnit> units, int idBase, int gets, int posts, int puts, int deletes)
    {
        for (var i = 0; i < gets; i++)
        {
            // Alternate the two collection endpoints so neither dominates the route breakdown.
            var path = i % 2 == 0 ? "/api/departments" : "/api/employees";
            units.Add(new LoadUnit(units.Count, [new(HttpMethod.Get, path, null, 200, "list collection")]));
        }

        // A surplus POST has no matching DELETE, so its row survives the run. Reported at the end.
        for (var i = 0; i < posts; i++)
        {
            var id = idBase + i;
            var body = new { DeptNo = id, DeptName = $"LOAD-Dept-{id}", Location = "Pune" };
            units.Add(new LoadUnit(units.Count,
                [new(HttpMethod.Post, "/api/departments", body, 201, "surplus create department")]));
        }

        for (var i = 0; i < puts; i++)
        {
            var id = 990_000 + i;
            var body = new { DeptNo = id, DeptName = $"LOAD-Missing-{id}", Location = "Nowhere" };
            units.Add(new LoadUnit(units.Count,
                [new(HttpMethod.Put, $"/api/departments/{id}", body, 404, "surplus update, no such row")]));
        }

        for (var i = 0; i < deletes; i++)
        {
            units.Add(new LoadUnit(units.Count,
                [new(HttpMethod.Delete, $"/api/departments/{990_000 + i}", null, 404, "surplus delete, no such row")]));
        }
    }

    /// <summary>Counts the plan by verb, so the run can state up front what it is about to send.</summary>
    public static Dictionary<string, int> CountByMethod(IReadOnlyList<LoadUnit> units)
    {
        var counts = new Dictionary<string, int>(StringComparer.Ordinal);

        foreach (var operation in units.SelectMany(u => u.Operations))
        {
            counts[operation.Method.Method] = counts.GetValueOrDefault(operation.Method.Method) + 1;
        }

        return counts;
    }
}

MVC_App.ApiClient/LoadReport.cs

The load report summarizes status counts, throughput and latency. Check LoadPlan rather than relying on the report's hard-coded ID-range message when deciding which rows belong to a run; the displayed range can differ from the generated one.

using System.Net;

namespace MVC_App.ApiClient;

/// <summary>Console output for a bulk load run: the plan up front, the results at the end.</summary>
public static class LoadReport
{
    public static void WritePlan(ClientOptions options, IReadOnlyList<LoadUnit> units, int idBase)
    {
        var planned = LoadPlan.CountByMethod(units);
        var total = planned.Values.Sum();

        Console.WriteLine();
        Console.WriteLine(new string('=', 92));
        Console.WriteLine("Load run");
        Console.WriteLine(new string('=', 92));
        Console.WriteLine();
        Console.WriteLine($"  Target          {options.BaseAddress}");
        Console.WriteLine($"  Concurrency     {options.Concurrency} workers");
        Console.WriteLine($"  Entity ids      {idBase}..{idBase + units.Count} (created and deleted by this run)");
        Console.WriteLine();
        Console.WriteLine($"  Planned requests");

        foreach (var method in new[] { "GET", "POST", "PUT", "DELETE" })
        {
            var requested = method switch
            {
                "GET" => options.GetCount,
                "POST" => options.PostCount,
                "PUT" => options.PutCount,
                _ => options.DeleteCount
            };

            var actual = planned.GetValueOrDefault(method);
            var note = actual == requested ? "" : $"   (requested {requested})";
            Console.WriteLine($"    {method,-8}{actual,8:N0}{note}");
        }

        Console.WriteLine($"    {"TOTAL",-8}{total,8:N0}");
        Console.WriteLine();

        if (options.ProgressEvery > 0)
        {
            Console.WriteLine($"  Progress every {options.ProgressEvery:N0} requests");
            Console.WriteLine();
        }
    }

    public static void WriteResult(LoadStats stats, LoadRunner runner, ClientOptions options, int idBase)
    {
        var original = Console.ForegroundColor;
        var seconds = runner.Elapsed.TotalSeconds;
        var sorted = stats.LatenciesMs.ToArray();
        Array.Sort(sorted);

        Console.WriteLine();
        Console.WriteLine(new string('=', 92));
        Console.WriteLine("Load summary");
        Console.WriteLine(new string('=', 92));
        Console.WriteLine();
        Console.WriteLine($"  Requests sent   {stats.Total,8:N0}");
        Console.WriteLine($"  Elapsed         {seconds,8:F1} s");
        Console.WriteLine($"  Throughput      {(seconds > 0 ? stats.Total / seconds : 0),8:F0} req/s");

        Console.ForegroundColor = stats.Unexpected == 0 ? ConsoleColor.Green : ConsoleColor.Yellow;
        Console.WriteLine($"  Unexpected      {stats.Unexpected,8:N0} (status other than the one planned)");
        Console.ForegroundColor = original;

        if (stats.TransportErrors > 0)
        {
            Console.ForegroundColor = ConsoleColor.Red;
            Console.WriteLine($"  Unreachable     {stats.TransportErrors,8:N0} (the app did not answer)");
            Console.ForegroundColor = original;
        }

        Console.WriteLine();
        Console.WriteLine("  Requests by method");
        foreach (var method in new[] { "GET", "POST", "PUT", "DELETE" })
        {
            if (stats.ByMethod.TryGetValue(method, out var count))
            {
                Console.WriteLine($"    {method,-8}{count,8:N0}");
            }
        }

        Console.WriteLine();
        Console.WriteLine("  Responses by status code");
        foreach (var (status, count) in stats.ByStatus.OrderBy(s => s.Key))
        {
            Console.Write("    ");
            Console.ForegroundColor = ColourForStatus(status);
            Console.Write($"{status,-5}");
            Console.ForegroundColor = original;
            Console.WriteLine($"{count,8:N0}  {(HttpStatusCode)status}");
        }

        if (stats.ByErrorCategory.Count > 0)
        {
            Console.WriteLine();
            Console.WriteLine("  Error conditions (error.category in Kibana)");
            foreach (var (category, count) in stats.ByErrorCategory.OrderByDescending(c => c.Value))
            {
                Console.WriteLine($"    {category,-22}{count,8:N0}");
            }
        }

        if (sorted.Length > 0)
        {
            Console.WriteLine();
            Console.WriteLine("  Latency, milliseconds");
            Console.WriteLine($"    min      {sorted[0],8:F1}");
            Console.WriteLine($"    p50      {LoadStats.Percentile(sorted, 50),8:F1}");
            Console.WriteLine($"    p95      {LoadStats.Percentile(sorted, 95),8:F1}");
            Console.WriteLine($"    p99      {LoadStats.Percentile(sorted, 99),8:F1}");
            Console.WriteLine($"    max      {sorted[^1],8:F1}");
            Console.WriteLine($"    mean     {sorted.Average(),8:F1}");
        }

        if (runner.UnitsAbandoned > 0)
        {
            Console.WriteLine();
            Console.ForegroundColor = ConsoleColor.Yellow;
            Console.WriteLine($"  {runner.UnitsAbandoned} lifecycle(s) did not reach their DELETE, so some rows were left behind.");
            Console.WriteLine($"  They are in the id range {idBase}..{idBase + 200000}, named 'LOAD-Dept-*'.");
            Console.ForegroundColor = original;
        }
        else if (options.PostCount <= options.DeleteCount)
        {
            Console.WriteLine();
            Console.WriteLine("  Every row this run created was deleted again; the database is as it was.");
        }
        else
        {
            Console.WriteLine();
            Console.ForegroundColor = ConsoleColor.Yellow;
            Console.WriteLine($"  {options.PostCount - options.DeleteCount} surplus POST(s) had no matching DELETE, so those rows remain.");
            Console.ForegroundColor = original;
        }

        Console.WriteLine();
        Console.WriteLine("  In Kibana, this run is:");
        Console.WriteLine("    log_attributes.http.request.is_api : true and log_attributes.url.path : \"/api/*\"");
        Console.WriteLine("    log_attributes.http.request.method : (\"POST\" or \"PUT\" or \"DELETE\")");
        Console.WriteLine();
    }

    private static ConsoleColor ColourForStatus(int statusCode) => statusCode switch
    {
        >= 500 => ConsoleColor.Red,
        401 or 403 => ConsoleColor.Magenta,
        >= 400 => ConsoleColor.Yellow,
        >= 200 and < 300 => ConsoleColor.Green,
        _ => ConsoleColor.Gray
    };
}

MVC_App.ApiClient/LoadRunner.cs

A bounded worker pool executes the plan, reports progress and responds to cancellation. Failed operations are not retried, which helps keep the reported attempts aligned with the workload you requested.

using System.Diagnostics;

namespace MVC_App.ApiClient;

/// <summary>
/// Executes a <see cref="LoadPlan"/> across a fixed number of workers. Workers pull whole units
/// from a shared cursor, so each unit's operations stay in order and no two workers ever touch
/// the same entity. Nothing is retried: a load run is meant to report what the API did, not to
/// paper over it.
/// </summary>
public sealed class LoadRunner
{
    private readonly MvcAppApiClient _client;
    private readonly ClientOptions _options;
    private readonly IReadOnlyList<LoadUnit> _units;
    private readonly Lock _consoleLock = new();

    private int _cursor = -1;
    private int _completed;
    private int _unitsStarted;
    private int _unitsFinished;

    public LoadRunner(MvcAppApiClient client, ClientOptions options, IReadOnlyList<LoadUnit> units)
    {
        _client = client;
        _options = options;
        _units = units;
    }

    /// <summary>Units begun but not carried through to their DELETE, i.e. rows possibly left behind.</summary>
    public int UnitsAbandoned => _unitsStarted - _unitsFinished;

    public async Task<LoadStats> RunAsync(CancellationToken cancellationToken)
    {
        var total = _units.Sum(u => u.Operations.Count);
        var stopwatch = Stopwatch.StartNew();

        var workers = new Task<LoadStats>[Math.Min(_options.Concurrency, Math.Max(_units.Count, 1))];
        for (var i = 0; i < workers.Length; i++)
        {
            workers[i] = Task.Run(() => WorkerAsync(total, stopwatch, cancellationToken), cancellationToken);
        }

        LoadStats[] results;
        try
        {
            results = await Task.WhenAll(workers);
        }
        catch (OperationCanceledException)
        {
            // Collect whatever the workers that did finish managed to record.
            results = workers
                .Where(w => w.IsCompletedSuccessfully)
                .Select(w => w.Result)
                .ToArray();
        }

        stopwatch.Stop();
        Elapsed = stopwatch.Elapsed;

        return LoadStats.Merge(results);
    }

    public TimeSpan Elapsed { get; private set; }

    private async Task<LoadStats> WorkerAsync(int total, Stopwatch stopwatch, CancellationToken cancellationToken)
    {
        var stats = new LoadStats();

        while (!cancellationToken.IsCancellationRequested)
        {
            var index = Interlocked.Increment(ref _cursor);
            if (index >= _units.Count)
            {
                break;
            }

            var unit = _units[index];
            Interlocked.Increment(ref _unitsStarted);

            foreach (var operation in unit.Operations)
            {
                if (cancellationToken.IsCancellationRequested)
                {
                    return stats;
                }

                var result = await _client.SendAsync(
                    operation.Label,
                    operation.Method,
                    operation.Path,
                    operation.ExpectedStatus,
                    operation.Body,
                    apiKey: null,
                    retainBody: false,
                    cancellationToken: cancellationToken);

                stats.Record(result);
                ReportProgress(total, stopwatch);

                if (_options.Delay > TimeSpan.Zero)
                {
                    await Task.Delay(_options.Delay, cancellationToken);
                }
            }

            Interlocked.Increment(ref _unitsFinished);
        }

        return stats;
    }

    private void ReportProgress(int total, Stopwatch stopwatch)
    {
        var completed = Interlocked.Increment(ref _completed);

        if (_options.ProgressEvery <= 0 || (completed % _options.ProgressEvery != 0 && completed != total))
        {
            return;
        }

        var seconds = stopwatch.Elapsed.TotalSeconds;
        var rate = seconds > 0 ? completed / seconds : 0;
        var percent = total > 0 ? completed * 100d / total : 0;

        lock (_consoleLock)
        {
            Console.WriteLine(
                $"  {completed,7:N0} / {total,-7:N0} {percent,5:F1}%   {rate,7:F0} req/s   elapsed {seconds,6:F1}s");
        }
    }
}

MVC_App.ApiClient/LoadStats.cs

Each worker collects counts and timing samples before the results are merged. Percentiles use the nearest-rank method. These numbers describe the client's observed call time, not just server execution, and are separate from the backend's OTel metrics.

namespace MVC_App.ApiClient;

/// <summary>
/// Per-worker tallies, merged once at the end. Keeping a separate instance per worker avoids
/// locking on the hot path; the only shared state during the run is the completed counter in
/// <see cref="LoadRunner"/>.
/// </summary>
public sealed class LoadStats
{
    private readonly List<double> _latenciesMs = [];

    public Dictionary<string, int> ByMethod { get; } = new(StringComparer.Ordinal);

    public Dictionary<int, int> ByStatus { get; } = [];

    public Dictionary<string, int> ByErrorCategory { get; } = new(StringComparer.Ordinal);

    public int Total { get; private set; }

    public int Unexpected { get; private set; }

    public int TransportErrors { get; private set; }

    public IReadOnlyList<double> LatenciesMs => _latenciesMs;

    public void Record(ApiCallResult result)
    {
        Total++;
        Increment(ByMethod, result.Method.Method);

        if (result.TransportError is not null)
        {
            TransportErrors++;
            Unexpected++;
            return;
        }

        _latenciesMs.Add(result.Elapsed.TotalMilliseconds);
        ByStatus[result.ActualStatus] = ByStatus.GetValueOrDefault(result.ActualStatus) + 1;

        if (result.ErrorCategory is { Length: > 0 } category && category != "none")
        {
            Increment(ByErrorCategory, category);
        }

        if (!result.Matched)
        {
            Unexpected++;
        }
    }

    public static LoadStats Merge(IEnumerable<LoadStats> parts)
    {
        var merged = new LoadStats();

        foreach (var part in parts)
        {
            merged.Total += part.Total;
            merged.Unexpected += part.Unexpected;
            merged.TransportErrors += part.TransportErrors;
            merged._latenciesMs.AddRange(part._latenciesMs);

            foreach (var (key, value) in part.ByMethod)
            {
                merged.ByMethod[key] = merged.ByMethod.GetValueOrDefault(key) + value;
            }

            foreach (var (key, value) in part.ByStatus)
            {
                merged.ByStatus[key] = merged.ByStatus.GetValueOrDefault(key) + value;
            }

            foreach (var (key, value) in part.ByErrorCategory)
            {
                merged.ByErrorCategory[key] = merged.ByErrorCategory.GetValueOrDefault(key) + value;
            }
        }

        return merged;
    }

    /// <summary>
    /// Nearest-rank percentile over the recorded latencies. The caller sorts once and passes the
    /// sorted array in, so a report asking for p50/p95/p99 doesn't sort three times.
    /// </summary>
    public static double Percentile(double[] sortedLatencies, double percentile)
    {
        if (sortedLatencies.Length == 0)
        {
            return 0;
        }

        var rank = (int)Math.Ceiling(percentile / 100d * sortedLatencies.Length);
        return sortedLatencies[Math.Clamp(rank - 1, 0, sortedLatencies.Length - 1)];
    }

    private static void Increment(Dictionary<string, int> counter, string key)
        => counter[key] = counter.GetValueOrDefault(key) + 1;
}

MVC_App.ApiClient/MvcAppApiClient.cs

A deliberate 409 or 500 is a test result, not a reason to stop the whole run. This wrapper returns ApiCallResult for HTTP and transport failures and applies a 30-second timeout, allowing the reporting code to compare the outcomes consistently.

using System.Diagnostics;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;

namespace MVC_App.ApiClient;

/// <summary>
/// Thin wrapper over <see cref="HttpClient"/> that never throws on a non-success status code,
/// because here the status code <em>is</em> the result. Every call returns an
/// <see cref="ApiCallResult"/> carrying the expected and the actual status side by side.
/// </summary>
public sealed class MvcAppApiClient : IDisposable
{
    private readonly HttpClient _http;
    private readonly ClientOptions _options;

    private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web);

    public MvcAppApiClient(ClientOptions options)
    {
        _options = options;

        var handler = new HttpClientHandler
        {
            // The app calls UseHttpsRedirection, so an http base address answers 307 before the
            // request ever reaches a controller. Following redirects keeps `--url http://...`
            // usable; the API endpoints themselves never redirect, they answer 401 directly.
            AllowAutoRedirect = true,

            // The default per-server connection limit would serialise a concurrent load run
            // behind a handful of sockets and make the measured latencies meaningless.
            MaxConnectionsPerServer = Math.Max(options.Concurrency * 2, 16)
        };

        if (options.Insecure)
        {
            handler.ServerCertificateCustomValidationCallback =
                HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;
        }

        _http = new HttpClient(handler)
        {
            BaseAddress = options.BaseAddress,
            Timeout = TimeSpan.FromSeconds(30)
        };

        _http.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
        _http.DefaultRequestHeaders.UserAgent.ParseAdd("MVC_App.ApiClient/1.0");
    }

    /// <summary>
    /// Sends one request. <paramref name="apiKey"/> defaults to the configured key; pass an empty
    /// string to send no key at all (the 401 case) or a wrong value to test key rejection.
    /// </summary>
    public async Task<ApiCallResult> SendAsync(
        string scenario,
        HttpMethod method,
        string path,
        int expectedStatus,
        object? body = null,
        string? apiKey = null,
        bool retainBody = true,
        CancellationToken cancellationToken = default)
    {
        var stopwatch = Stopwatch.StartNew();

        using var request = new HttpRequestMessage(method, path);

        var key = apiKey ?? _options.ApiKey;
        if (!string.IsNullOrEmpty(key))
        {
            request.Headers.TryAddWithoutValidation("X-Api-Key", key);
        }

        if (body is not null)
        {
            request.Content = new StringContent(
                JsonSerializer.Serialize(body, JsonOptions), Encoding.UTF8, "application/json");
        }

        try
        {
            using var response = await _http.SendAsync(request, cancellationToken);

            // Always read the body to completion, even when it is being discarded: leaving it
            // unread would hold the connection open and skew the timings of every later call.
            var responseBody = await response.Content.ReadAsStringAsync(cancellationToken);
            stopwatch.Stop();

            var (category, type, traceId) = ApiCallResult.ReadProblemFields(responseBody);

            return new ApiCallResult(
                scenario,
                method,
                path,
                expectedStatus,
                (int)response.StatusCode,
                category,
                type,
                traceId,
                // A load run holds thousands of results; keeping every body would retain
                // megabytes of JSON nobody reads.
                retainBody ? responseBody : string.Empty,
                stopwatch.Elapsed);
        }
        catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
        {
            stopwatch.Stop();
            return ApiCallResult.Transport(
                scenario, method, path, expectedStatus, Describe(ex), stopwatch.Elapsed);
        }
    }

    /// <summary>
    /// Turns the usual "can't reach the app" failures into a message that says what to do, rather
    /// than a nested exception chain.
    /// </summary>
    private string Describe(Exception ex)
    {
        var root = ex;
        while (root.InnerException is not null)
        {
            root = root.InnerException;
        }

        var message = $"{ex.GetType().Name}: {root.Message}";

        if (root.Message.Contains("certificate", StringComparison.OrdinalIgnoreCase))
        {
            message += "  ->  run 'dotnet dev-certs https --trust', or re-run this client with --insecure.";
        }
        else if (root.Message.Contains("refused", StringComparison.OrdinalIgnoreCase) ||
                 root.Message.Contains("actively refused", StringComparison.OrdinalIgnoreCase))
        {
            message += $"  ->  is MVC_App running on {_options.BaseAddress}?";
        }

        return message;
    }

    public void Dispose() => _http.Dispose();
}

MVC_App.ApiClient/Program.cs

The console starts by choosing a scenario or load run, then sets up cancellation and the API wrapper. It masks the credential in its display and uses exit codes to report the outcome. The telemetry to inspect is produced by the backend; this executable does not register OTel providers.

using MVC_App.ApiClient;

Console.OutputEncoding = System.Text.Encoding.UTF8;

if (!ClientOptions.TryParse(args, out var options, out var error))
{
    if (!string.IsNullOrEmpty(error))
    {
        Console.Error.WriteLine(error);
        Console.Error.WriteLine();
    }

    Console.WriteLine(ClientOptions.Usage);
    return string.IsNullOrEmpty(error) ? 0 : 2;
}

using var cancellation = new CancellationTokenSource();
Console.CancelKeyPress += (_, eventArgs) =>
{
    eventArgs.Cancel = true;
    cancellation.Cancel();
};

Console.WriteLine(new string('=', 92));
Console.WriteLine("MVC_App API client");
Console.WriteLine(new string('=', 92));
Console.WriteLine($"  Base address   {options.BaseAddress}");
Console.WriteLine($"  API key        {Mask(options.ApiKey)} (sent as the X-Api-Key header)");

using var client = new MvcAppApiClient(options);

return options.Load
    ? await RunLoadAsync(client, options, cancellation.Token)
    : await RunScenariosAsync(client, options, cancellation.Token);

// ---------------------------------------------------------------------------

static async Task<int> RunScenariosAsync(
    MvcAppApiClient client, ClientOptions options, CancellationToken cancellationToken)
{
    Console.WriteLine($"  Iterations     {options.Iterations}");
    if (options.Delay > TimeSpan.Zero)
    {
        Console.WriteLine($"  Delay          {options.Delay.TotalMilliseconds:F0} ms between calls");
    }
    Console.WriteLine();
    Console.WriteLine("  Every call below is logged by the app with an error.category field, so the");
    Console.WriteLine("  same run is visible in Kibana under the mvcapp-logs-* data view.");

    var runner = new ScenarioRunner(client, options);

    try
    {
        for (var iteration = 1; iteration <= options.Iterations; iteration++)
        {
            await runner.RunAsync(iteration, cancellationToken);
        }
    }
    catch (OperationCanceledException)
    {
        Console.WriteLine();
        Console.WriteLine("Cancelled. Reporting on the calls completed so far.");
    }

    ConsoleReport.WriteSummary(runner.Results, options);

    // 0 when every call returned the status it was supposed to, 1 otherwise, so this is usable as
    // a smoke test in a build pipeline and not only as a telemetry generator.
    return runner.Results.Count > 0 && runner.Results.All(r => r.Matched) ? 0 : 1;
}

static async Task<int> RunLoadAsync(
    MvcAppApiClient client, ClientOptions options, CancellationToken cancellationToken)
{
    var units = LoadPlan.Build(options, out var idBase);

    if (units.Count == 0)
    {
        Console.Error.WriteLine("Nothing to do: every request count is zero.");
        return 2;
    }

    LoadReport.WritePlan(options, units, idBase);

    var runner = new LoadRunner(client, options, units);
    var stats = await runner.RunAsync(cancellationToken);

    if (cancellationToken.IsCancellationRequested)
    {
        Console.WriteLine();
        Console.WriteLine("Cancelled. Reporting on the requests completed so far.");
    }

    LoadReport.WriteResult(stats, runner, options, idBase);

    return stats.Unexpected == 0 && !cancellationToken.IsCancellationRequested ? 0 : 1;
}

static string Mask(string key)
    => key.Length <= 6 ? new string('*', key.Length) : $"{key[..3]}{new string('*', key.Length - 6)}{key[^3..]}";

MVC_App.ApiClient/ScenarioRunner.cs

The scenario walks through successful CRUD and known failure cases, then attempts cleanup. Validation errors, missing records, conflicts and simulated dependency failures are intentional here. Keep that test traffic separate from a dashboard meant to measure normal availability.

namespace MVC_App.ApiClient;

/// <summary>
/// Walks the whole API surface once, deliberately producing every status code the ELK dashboard
/// charts: 200, 201, 204, 400, 401, 403, 404, 405, 409, 500 and 503.
///
/// The happy-path calls use a department and an employee the run creates for itself (ids in the
/// 900000+ range, well clear of real data) and deletes again at the end, so the suite can be run
/// repeatedly against the same database.
/// </summary>
public sealed class ScenarioRunner
{
    private readonly MvcAppApiClient _client;
    private readonly ClientOptions _options;
    private readonly List<ApiCallResult> _results = [];

    public ScenarioRunner(MvcAppApiClient client, ClientOptions options)
    {
        _client = client;
        _options = options;
    }

    public IReadOnlyList<ApiCallResult> Results => _results;

    public async Task RunAsync(int iteration, CancellationToken cancellationToken)
    {
        // Unique per run so a repeat pass doesn't collide with leftovers from a previous one.
        var deptNo = 900_000 + Random.Shared.Next(1, 90_000);
        var empNo = 900_000 + Random.Shared.Next(1, 90_000);
        var deptName = $"QA-Dept-{deptNo}";

        Console.WriteLine();
        Console.WriteLine($"--- Iteration {iteration}: department {deptNo} '{deptName}', employee {empNo} ---");

        await ConnectivityAndAuthAsync(cancellationToken);
        await DepartmentCrudAsync(deptNo, deptName, cancellationToken);
        await EmployeeCrudAsync(empNo, deptName, cancellationToken);
        await NotFoundConditionsAsync(cancellationToken);
        await ServerErrorConditionsAsync(cancellationToken);
        await SimulatedFaultsAsync(cancellationToken);
        await CleanupAsync(empNo, deptNo, cancellationToken);
    }

    // -----------------------------------------------------------------------
    // Connectivity, 200 and the 401 condition
    // -----------------------------------------------------------------------

    private async Task ConnectivityAndAuthAsync(CancellationToken cancellationToken)
    {
        await RecordAsync("Liveness ping (anonymous)",
            HttpMethod.Get, "/api/diagnostics/ping", 200, cancellationToken: cancellationToken);

        await RecordAsync("401: no API key supplied",
            HttpMethod.Get, "/api/departments", 401, apiKey: string.Empty, cancellationToken: cancellationToken);

        await RecordAsync("401: wrong API key supplied",
            HttpMethod.Get, "/api/departments", 401, apiKey: "not-a-real-key", cancellationToken: cancellationToken);

        await RecordAsync("401: raised inside the endpoint",
            HttpMethod.Get, "/api/diagnostics/unauthorized", 401, cancellationToken: cancellationToken);

        await RecordAsync("403: authenticated but not permitted",
            HttpMethod.Get, "/api/diagnostics/forbidden", 403, cancellationToken: cancellationToken);

        await RecordAsync("200: API key accepted",
            HttpMethod.Get, "/api/diagnostics/whoami", 200, cancellationToken: cancellationToken);
    }

    // -----------------------------------------------------------------------
    // Department: GET / POST / PUT / DELETE
    // -----------------------------------------------------------------------

    private async Task DepartmentCrudAsync(int deptNo, string deptName, CancellationToken cancellationToken)
    {
        var department = new { DeptNo = deptNo, DeptName = deptName, Location = "Pune" };

        await RecordAsync("200: list all departments",
            HttpMethod.Get, "/api/departments", 200, cancellationToken: cancellationToken);

        await RecordAsync("201: create a department",
            HttpMethod.Post, "/api/departments", 201, body: department, cancellationToken: cancellationToken);

        await RecordAsync("200: read the department back",
            HttpMethod.Get, $"/api/departments/{deptNo}", 200, cancellationToken: cancellationToken);

        await RecordAsync("200: read the department by name",
            HttpMethod.Get, $"/api/departments/by-name/{Uri.EscapeDataString(deptName)}", 200,
            cancellationToken: cancellationToken);

        await RecordAsync("409: create the same department twice",
            HttpMethod.Post, "/api/departments", 409, body: department, cancellationToken: cancellationToken);

        await RecordAsync("400: create with an empty DeptName",
            HttpMethod.Post, "/api/departments", 400,
            body: new { DeptNo = deptNo + 1, DeptName = "", Location = "Pune" },
            cancellationToken: cancellationToken);

        await RecordAsync("200: update the department",
            HttpMethod.Put, $"/api/departments/{deptNo}", 200,
            body: new { DeptNo = deptNo, DeptName = deptName, Location = "Mumbai" },
            cancellationToken: cancellationToken);

        await RecordAsync("400: route id does not match the body",
            HttpMethod.Put, $"/api/departments/{deptNo}", 400,
            body: new { DeptNo = deptNo + 7, DeptName = deptName, Location = "Mumbai" },
            cancellationToken: cancellationToken);
    }

    // -----------------------------------------------------------------------
    // Employee: GET / POST / PUT / DELETE
    // -----------------------------------------------------------------------

    private async Task EmployeeCrudAsync(int empNo, string deptName, CancellationToken cancellationToken)
    {
        var employee = new
        {
            EmpNo = empNo,
            EmpName = $"QA Employee {empNo}",
            Salary = 55_000,
            DeptName = deptName,
            Designation = "Engineer"
        };

        await RecordAsync("200: list all employees",
            HttpMethod.Get, "/api/employees", 200, cancellationToken: cancellationToken);

        await RecordAsync("201: create an employee",
            HttpMethod.Post, "/api/employees", 201, body: employee, cancellationToken: cancellationToken);

        await RecordAsync("200: read the employee back",
            HttpMethod.Get, $"/api/employees/{empNo}", 200, cancellationToken: cancellationToken);

        await RecordAsync("200: list employees in that department",
            HttpMethod.Get, $"/api/employees?deptName={Uri.EscapeDataString(deptName)}", 200,
            cancellationToken: cancellationToken);

        await RecordAsync("409: create the same employee twice",
            HttpMethod.Post, "/api/employees", 409, body: employee, cancellationToken: cancellationToken);

        await RecordAsync("400: create with a zero salary",
            HttpMethod.Post, "/api/employees", 400,
            body: new
            {
                EmpNo = empNo + 1,
                EmpName = "QA",
                Salary = 0,
                DeptName = deptName,
                Designation = "Engineer"
            },
            cancellationToken: cancellationToken);

        await RecordAsync("200: update the employee",
            HttpMethod.Put, $"/api/employees/{empNo}", 200,
            body: employee,
            cancellationToken: cancellationToken);
    }

    // -----------------------------------------------------------------------
    // The not-found condition, from five different directions
    // -----------------------------------------------------------------------

    private async Task NotFoundConditionsAsync(CancellationToken cancellationToken)
    {
        await RecordAsync("404: department id that does not exist",
            HttpMethod.Get, "/api/departments/987654", 404, cancellationToken: cancellationToken);

        await RecordAsync("404: department name that does not exist",
            HttpMethod.Get, "/api/departments/by-name/NoSuchDepartment", 404, cancellationToken: cancellationToken);

        await RecordAsync("404: employee id that does not exist",
            HttpMethod.Get, "/api/employees/987654", 404, cancellationToken: cancellationToken);

        await RecordAsync("404: filter by a department that does not exist",
            HttpMethod.Get, "/api/employees?deptName=NoSuchDepartment", 404, cancellationToken: cancellationToken);

        await RecordAsync("404: update a department that does not exist",
            HttpMethod.Put, "/api/departments/987654", 404,
            body: new { DeptNo = 987654, DeptName = "Ghost", Location = "Nowhere" },
            cancellationToken: cancellationToken);

        await RecordAsync("404: delete an employee that does not exist",
            HttpMethod.Delete, "/api/employees/987654", 404, cancellationToken: cancellationToken);

        await RecordAsync("404: raised inside the endpoint",
            HttpMethod.Get, "/api/diagnostics/not-found", 404, cancellationToken: cancellationToken);

        await RecordAsync("404: route that matches no endpoint at all",
            HttpMethod.Get, "/api/no-such-resource", 404, cancellationToken: cancellationToken);

        await RecordAsync("405: correct route, wrong verb",
            HttpMethod.Delete, "/api/departments", 405, cancellationToken: cancellationToken);
    }

    // -----------------------------------------------------------------------
    // The 500 condition, both classified and unanticipated
    // -----------------------------------------------------------------------

    private async Task ServerErrorConditionsAsync(CancellationToken cancellationToken)
    {
        await RecordAsync("500: classified server-side failure",
            HttpMethod.Get, "/api/diagnostics/server-error", 500, cancellationToken: cancellationToken);

        await RecordAsync("500: unanticipated exception (DivideByZero)",
            HttpMethod.Get, "/api/diagnostics/unhandled", 500, cancellationToken: cancellationToken);

        await RecordAsync("503: dependency unavailable",
            HttpMethod.Get, "/api/diagnostics/unavailable", 503, cancellationToken: cancellationToken);

        await RecordAsync("400: validation failure raised directly",
            HttpMethod.Get, "/api/diagnostics/bad-request", 400, cancellationToken: cancellationToken);

        await RecordAsync("409: conflict raised directly",
            HttpMethod.Get, "/api/diagnostics/conflict", 409, cancellationToken: cancellationToken);
    }

    // -----------------------------------------------------------------------
    // The same conditions, raised on the real resource routes via ?simulate=
    // -----------------------------------------------------------------------

    private async Task SimulatedFaultsAsync(CancellationToken cancellationToken)
    {
        await RecordAsync("401 on a real route (?simulate=401)",
            HttpMethod.Get, "/api/departments?simulate=401", 401, cancellationToken: cancellationToken);

        await RecordAsync("404 on a real route (?simulate=404)",
            HttpMethod.Get, "/api/employees?simulate=404", 404, cancellationToken: cancellationToken);

        await RecordAsync("500 on a real route (?simulate=500)",
            HttpMethod.Get, "/api/departments?simulate=500", 500, cancellationToken: cancellationToken);

        await RecordAsync("500 on a write route (?simulate=500)",
            HttpMethod.Post, "/api/employees?simulate=500", 500,
            body: new { EmpNo = 1, EmpName = "Ignored", Salary = 1, DeptName = "Ignored", Designation = "Ignored" },
            cancellationToken: cancellationToken);

        await RecordAsync("404 on a delete route (?simulate=404)",
            HttpMethod.Delete, "/api/departments/1?simulate=404", 404, cancellationToken: cancellationToken);
    }

    // -----------------------------------------------------------------------
    // 204, and leaving the database as we found it
    // -----------------------------------------------------------------------

    private async Task CleanupAsync(int empNo, int deptNo, CancellationToken cancellationToken)
    {
        await RecordAsync("204: delete the employee",
            HttpMethod.Delete, $"/api/employees/{empNo}", 204, cancellationToken: cancellationToken);

        await RecordAsync("404: delete the same employee again",
            HttpMethod.Delete, $"/api/employees/{empNo}", 404, cancellationToken: cancellationToken);

        await RecordAsync("204: delete the department",
            HttpMethod.Delete, $"/api/departments/{deptNo}", 204, cancellationToken: cancellationToken);

        await RecordAsync("404: delete the same department again",
            HttpMethod.Delete, $"/api/departments/{deptNo}", 404, cancellationToken: cancellationToken);
    }

    // -----------------------------------------------------------------------

    private async Task RecordAsync(
        string scenario,
        HttpMethod method,
        string path,
        int expectedStatus,
        object? body = null,
        string? apiKey = null,
        CancellationToken cancellationToken = default)
    {
        var result = await _client.SendAsync(
            scenario, method, path, expectedStatus, body, apiKey, cancellationToken: cancellationToken);
        _results.Add(result);
        ConsoleReport.WriteCallLine(result, _options.Verbose);

        if (_options.Delay > TimeSpan.Zero)
        {
            await Task.Delay(_options.Delay, cancellationToken);
        }
    }
}

 

Solution and observability support

These files connect the application exporters to storage and Kibana. Read the Collector and Logstash configuration together to see why logs take a different route from traces and metrics.

compose.yaml

Runs Elasticsearch and Kibana 9.5.4, Collector Contrib 0.161.0 and Logstash 9.5.4 on a telemetry bridge network. Publishes only loopback 9200/5601/4318. Mounts Collector configuration, shared log files and Logstash pipeline, and persists Logstash read offsets in logstash-data. Applications/databases are not services in this Compose file. Elasticsearch has no explicitly managed named data volume here.

name: mvc-app-observability

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.5.4
    environment:
      discovery.type: single-node
      xpack.security.enabled: "false"
      xpack.security.http.ssl.enabled: "false"
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
    ports:
      - "127.0.0.1:9200:9200"
    networks:
      - telemetry
    mem_limit: 1g
    healthcheck:
      test: ["CMD-SHELL", "curl -s http://localhost:9200/_cluster/health | grep -q '\"status\":\"green\"\\|\"status\":\"yellow\"'"]
      interval: 10s
      timeout: 5s
      retries: 30

  kibana:
    image: docker.elastic.co/kibana/kibana:9.5.4
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    ports:
      - "127.0.0.1:5601:5601"
    depends_on:
      elasticsearch:
        condition: service_healthy
    networks:
      - telemetry
    mem_limit: 2g

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.161.0
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./observability/otel-collector.yaml:/etc/otelcol/config.yaml:ro
      - ./observability/otel-output:/otel-output
    ports:
      - "127.0.0.1:4318:4318"
    depends_on:
      elasticsearch:
        condition: service_healthy
    networks:
      - telemetry
    mem_limit: 256m

  logstash:
    image: docker.elastic.co/logstash/logstash:9.5.4
    environment:
      LS_JAVA_OPTS: "-Xms512m -Xmx512m"
      XPACK_MONITORING_ENABLED: "false"
      # One input event here is a whole OTLP export batch (~870KB, 512 log records), not a
      # single log line, so the stock 8 workers x 125 events would hold ~1000 of them in
      # flight. Keep the in-flight set small and size the heap for it.
      PIPELINE_WORKERS: "2"
      PIPELINE_BATCH_SIZE: "8"
    volumes:
      - ./observability/logstash/pipeline:/usr/share/logstash/pipeline:ro
      - ./observability/otel-output:/otel-output:ro
      # Persists the file input's sincedb (read offsets, see otel-logs.conf). Without it a
      # container recreate resets every offset and the whole otel-output backlog is replayed
      # into Elasticsearch as duplicate documents.
      - logstash-data:/usr/share/logstash/data
    depends_on:
      elasticsearch:
        condition: service_healthy
      otel-collector:
        condition: service_started
    networks:
      - telemetry
    mem_limit: 1g

networks:
  telemetry:
    driver: bridge

volumes:
  logstash-data:

observability/otel-collector.yaml

Accepts OTLP/HTTP on 4318. Memory limiter and batch processor precede every exporter. Traces/metrics go directly to Elasticsearch's OTel-compatible data streams; logs go to a rotating JSONL file for Logstash. The Elasticsearch queue/retry settings do not make a durable disk-backed trace/metric spool.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 192
    spike_limit_mib: 48
  batch:
    timeout: 5s
    send_batch_size: 512

exporters:
  elasticsearch:
    endpoints: ["http://elasticsearch:9200"]
    require_data_stream: true
    # No logs_index/traces_index/metrics_index override: OTel mapping mode writes through
    # Elasticsearch data streams (require_data_stream=true) and only auto-creates data
    # streams matching Elasticsearch's built-in otel-data index templates, e.g.
    # logs-generic.otel-default, metrics-generic.otel-default, traces-generic.otel-default.
    # A custom static index name here does not match those templates and fails with
    # index_not_found_exception, so the exporter's own default naming is used instead.
    #logs_index: "mvcapp-logs"
    mapping:
      allowed_modes: [otel, ecs]
    sending_queue:
      enabled: true
      queue_size: 256
    retry:
      enabled: true
      max_retries: 5
      initial_interval: 1s
      max_interval: 10s
  # Logs are handed to Logstash instead of going straight to Elasticsearch, so a real
  # Logstash pipeline sits in the path (parsing, enrichment) before indexing.
  # Written as newline-delimited OTLP JSON (one ExportLogsServiceRequest per line);
  # Logstash tails this file from the shared otel-logs volume.
  file/logs:
    path: /otel-output/logs.jsonl
    format: json
    rotation:
      max_megabytes: 10
      max_days: 3
      max_backups: 3

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [elasticsearch]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [elasticsearch]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [file/logs]

observability/logstash/pipeline/otel-logs.conf

Tails split JSON lines with the json codec and persistent sincedb offsets. Ruby decodes OTLP AnyValue, traverses resourceLogs/scopeLogs/logRecords and emits one small event per record, avoiding nested-split copying costs. Normalizes legacy http.route.matched, copies resource/scope/trace fields, converts timestamps, cancels the wrapper and indexes mvcapp-logs-YYYY.MM.dd.

# Reads the newline-delimited OTLP JSON that the OpenTelemetry Collector's file exporter
# writes for the "logs" pipeline (see observability/otel-collector.yaml), flattens each
# OTLP LogRecord out of its resourceLogs/scopeLogs/logRecords nesting, and indexes it into
# Elasticsearch. This is the one signal routed through Logstash; traces and metrics still
# go straight from the Collector to Elasticsearch (see observability/otel-collector.yaml).

input {
  file {
    path => "/otel-output/logs*.jsonl*"
    start_position => "beginning"
    sincedb_path => "/usr/share/logstash/data/sincedb-otel-logs"
    # The file input already splits the file into discrete lines itself (one line per
    # ExportLogsServiceRequest) before handing each to the codec, so the codec here only
    # needs to decode one already-isolated JSON document per call. `json_lines` is for
    # stream inputs (tcp/udp/stdin) that haven't split lines yet; used on a `file` input it
    # buffers forever waiting for a trailing delimiter that was already stripped, and no
    # event is ever decoded.
    codec => json {
      ecs_compatibility => disabled
    }
  }
}

filter {
  # Each line is one ExportLogsServiceRequest carrying many resources, scopes and log
  # records (the Collector batches up to `send_batch_size` records per export, so a line
  # is routinely ~870KB holding 512 records).
  #
  # This fan-out is deliberately NOT done with nested `split` filters. `split` deep-copies
  # the entire event once per array element, and the copy is made while the field still
  # holds the whole array -- so one 870KB line became 512 copies of itself, several hundred
  # megabytes, and Logstash died with OutOfMemoryError before indexing anything. Emitting
  # each record as a fresh, small event via new_event_block keeps memory proportional to
  # the line rather than to (line x records).
  ruby {
    code => '
      # Decodes an OTLP AnyValue (e.g. {"stringValue" => "x"}, {"intValue" => "5"}) into a
      # plain Ruby value. OTLP JSON always wraps scalars this way.
      def otlp_any_value(any_value)
        return nil if any_value.nil?
        if any_value.key?("stringValue")
          any_value["stringValue"]
        elsif any_value.key?("intValue")
          any_value["intValue"].to_i
        elsif any_value.key?("doubleValue")
          any_value["doubleValue"]
        elsif any_value.key?("boolValue")
          any_value["boolValue"]
        elsif any_value.key?("arrayValue")
          (any_value.dig("arrayValue", "values") || []).map { |v| otlp_any_value(v) }
        elsif any_value.key?("kvlistValue")
          otlp_attrs_to_hash(any_value.dig("kvlistValue", "values"))
        else
          nil
        end
      end

      # Attribute names the app has since renamed, mapped old => current. Elasticsearch
      # expands dots in a field name into an object path, so a record carrying the old
      # "http.route.matched" forces log_attributes.http.route to be an *object*, which then
      # rejects every record carrying the current "http.route" string (and vice versa) with
      # a document_parsing_exception. MVC_App/Middleware/RequestLogging.cs renamed the
      # attribute to "http.route_matched" for exactly this reason, but the older records
      # already sitting in otel-output still use the pre-rename key, so normalise on read.
      # A method rather than a constant: this filter body is evaluated once per event, so a
      # constant assignment here re-initialises it every time and floods the log with
      # "already initialized constant" warnings.
      def normalize_attribute_key(key)
        key == "http.route.matched" ? "http.route_matched" : key
      end

      # OTLP attributes are a list of {"key" => ..., "value" => AnyValue}; turn that into
      # a flat hash keyed by attribute name.
      def otlp_attrs_to_hash(attributes)
        (attributes || []).each_with_object({}) do |kv, hash|
          key = normalize_attribute_key(kv["key"])
          # A record that somehow carries both spellings keeps the current one.
          next if hash.key?(key) && key != kv["key"]
          hash[key] = otlp_any_value(kv["value"])
        end
      end

      (event.get("resourceLogs") || []).each do |resource_logs|
        resource_attrs = otlp_attrs_to_hash(resource_logs.dig("resource", "attributes"))
        service_name = resource_attrs["service.name"]
        deployment_environment = resource_attrs["deployment.environment.name"]

        (resource_logs["scopeLogs"] || []).each do |scope_logs|
          scope_name = scope_logs.dig("scope", "name")

          (scope_logs["logRecords"] || []).each do |record|
            out = LogStash::Event.new
            out.set("message", otlp_any_value(record["body"]) || "")
            out.set("severity_text", record["severityText"])
            out.set("severity_number", record["severityNumber"])
            out.set("trace_id", record["traceId"])
            out.set("span_id", record["spanId"])
            out.set("log_attributes", otlp_attrs_to_hash(record["attributes"]))
            out.set("resource_attributes", resource_attrs)
            out.set("service_name", service_name)
            out.set("deployment_environment", deployment_environment)
            out.set("scope_name", scope_name)

            time_unix_nano = record["timeUnixNano"] || record["observedTimeUnixNano"]
            if time_unix_nano
              out.set("@timestamp", LogStash::Timestamp.at(time_unix_nano.to_i / 1_000_000_000.0))
            end

            new_event_block.call(out)
          end
        end
      end

      # The raw ExportLogsServiceRequest wrapper is not itself a log line. Everything useful
      # has been emitted above, so drop it (this also covers empty batches).
      event.cancel
    '
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "mvcapp-logs-%{+YYYY.MM.dd}"
  }
}

The code for this article can be downloaded from this link.

The Execution of this application can be read from the following video.

Conclusion

A slow page or a failed API call is much easier to investigate when you can follow it beyond the response status. In this solution, the request logs describe the outcome, the traces connect HTTP and SQL work, and the metrics show whether the behavior is isolated or recurring.

Start with one successful request and one deliberate failure. Find their completion logs in Kibana, inspect the status and error fields, and use the trace ID to follow the related events. Once that path makes sense, the workload clients and dashboard panels become useful ways to explore larger patterns not just ways to produce more data.