Part of the series: Fun with OpenTelemetry .NET
I worked with a couple teams last year who both had the same feature request (more or less). Both teams wanted to add some contextual information to the next span to be created, not the current span (Activity.Current). One team was doing this type of custom ORM thing over a SQL backend. Their library has a lot of information but the SQL span only has basic info (like the stored procedure name). Another team was building a RESTful service and wanted to push some user information on the next HTTP span that would be made to service the current request. They wanted to add their specific great contextual information so it would show up in the trace visualization, which I think is instinctual and a good thing, but neither team could really solve this with what is provided in OpenTelemetry .NET.
In terms of OpenTelemetry what we’re talking about is span enrichment. OpenTelemetry .NET typically provides an options class and an “Enrich” callback for this type of thing.
Examples:
Great! So we’re done, right? Well, it wouldn’t be a very interesting blog if that was the case!
There are a couple of problems with these callbacks. The big problem is they are global things, configured when the application is starting up. You can’t exactly hand them a closure before you make a call. The second problem is the objects that are passed in are fixed by the instrumentation, which doesn’t understand your code. You will get something like HttpRequestMessage or SqlCommand. If the information you want to enrich is on the object, you are happy. If you want something else, you are sad. A lesser problem with this is if you have a bunch of developers working on an application you don’t really want them all messing with these special magic global things all calls flow through which also happen to be managed by your sensitive startup code, do you? It just sounds like a recipe for disaster!
So what can we do about this?
The good news is that the OpenTelemetry Specification specifically names some extension points SDKs must implement:
Span processor
opentelemetry-specification/sdk.md at main ยท open-telemetry/opentelemetry-specification (github.com)
Span processor is an interface which allows hooks for span start and end method invocations.
Span processors allow us to hook into the pipeline so we can mess with spans as they are started and/or stopped. Sounds like a good starting point, eh?
Enrichment scope
I’ve written a couple loggers using the ILogger/ILoggerProvider interfaces. One of the things that is kind of neat in that world is the notion of scopes. You can wrap some logic in a scope and that information will flow to any log message written under that scope.
It works like this:
using IDisposable scope = logger.BeginScope(MyUser);
logger.LogInformation("Operation completed successfully.");
That simple syntax will make MyUser available to the log engine.
My idea was to do the same type of thing but for enriching spans (Activity objects in OpenTelemetry .NET). Something like this:
using IDisposable scope = ActivityEnrichmentScope.Begin(
activity => activity.SetTag("user.name", MyUser.Name));
PerformSqlOperation();
That will solve the immediate ask but if you think about it, it also opens up some other interesting possible use cases. If you wanted to, you could use this to create a scope in middleware over your entire request pipeline and decorate every span in a request trace with some useful stuff your application knows about. This is exactly how ASP.NET Core pushes TraceId onto log messages. Or you could do something like wrap a background thread so that each time it wakes up to run it automatically tags its spans as background? Be creative with it, have fun.
Anyway, to make it work conceptually what we need that to do is create a closure that will flow with the current thread until some span processor runs and executes the delegate. Delegates and closures kind of suck for perf, so we should also provide a <TState> type of version too. I’m not going to cover that here because the code is ugly, but it is in the final library.
For the flow part Reiley actually already added a really neat mechanism to the OpenTelemetry .NET SDK to help with the flowing data with the thread: RuntimeContextSlot<T>. On .NET Core it’s going to be AsyncLocal. But it also has some magic to work on .NET Framework. The best part is we don’t need to worry about it, we can just use it!
All we really need to worry about is the closure/delegate. Here’s how we can do the scope:
internal sealed class ActionActivityEnrichmentScope : IDisposable
{
private static readonly RuntimeContextSlot<ActionActivityEnrichmentScope?> s_RuntimeContextSlot
= RuntimeContext.RegisterSlot<ActionActivityEnrichmentScope?>("otel.activity_enrichment_scope");
public static ActionActivityEnrichmentScope? Current => s_RuntimeContextSlot.Get();
private readonly Action<Activity> _EnrichmentAction;
public ActivityEnrichmentScopeBase? Parent { get; private set; }
public ActionActivityEnrichmentScope(Action<Activity> enrichmentAction)
{
_EnrichmentAction = enrichmentAction ?? throw new ArgumentNullException(nameof(enrichmentAction));
Parent = Current;
s_RuntimeContextSlot.Set(this);
}
public void Enrich(Activity activity)
=> _EnrichmentAction?.Invoke(activity);
public void Dispose()
{
if (s_RuntimeContextSlot.Get() == this)
{
s_RuntimeContextSlot.Set(Parent);
}
}
}
(The actual code, linked at the bottom, is a bit more complicated but that is the gist of it.)
The next thing we need to do is add a span processor that will apply any active scopes to the spans it sees:
internal sealed class ActivityEnrichmentScopeProcessor : BaseProcessor<Activity>
{
public override void OnEnd(Activity activity)
{
ActivityEnrichmentScope? scope = ActivityEnrichmentScope.Current;
while (scope != null)
{
scope.Enrich(activity);
scope = scope.Parent;
}
}
}
That’s about it. Register that span processor where you configure the OpenTelemetry .NET SDK and start using the scopes.
Would Baggage work for this?
OpenTelemetry (spec + .NET) has a Baggage API you can use to attach data to a span. You could add your contextual information as baggage (so long as you can represent it as a string) and make a span processor that picks it up from there. Perfectly valid to do that, just make sure you remove things you don’t want to propagate downstream. Also worth noting is baggage is an immutable structure so adding or removing from it has an allocation + copy penalty. I think it was more intended to be used for stuff you want to add once and flow everywhere but it is there if you want to use it. The ActivityEnrichmentScope above has to allocate at least a class so neither solution comes for free. I do think ActivityEnrichmentScope will have less of an impact. Lots of items in baggage needing to be copied and having to convert things to strings that aren’t already strings would be expensive.
Try it out today!
ActivityEnrichmentScope is available in Macross.OpenTelemetry.Extensions or in the repo on GitHub.
Update: Decorating controllers for span enrichment
All the stuff above works in .NET Core and .NET Framework. One of my coworkers happens to be working in a .NET Framework MVC application and asked if there was a way to decorate an entire controller so that all of its spans would be tagged with the internal name for the area of the product. Below is what we cooked up, sharing just in case anyone else finds it useful. This won’t be part of the NuGet because I don’t want to add a dependency on System.Web.Mvc…
using System;
using System.Diagnostics;
using System.Web.Mvc;
namespace Awesome.Company
{
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method, Inherited = true, AllowMultiple = true)]
public sealed class TelemetryTagFilterAttribute : ActionFilterAttribute
{
private readonly string _HttpContextItemsKey;
private readonly string _TagName;
private readonly string _TagValue;
public TelemetryTagFilterAttribute(string tagName, string tagValue)
{
_HttpContextItemsKey = $"{nameof(TelemetryTagFilterAttribute)}.{_TagName}";
_TagName = tagName;
_TagValue = tagValue;
}
public override void OnActionExecuting(ActionExecutingContext filterContext)
{
if (filterContext != null)
{
filterContext.HttpContext.Items[_HttpContextItemsKey] = ActivityEnrichmentScope.Begin(EnrichActivity, this);
}
}
public override void OnActionExecuted(ActionExecutedContext filterContext)
{
if (filterContext != null && filterContext.HttpContext.Items[_HttpContextItemsKey] is IDisposable scope)
{
scope.Dispose();
filterContext.HttpContext.Items.Remove(_HttpContextItemsKey);
}
}
private static void EnrichActivity(Activity activity, TelemetryTagFilterAttribute filter)
=> activity.SetTag(filter._TagName, filter._TagValue);
}
}
Used like this:
[TelemetryTagFilter("awesomeapp.area", "ProductDetailPage")]
public class PDPController : Controller
{
}