Capturing and Tracing All HTTP Requests in C# and .Net

Modern applications are complex and often rely on a large number of external resources increasingly accessed using HTTP – for example most Azure services are interacted with using the HTTP protocol.

That being the case it can be useful to get a view of the requests your application is making and while this can be done with a tool like Fiddler that’s not always convenient in a production environment.

If you’re using the HttpClient class another option is to pass a custom message handler to it’s constructor but this relies on you being in direct control of all the code making HTTP requests and that’s unlikely.

A simple way of capturing this information without getting into all the unpleasantness of writing a TCP listener or HTTP proxy is to use the System.Diagnostics namespace. From .net 4.5 onwards the framework has been writing HTTP events to the System.Diagnostics.Eventing.FrameworkEventSource source. This isn’t well documented and I found the easiest way to figure out what events are available, and their Event IDs, is to read the source.

Once you’ve found the HTTP events it’s quite straightforward to write an event listener that listens to this source. The below class will do this and output the details to the trace writer (so you can view it in the Visual Studio Debug Output window) but you can easily output it to a file, table storage, or any other output format of your choosing.

To set it running all you need to do is instantiate the class.

If you’d like to see this kind of data and much more, collected, correlated and analysed then you might want to check out my project Hub Analytics that is currently running a free beta.

Changing the App Service Plan of an Azure App Service

To allow a number of App Services to scale independently I needed to pull one of them out of an App Service Plan where it had lived with 3 others to sit in it’s own plan – experience had shown me that it’s scaling characteristics are really quite different from the other App Services.

You can do this straightforwardly and pretty much instantly either in the Portal (there’s a Change App Service Plan option in Settings) or with PowerShell (with the Set-AzureRmAppServicePlan cmdlet).

Super simple – but I did encounter one gotcha. This doesn’t move any deployment slots you might have created and so you end up in a situation with the main App Service sat in one App Service Plan and it’s deployment slots in another which probably isn’t what you want and, in any case, Azure won’t let you swap slots in different service plans.

The solution is simple: you can also move them between App Service Plans in the same way.

 

Serving Static Markdown Content from ASP.Net MVC to JavaScript

I recently moved a bunch of documentation into the Markdown format as I wanted to render it into multiple output formats and inside multiple hosting technologies – including an AngularJS based single page applicaiton.

While doing this I decided it would make sense to have a single source of truth for these files and so placed them as content inside my MVC 5 based website that is entirely public access. After dropping them into the website the first step is to configure ASP.Net to serve the content which involves adding the below to a web.config file:

This enables the Markdown to be served to a browser but if you try and download it from JavaScript you find it blocked by CORS. I solved this with a small ASP.Net module that is installed in the web.config above and the code for which is below:

Hope it’s useful to you.

Signalling API Unavailability

When doing upgrades of websites it’s often useful to be able to signal to users that your service is offline for maintenance either in part or in entirety which is quite straightforward to implement unless you’ve got something like an AngularJS or React app, that could well be cached in the browser, and that actually wants to respond to 503 status calls returned from a web based API. Then CORS has a habit of getting in the way.

To help with that I’ve just pushed this super simple and lightweight ASP.Net website to GitHub that will respond with a 503 status code to any request made of it while ensuring that the CORS protocol will succeed meaning that the 503 status code will make its way through to your own error handling.

It’s ideal for hosting in an Azure deployment slot during upgrades that swap slots.

Note: an alternative approach would be to use the URL rewriter in web.config. It’s not particularly intuitive or, to my taste, readable but I believe can be configured to perform the same task.

Accidental Fish Application Support v3.3.0 Release

Last night I published a minor update to this framework to GitHub and NuGet that adds new timer capabilities via the new ITimerFactory interface.

An interval timer is available that runs an action, then pauses for the specified on completion, and runs the task again until cancelled (either by the action itself or a cancellation token). A regular metronomic timer is available that runs an action n seconds irrespective of task duration. Importantly in the latter case if the action takes longer than the duration of the timer to complete it will be cancelled to prevent compounding overlapping action issues (excessive CPU usage, out of memory etc.).

I’ve also added an IBackoffFactory interface that allows the backoff policies to be created with custom backoff timings. The backoff policies continue to be directly injectable with their default timings.

I hope these additions are useful. Feedback is, as ever, welcome.

Microservice Analytics – Update

For those who’ve got in touch asking for a beta code – thank you for your interest, it’s very much appreciated and I’m nearly ready to start issuing them.

Before releasing them I wanted to make sure I was capturing user and session data properly and that that was getting integrated deep across the other measures and data I capture. Happily this is all working and I’ve just deployed it to my live site.

In this initial release, unless you configure data capture otherwise, then magic numbers will be generated for users and sessions (you can override this to provide your own IDs if you so choose) and these are correlated with everything else that is happening so that it is possible to look at the data from both perspectives:

  • What are your users doing and what system activity is that generating in your applications?
  • Given an system activity (say a SQL command or Web API call) which user initiated it and/or under which session?

There are some specific views to help with the above but in addition you can tag any user or session and then the whole user interface is filtered by that allowing you to explore quite organically this subset of data.

I’ve attached a couple of screenshots below (taken immediately after deploying the new code to live – so not many users and sessions captured yet!) and as I mentioned earlier I’m nearly ready to share those invite codes, thanks for your patience.

sessions

pinned

Microservice Analytics – New Analytic / Diagnostic Solution

I spend a lot of time working with highly distributed systems that use an SOA / microservice type architecture and increasingly are accessed through clients such as native apps, Cordova app, or JavaScript SPAs and it can be challenging to understand what is going on across multiple servers and devices when faults occur or you need to understand something.

I wanted a tool that was fairly non-invasive, had a highly interactive user interface, and would capture a wide breadth of data and was reasonably priced. There were solutions that did bits of what I wanted (particularly at the high end in terms of price) but nothing that really hit the sweet spot for me particularly in terms of being able to drill through and across data to form a coherent picture.

This sort of thing really is my cup of tea as a developer and so I went off and built it coming at this from two directions:

  1. When errors occur in the system I want to know everything that was happening around them across devices and servers and to be able to drill into it all and find related items.
  2. To understand how users are behaving and experiencing the system and to be able to jump off and drill into areas of interest.

I’ve been dogfooding on it for a while now and the first part is pretty much ready to roll and the second part has most of the plumbing in place and is getting expanded out over the next few weeks. It’s tentatively called Microservice Analytics and there are some screenshots below to whet your appetite.

It’s one of those projects that is almost endless in terms of where you might take it (I’ve got a list of enhancements as long as my arm just from my own usage) but at some point I need some feedback and to figure out it its useful to anyone but me and so I’m just starting to issue beta invite codes.

If you’re interested get in touch with a comment below.

overview

related_summary

 

related_timeline

http

Azure Resource Manager and Powershell 1.0

Microsoft recently updated Azure Powershell to 1.0 in the process introducing a large number of breaking changes to the CmdLet’s. Essentially they’ve removed Switch-AzureMode and instead of that the Azure Resource Manager cmdlets have all had an Rm introduced to them so for example:

New-AzureResourceGroupDeployment

becomes

New-AzureRmResourceGroupDeployment

For the most part the errors on upgrading a Powershell script are obvious and a rename will get you going however somewhat confusingly there are now pairs of cmdlet’s that do the same thing but only effect the different cmdlet sets.

An example of one that caught me out is Select-AzureSubscription. I’ve got numerous Azure subscriptions and scripts that deploy into different subscriptions depending on what I’m doing. Previously I used Select-AzureSubscription along with Switch-AzureMode and this worked. The problem is that Select-AzureSubscription still works – but it has no effect on the cmdlets that use the Rm prefix.

This led, confusingly, to a resource group being created in a different subscription to the one I intended and resources with the same name as in my intended resource group (resources which already existed) being created, or attempted to be created, in this new resource group. They failed as they already existed with the same names elsewhere.

The fix was to use Select-AzureRmSubscription.

 

Accidental Fish Application Support v3.1.0 Release

This week I’ve published a couple of updates to this framework taking it to v3.1.0.

Release notes for v3.0.0 are here and for v3.1.0 here.

There are breaking changes moving to the v3.x series as I’ve completely overhauled the logging framework to allow for the introduction of robust logging providers deeply into the framework including a new NuGet package for Serilog. There is a sample showing how to use this package and the new logging framework.

In addition I’ve upgraded the queueing system with some new features.

Firstly there is brand new support for large message queues – as in queues with message sizes in the megabytes or gigabytes. This is surfaced through the ILargeMessageQueueFactory and ILargeMessageQueue interface and combines a standard queue with a blob repository. The ILargeMessageQueue interface is derived from the standard IAsynchronousQueue and so can be used interchangeably and transparently with existing queue consumers.

Finally the queues now provide access to message properties where the underlying queue supports this feature – for example Azure Service Bus queues, topics and subscriptions.

I’m planning to add support for DNX and Entity Framework 7 in the near future (weeks, not months).

As ever if you have any issues or feedback then you can get in touch here or over on GitHub.