I created a new plan this morning with a view to exploring the results further and I think I can provide some additional insight into this.
You shouldn’t have to “warm” a Premium Plan but to be sure and consistent I ran these tests after allowing for an idle / scale down period and then making a single request for a Mandelbrot.
The data here is based around a test making 32 concurrent requests to the Function App for a single Mandelbrot. Here is the graph for the initial run.
First if we consider the overall statistics for the full run – they are still not great. If I pop those into the comparison chart I used yesterday EP1 is still trailing – the blue column is yesterdays EP1 result and the green line todays.
Its improved – but its still poor. However if we look at the graph of the run over time we can see its something of a graph of two halves and I’ve highlighted two sections of it (with the highlight numbers in the top half):
There is a marked increase in response time and request per second rate between the two halves. Although I’m not tracking the instance IDs I would conclude that Azure Functions scaled up to involve a second App Service Instance and that resulted in the improved throughput.
To verify this I immediately ran the test again to take advantage of the increased resource availability in the Function Plan and that result is shown below along with another comparative graph of the run in context.
We can see here that the EP1 plan is now in the same kind of ballpark as Lambda and the EP2 plan. As two EP1 instances in play we are now running with a similar amount of total compute as the EP1 plan – just on two 210 ACU instances rather than one 420 ACU instance.
To obtain this level of performance we are sacrificing consumption based billing and moving to a baseline cost of £0.17 per hour (£125 per month) bursting to £0.34 per hour (£250 per month) to cover this low level of load.
Conclusions
I would argue this verifies yesterdays results – with a freshly deployed Function App we have obtained similar results and by looking at its behavior over time we can see how Azure Functions is adding resource to an EP1 plan then giving us similar total resource to the EP2 plan and similar results.
Every workload is different and I would always encourage this but based on this I would strongly suggest that if you’re using Premium Plan’s you dive into your workload and seek to understand if it is a cost effective use of your spend.
Update: the results below showed the EP1 plan to be a clear outlier in some of its performance results. I’ve since retested on a fresh EP1 plan and confirmed these results as accurate and been able to provide further insight into the performance: Azure Functions Performance – Update on EP1 Results – Azure From The Trenches.
I was recently asked if I would spend some time comparing AWS Lambda with Azure Functions at a meetup – of course, happily! As part of that preparing for that I did a bit of a dive into the performance aspects of the two systems and I think the results are interesting and useful and so I’m also going to share them here.
Test Methodology
Methodology may be a bit grand but here’s how I ran the tests.
The majority of the tests were conducted with SuperBenchmarker against systems deployed entirely in the UK (eu-west-2 on AWS and UK South on Azure). I interleaved the results – testing on AWS, testing on Azure, and ran the tests multiple times to ensure I was getting consistent results.
I’ve not focused on cold start as Mikhail Shilkov has covered that ground excellently and I really have nothing to add to his analysis:
I essentially focused on two sets of tests – an IO workload (flushing a queue and writing some blobs) and a compute workload (calculating a mandelbrot and returning it as an image).
All tests are making use of .NET Core 3.1 and I’ve tested on the following configurations:
Azure Functions – Consumption Plan
Azure Functions – EP1 Premium Plan
Azure Functions – EP2 Premium Plan
AWS Lambda – 256 Mb
AWS Lambda – 1024 Mb
AWS Lambda – 2048 Mb
Its worth noting that Lambda assigns a proportion of CPU(s) based on the allocated memory – more memory means more horsepower and potentially multiple cores (beyond the 1.8Gb mark if memory serves).
Queue Processing
For this test I preloaded a queue with 10,000 and 100,000 queue items and wrote the time the queue item was processed to a blob file (a file per queue item). The measured times are between the time the first blob was created and the time the last blob was created.
On Azure I made use of Azure Queue Storage and Azure Blob Storage and on AWS I used SQS and S3.
AWS was the clear winner of this test and from the shape of the data it appeared that AWS was accelerating faster than Azure – more eager to process more items but I would need to do further testing to compare. However it is possible the other services were a influencing factor. However its a reasonable IO test on common services by a function.
HTTP Trigger under steady load
This test was focused on a compute workload – essentially calculating a Mandelbrot. The Function / Lambda will generate n lambda’s based on a query parameter. The Mandelbrots are generated in parallel using the Task system.
32 concurrent requests, 1 Mandelbrot per request
Percentile and average response times can be seen in the graph below (lower is better):
With this low level of low all the services performed acceptable. The Azure Premium Plans strangely perform the worst with the EP1 service being particularly bad. I reran this several times and received similar results.
The range of response times (min, max) can be seen below alongside the average where we can see again followed by the total number of requests served over the 2 minute period:
32 concurrent requests, 8 Mandelbrots per request
In this test each request results in the Lambda / Function calculating the Mandelbrot 8 times in parallel and then returning one of the Mandelbrots as an image.
Percentile and average response times can be seen in the graph below (lower is better):
Things get a bit more interesting here. The level of compute is beyond the proportion of CPU assigned to the 256Mb Lambda and it struggles constantly. The Consumption Plan and EP1 Premium Plan fair a little better but are still impacted. However the 1024Mb and 2048Mb Lambda’s are comfortable – with the latter essentially being unchanged.
The range of response times (min, max) can be seen below alongside the average where we can see again followed by the total number of requests served over the 2 minute period:
I don’t think there’s much to add here – it largely plays out as you’d expect.
HTTP undergoing a load spike
In this test I ran at a low and steady rate of 2 concurrent requests for 1 Mandelbrot over 4 minutes. After around 1 to 2 minutes time I then loaded the system, independently, with a spike of 64 concurrent requests for 8 Mandelbrots.
Azure
First up Azure with the Consumption Plan:
Its clear to see in this graph where the additional load begins and, unfortunately, Azure really struggles with this. Served requests largely flatline throughout the spike. To provide more insight here’s a graph of the spike (note: I actually captured this from a later run but the results were the same as this first run).
Azure struggled to serve any of this. It didn’t fail any requests but performance really has nosedived.
I captured the same data for the EP1 and EP2 Premium Plans and these can be seen below:
Unfortunately Azure continues to struggle – even on an EP2 plan (costing £250 per month at a minimum). The spike data was broadly the same as in the Consumption plan run.
I would suggest this is due to Azure’s fairly monolithic architecture – all of our functions are running in shared resource and the more expensive requests can sink the entire shared space and Azure isn’t able to address this.
Lambda
First up the 256Mb Lambda:
We can see here that the modest 1 Mandelbrot requests made to the Lambda are untroubled by the Spike. You can see a slight rise in response time and drop in RPS when the additional load hits but overall the Lambda maintains consistent performance. You can see what is happening in the spike below:
Like in our earlier tests the 256 Mb Lambda struggles with the request for 8 Mandelbrot’s – but its performance is isolated away from the smaller requests due to Lambda’s more isolated architecture. The additional spikes showed characteristics similar to the runs measured earlier. The 1024 Mb and 2048 Mb run are shown below:
Again they run at a predicable and consistent rate. The graphs for the spikes behaved in line with performance of their respective consistent loads.
Spike Test Results Summary
Based on the above its unsurprising that the overall metrics are heavily in favour of AWS Lambda.
Concluding Thoughts
Its fascinating to see how the different architectures impact the scaling and performance characteristics of the service. With Lambda being based around containerized Functions then as long as the workload of a specific request fits within a containers capabilities performance remains fairly constant and consistent and any containers that are stretched have isolated impact.
As long as you measure, understand and plan for the resource requirements of your workloads Lambda can present a fairly consistent consumption based pricing scaling model.
Whereas Azure Functions uses the coarse App Service Instance scaling model – many functions are running within a single resource and this means that additional workload can stretch a resource beyond its capabilities and have an adverse effect on the whole system. Even spending more money has a limited impact for spikes – we would need resources that can manage the “peak peak” request workloads.
I’ve obviously only pushed these systems so far in these tests – enough to show the weaknesses in the Azure Functions compute architecture but not enough to really expose Lambda. That’s something to return to in a future installment.
In my view the Azure Functions infrastructure needs a major overall in order to be performance competitive. It has a FaaS programming model shackled to and held back by a traditional web server architecture. Though I am surprised by the EP1 results and will reach out to the Functions team.
As I’ve worked with a wider variety of cloud vendors in recent months I’m becoming increasingly unsure who Azure is a good fit for. To contextualise this a lot of my work has been based around fairly common application patterns: APIs, SPAs, data storage, message brokers, background activities. As some of this is self funded I’m often interested in cost but still want good performance for users.
For simple projects (lets say an SPA, an API and a database) you now have services like Digital Ocean which will deploy your app direct from GitHub and let you set up a database and CDN with a few lines of code or couple of button pushes in the portal. The portals are super easy to use and focused. Digital Ocean can also be cheap. If you’re a developer focused on code and product its about as simple as it gets.
Azure has some of this but its far less streamlined and doesn’t go as low on entry level price. It’s also mired in inconsistencies across its services.
So for a simple project – I don’t think it competes either on usability or price. And on the commericals – Microsoft are difficult to approach as a startup or as someone looking to migrate workloads. In contrast from my recent experiences AWS are far more aggressive in this regard.
At the other end of things if you want more control and access to more complex services you have AWS – like Azure they have a vast array of services for you to choose from. I would argue AWS has a slightly steeper learning curve than Azure – you quickly get involved with networking and IAM policies and roles – but its more consistent than Azure. Once you’ve got your head around those concepts things start to flow nicely. AWS feels like it was built bottom up (infrastructure upwards) and if you look at its history it was. Its benefiting from that now as it builds higher level components on top of that base. Azure in comparison feels (and was) built top down – it started with PaaS and has moved downwards. Unfortunately its harder to patch in a foundation and some of the issues we experience I would argue are due to that. I do think Azure has a better portal that brings deployed services together in a clearer way – but if you’re doing the more complex work that makes sense on something like Azure or AWS you’re almost certainly using Infrastructure as Code technologies (and if not – you should be!) so it becomes less of an advantage at this end of the scale.
And worryingly Microsoft have missed the two big recent advances in compute: serverless and ARM.
With serverless Azure Functions is barely a competitor to Lambda at this point. Its got some nice trimmings in things like Durable Functions but as a core serverless offering its slow both in terms of raw performance and cold start and its inflexible – its still suffering from being cobbled together like a monolithic Frankenstein’s monster from the parts Microsoft had lying around when Lambda was launched (the perils of Marketing Led Development).
On ARM, Amazon are on v2 of their Graviton support and you can obtain 20% to 40% cost savings by moving typical .NET 5 workloads onto ARM backed EC2 instances. Azure don’t even have anything in preview. How long will it be until AWS have got ARM rolled out in further services?
So ultimately who is Azure for? The only audience I’m coming up with are existing Microsoft customers who have large investments in the ecosystem or those locked into spending with Microsoft.
Otherwise I’m struggling to see why, if given a free choice, you would now choose it. I’d looked to put my latest side project on Azure – I felt I ought to give it another chance, I’m an Azure MVP for goodness sake – but I can’t find a compelling reason to and deploying it was painful. Performance experiments today (that will be published after my meetup talk tomorrow) were disappointing. Whereas I can find compelling reasons to deploy on AWS.
It genuinely saddens me to be writing this – I’ve had a lot of success with Azure over the years but Microsoft seem to have lost their way. In the chase for every feature and every customer the commercial focus has gone and it feels like they’re paying a price for chasing AWS from weak foundations. I’ll continue to engage with Product Teams and users as I can and I don’t think of myself as “walking away” from Azure but my latest project is headed to AWS. Commercially and productivity wise its a no brainer.
And all in it makes it very difficult to recommend Azure at this point if you have a free choice of cloud vendor.
Update: Its worth noting that some folk I trust, such as Howard van Roojen of Endjin, speak highly of Azure’s data platform capabilities and have delivered some serious systems based around that so as ever you do have to look at your use case. And if you’re looking at making a significant investment with a cloud vendor I highly recommend finding someone independent who has experience in the space and test with some representative workloads as soon as you can. Don’t rely on what the vendor is telling you. And don’t assume that because you’re using .NET that Azure is the answer – it may be, but it may not.
(I’m happy to engage with anyone about MS / Azure about this but I don’t think their is any new feedback here sadly – its a “more of the same” thing)
This last week I needed to deploy a greenfield system to Azure for the first time in a good while and so it seemed like a good point to reflect on the state of Azure deployment. tl;dr – it was like pulling teeth.
The simple is fairly typical – it uses a variety of Azure components to allow users to access a React (Fable) based website that talks to APIs (.NET 5) and a PostgreSQL database and a simple blob storage based key/document store to allow users to manage risk and capabilities in their organisations.
As its greenfield I’ve gone all in on managed identity and Key Vault wherever possible. I use Docker to run the API and within Web Apps for Containers and I use the Azure CDN backed by a storage account to serve the React app.
Build, release and deployment occurs via a GitHub Action with the real work taking place inside Farmer and so ultimately the deployment is based on ARM.
The system is written end to end in F# and the architecture is shown below.
Just a couple of notes on this architecture based on common questions.
Why not use AKS? For something this simple? Not needed – massive overkill.
Why use Docker then? I value being able to move things between vendors. For example I’ve experimented with this deployment in AWS, Azure and Digital Ocean. Using fairly standardised packaging mechanisms means I’m generally just worrying about configuration.
Why not use use Azure Static Web Apps. Somehow this is still in preview and I already knew how to do the same thing with the CDN and had to do a “manual” build anyway as I am using Fable (which isn’t hard – you can find some notes about that here).
I thought you loved Service Bus where’s the Service Bus? v1.1 🙂
The Good
Farmer is excellent – I’m a big fan of using actual code for infrastructure and see no reason at all to learn another (stunted) language like Bicep to do so. Pages of ARM become just a handful of lines of Farmer code and as it ultimately outputs and (optionally) executes ARM templates you’re not locked into anything. My final build and release script is entirely within F# – its not a Frankensteins monster of JSON, Bash, Powershell etc. though I do call out to az cli on occasion to work round problems in ARM.
At this point I’ve used ARM itself, Farmer, Pulumi, Terraform and played with Bicep and my favourite of these is definitely Farmer. It saves time and reduces cognitive load.
Managed Identity now feels pretty usable within Azure. Last time I tried this support was so patchy it just wasn’t worth the effort. This time round although not everywhere it is in many places, is fairly well documented, and supports local development easily – again last time I experimented with it local development (at least on a Mac at the time) was somewhat painful.
The Azure Portal is useful when you’re dealing with multiple services. Its got its issues for sure but it does bring things together in a helpful way.
Log Streaming for Web Apps is great – you can flick to that tab and quickly see application related issues around startup within your containers.
The Bad
Error reporting from the Azure Resource Manager itself is still dreadful. A times I was faced by utterly opaque errors and at others completely misleading errors. I’m 95% sure that if I cracked the lid on the underlying code I would find things like “try …. catch all… log generic error”.
Additionally what are basically validation errors “you can’t do this with that” are left to the various resources to handle. The problem with this is that this won’t occur until a long way into development and means the feedback loop for development is torturously slow.
You can burn days on this stuff. And I understand the need to decouple resource deployment from the orchestration of it but its not hard to see how validation couldn’t also be decoupled and done earlier stage for many errors.
Read the small print! Things in Azure are in a constant state of moving forward – and this is great for the most part – but when those systems are foundational things you find they are partially supported and that their are caveats. And you can still get surprised by things if you are a new Azure user: for example Functions not supporting .NET 5.
Things are declarative until they are not! The defence of ARM, and now Bicep, is that its declarative. The problem is it really isn’t – it requires orchestrating and some of this is in areas that really need smoothing out.
A great example of this is deployment from ACR to Web Apps for Containers. The Web App for Container won’t deploy with CI/CD turned on until ACR is both created and has an image inside it. This immediately means I have to split my deployment into two ARM templates and orchestrate between the two. This is such a common use case it really ought to be smoothed out – let the Web App be created but stay empty until a ACR is created and a container pushed. If I need to restart the web app ok but don’t prevent me creating it.
Bugs. Even in this simple deployment I’ve come across a few of these. Just a couple of examples below but the real issue is that when you combine this with the above issues you end up with very confusing situations.
Granting my web app managed identity access to blob storage fails on multiple runs on the ARM template. It will work when the account is created but fail on subsequent deployments. Workaround: do it using Az CLI.
For reasons unknown KeyVault references are not working on the Web App. Workaround: don’t use them, use the same managed identity to load them into the ASP.Net Core app configuration with AddKeyVault(). I’m hoping this is something that can be resolved but I’ve tried all sorts of things with no luck yet.
Closing Thoughts
Deploying to Azure is still a painful and unpredictable task for developers. It shouldn’t be. As a point of comparison this took me two days to complete for dev and live environments (the latter took about an hour) – I deployed a very similar system into AWS a couple of months back knowing absolutely nothing about AWS and it also took two days. Given I’ve been working with Azure for 10+ years that’s really disappointing.
I think one of the reasons its such a muddle is that Azure has been built “top down” rather than bottom up. The underlying compute, network and identity platform have been patched in under the original rather humble set of PaaS services. AWS on the other hand feels like it was built bottom up and so while it can feel lower level than Azure it also feels more consistent and predictable.
Why don’t these time burning and infuriating issues get attention? I think their are two things at play.
Firstly – Conway’s Law. Microsoft is huge and you can see the walls between the teams and divisions bleeding into common areas like this and their doesn’t seem to be a guiding set of minimum standards for Azure teams. And if they are they are clearly high up on the list of things to be sacrificed when things are running late.
Secondly – these things aren’t sexy. They can’t be launched at a huge PR event. They don’t result in a 5 minute demo. They don’t sell to CTOs of large big spending businesses who are a big distance from this stuff and the cost of these issues is often hidden – its buried in the minutiae.
Thirdly – Microsoft is at war with Eurasia and its always been at war with Eurasia. Its hard to announce you’re working on fixing something that is really sub-standard without admitting you have something really sub-standard. And so due to the marketing things are awesome until they can be replaced by a new feature that can be celebrated. Take ARM as an example – celebrated and championed, despite complaints of real world users, right up until Bicep when it became IL (though ironically it seems all the same hacks are still needed).
As ever I’m sure the Product teams are well meaning but like the rest of us they are wrapped in management, commercial concerns, and organisational systems that result in sub-optimal outcomes for users. Sadly we, as users of Azure, pay for this every day and the deployment area of Azure continues to feel at the sharp and expensive end of this.
If you follow me on Twitter you might have seen that as a side project I run a cycling performance analytics website called Performance For Cyclists – this is currently running on Azure.
Its built in F# and deploys to Azure largely like this (its moved on a little since I drew this but not massively):
It runs fine but if you’ve been following me recently you’ll know I’ve been looking at AWS and am becoming somewhat concerned that Microsoft are falling behind in a couple of key areas:
Support for .NET – AWS seem to always be a step ahead in terms of .NET running in serverless environments with official support for the latest runtimes rolling out quickly and the ability to deploy custom runtimes if you need. Cold starts are much better and they have the capability to run an ASP.Net Core application serverlessly with much less fuss.
I can also, already, run .NET on ARM on AWS which leads me to my second point (its almost as if I planned this)…
Lower compute costs – my recent tests demonstrated that I could achieve a 20% to 40% saving depending on the workload by making use of ARM on AWS. It seems clear that AWS are going to broaden out ARM yet further and I can imagine them using that to put some distance between Azure and AWS pricing.
I’ve poked around this as best I can with the channels available to me but can’t get any engagement so my current assumption is Microsoft aren’t listening (to me or more broadly), know but have no response, or know but aren’t yet ready to reveal a response.
(just want to be clear about something – I don’t have an intrinsic interest in ARM, its the outcomes and any coupled economic opportunities that I am interested in)
I’m also just plain and simpe curious. I’ve dabbled with AWS, mostly when clients were using it when I freelanced, but never really gone at it with anything of significant size.
I’m going to have to figure my way through things a bit, and doubtless iterate, but at the moment I’m figuring its going to end up looking something like this:
Leaving Azure Maps their isn’t a mistake – I’m not sure what service on AWS offers the functionality I need, happy to here suggestions on Twitter!
I may go through this process and decide I’m going to stick with Azure but worst case is that I learn something! Either way I’ll blog about what I learn. I’ve already got the API up and running in ECS backed by Fargate and being built and deployed through GitHub Actions and so I’ll write about that in my next post.
As I normally post from a developer perspective I thought it might be worth starting off with some additional context for this post. If you follow me on Twitter you might know that about 14 months ago I moved into a CTO role at a rapidly growing business – we’re making ever increasing use of the cloud both by migrating workloads and the introduction of new workloads. Operational cost is a significant factor in my budget. To me the cloud can be summarised as “cloud = economics + capabilities” and so if I have a similar set of capabilities (or at least capabilities that map to my needs) then reduction in compute costs has the potential to drive the choice of vendor and unlock budget I can use to grow faster.
In the last few posts I’ve been exploring the performance of ARM processors in the cloud but ultimately what matters to me is not a processor architecture but the economics it brings – how much am I paying for a given level of performance and set of characteristics.
It struck me there were some interesting differences across ARM, x86, Azure and AWS and I’ve expanded my testing and attempted here to present these findings in (hopefully) useful terms.
All tests have been run on CentOS Linux (or the AWS derivative) using the .NET 5 runtime with Apache acting as a reverse proxy to Kestrel. I’ve followed the same setup process on every VM and then run performance tests directly against their public IP using loader.io all within the USA.
I’ve run two workloads:
Generate a Mandelbrot – this is computationally heavy with no asynchronous yield points.
A test that simulates handing off asynchronously to remote resources. I’ve included a small degree of randomness in this.
At the bottom of the post is a table containing the full set of tests I’ve run on the many different VM types available. I’m going to focus on some of the more interesting scenarios here.
Computational Workload
2 Core Tests
For these tests I picked what on AWS is a mid range machine and on Azure the entry level D series machine:
AWS (ARM): t4g.large – a 2 core VM with 8GiB of RAM and costing $0.06720 per hour AWS (x86): t3.large – a 2 core VM with 8GiB of RAM and costing $0.08320 per hour Azure (x86): D2s v4 – a 2 core VM with 8GiB of RAM and costing $0.11100 per hour
On these machines I then ran the workloads with different numbers of clients per seconds and measured their response times and the failure rate (failure being categorised as a response of > 10 seconds):
Both Intel VMs generated too many errors at the 25 client per second rate and the load tester aborted.
Its clear from these results that the ARM VM running on AWS has a significant bang for buck advantage – its more performant than the Intel machines and is 20% cheaper than the AWS Intel machine and 40% cheaper than the Azure machine.
Interestingly the Intel machine on AWS lags behind the Intel machine on Azure particularly when stressed. It is however around 20% cheaper and it feels as if performance between the Intel machines is largely on the same economic path (the AWS machine is slightly ahead if you normalise the numbers).
4 Core Tests
I wanted to understand what a greater number of cores would do for performance – in theory it should let me scale past the 20 client per second level of the smaller instances. Having concluded that ARM represented the best value for money for this workload on AWS I didn’t do an x86 test on AWS. I used:
AWS: t4g.xlarge (ARM) – a 4 core VM with 16GiB of RAM and costing $0.13440 per hour Azure: D4s_v4 – a 4 core VM with 16GiB of RAM and costing $0.22200 per hour
I then ran the workloads with different numbers of clients per seconds and measured their response times and the failure rate (failure being categorised as a response of > 10 seconds):
The Azure instance failed the 55 client per second rate – it had so many responses above 10 seconds in duration that the load test tool aborted the test.
Its clear from these graphs that the ARM VM running on AWS outperforms Azure both in terms of response time and massively in terms of bang for buck – its nearly half the price of the corresponding Azure VM.
Starter Workloads
One of the nice things about AWS and Azure is they offer very cheap VMs. The Azure VMs are burstable (and there is some complexity here with banked credits) which makes them hard to measure but as we saw in a previous post the ARM machines perform very well at this level.
The three machines used are:
AWS (ARM): t4g.micro, 2 core, 1GiB of RAM costing $0.00840 per hour Azure (x86): B1S, 1 core, 1GiB of RAM costing $0.00690 per hour AWS (x86): t3.micro, 2 core, 1 GiB of RAM costing $0.00840 per hour
Its an easy victory for ARM on AWS here – its performant, cheap and predictable. The B1S instance on Azure couldn’t handle 15 or 20 clients per second at all but may be worth consideration if its bursting system works for you.
Simulated Async Workload
2 Core Tests
For these tests I used the same configurations as in the computational workload.
Their is less to separate the processors and vendors with a less computationally intensive workload. Interestingly the AWS machines have a less stable response time with more > 10 second response times but, in the case of the ARM chip, it does this while holding a lower average response time while under load.
Its worth noting that the ARM VM is doing this at 40% of the cost of the Azure VM and so I would argue again represents the best bang for buck. The AWS x86 VM is 20% cheaper than the Azure equivelant – if you can live with the extra “chop” that may still be worth it or you can use that saving to purchase a bigger tier unit.
4 Core Tests
For these tests I used the same virtual machines as for the computational workload:
There is little to separate the two VMs until they come under heavy load at which point we see mixed results – I would argue the ARM VM suffers more as it becomes much more spiky with no consistent benefit in average response time.
However in terms of bang for buck – this ARM VM is nearly half the price of the Azure VM. There’s no contest. I could put two of these behind a load balancer for nearly the same cost.
Starter Workloads
For these tests I used the same virtual machines as for the computational workload:
Its a pretty even game here until we hit the 100 client per second range at which point the AWS VMs begin to outperform the Azure VM though at the 200 client per second range at the expense of more long response times.
Conclusions
Given the results, at least with these workloads, its hard not to conclude that AWS currently offers significantly greater bang for buck than Azure for compute. Particularly with their use of ARM processors AWS seem to have taken a big leap ahead in terms of value for money for which, at the moment, Azure doesn’t look to have any response.
Perhaps tailoring Azure VMs to your specific workloads may get you more mileage.
I’ve tried to measure raw compute here in the simplest way I can – I’d stress that if you use more managed services you may see a different story (though ultimately its all running on the same infrastructure so my suspicion is not). And as always, particularly if you’re considering a switch of vendor, I’d recommend running and measuring representative workloads.
Having conducted my ARM and x64 tests on AWS yesterday I was curious to see how Azure would fair – it doesn’t support ARM but ultimately that’s a mechanism for delivering value (performance and price) and not an end in and of itself. And so this evening I set about replicating the tests on Azure.
In the end I’ve massively limited my scope to two instance sizes:
A2 – this has 2 CPUs and 4Gb of RAM (much more RAM than yesterdays) and costs $0.120 per hour
B1S – a burstable VM that has 1 CPUand 1Gb RAM (so most similar to yesterdays t2.micro) and costs $0.0124 per hour
Note – I’ve begun to conduct tests on D series too, preliminary findings is that the D1 is similar to the A2 in performance characteristics.
I was struggling to find Azure VMs with the same pricing as AWS and so had to start with a burstable VM to get something in the same kind of ballpark. Not ideal but they are the chips you are dealt on Azure! I started with the B1S which was still more expensive than the ARM VM. I created the VM, installed software, and ran the tests – the machine comes with 30 credits for bursting. However after running tests several times it was still performing consistently so these were either exhausted quickly, made little difference, or were used consistently.
I moved to the A2_V2 because, frankly, the performance was dreadful on my early tests with the B1S and I also wanted something that wouldn’t burst. I was also trying to match the spec of the AWS machines – 2 cores and 1Gb of RAM. I’ll attempt the same tests with a D series when I can.
Test setup was the same and all tests are run on VMs accessed directly on their public IP using Apache as a reverse proxy to Kestrel and our .NET application.
I’ve left the t2.micro instance out of this analysis
Mandelbrot
With 2 clients per test we see the following response times:
We can see that the two Azure instances are already off to a bad start on this computationally heavy test.
At 10 clients per second we continue to see this reflected:
However at this point the two Azure instances begin to experience timeout failures (the threshold being set at 10 seconds in the load tester):
The A2_V2 instance is faring particularly badly particularly given it is 10x the cost of the AWS instances.
Unfortunately their is no meaningful compaison I can make under higher load as both Azure instances collapse when I push to 15 clients per second. For complete sake here are the results on AWS at 20 clients per second (average response and total requests):
Simulated Async Workload
With our simulated async workload Azure fares better at low scale. Here are the results at 20 requests per second:
As we push the scale up things get interesting with different patterns across the two vendors. Here are the average response times at 200 clients per second:
At first glance AWS looks to be running away with things however both the t4g.micro and t3.micro suffer from performance degradation at the extremes – the max response time is 17 seconds for both while for the Azure instances it is around 9 seconds.
You can see this reflected in the success and total counts where the AWS instances see a number of timeout failures (> 10 seconds) while the Azure instances stay more consistent:
However the AWS instances have completed many more requests overall. I’ve not done a percentile breakdown (see comments yesterday) but it seems likely that at the edges AWS is fraying and degrading more severely than Azure leading to this pattern.
Conclusions
The different VMs clearly have different strengths and weaknesses however in the computational test the Azure results are disappointing – the VMs are more expensive yet, at best, offer performance with different characteristics (more consistent when pushed but lower average performance – pick your poison) and at worst offer much lower performance and far less value for money. They seem to struggle with computational load and nosedive rapdily when pushed in that scenario.
Farmer is a DSL for generating and executing ARM templates and one of the great things about it is that its based on .NET Core. That means that you can use it in combination with other components from the .NET ecosystem to create end to end typesafe deployment solutions.
As an aside – I recently posted a critique of Microsofts new DSL Bicep. One of the things I didn’t mention in that but did in a series of tweets was the shortcomings of inventing a new language that lives in its own ecosystem.
Ultimately Bicep will need to support “extension points” or you’ll have to wrap them in script and communicate information across boundaries (of course their can be benefits to that approach too). Not to mention they need to write all the tooling from scratch and developers / administrators need to learn another language.
By taking the approach Farmer has handling boundaries is a lot cleaner – as we’ll see – and we can take advantage of some neat language features.
In this example I’m going to provision an Azure SQL Database into Azure and then upgrade its schema using DbUp and we’ll run all this through GitHub Actions giving us an automated end to end deployment / upgrade system for our SQL database. You could do this with less F# code (almost none) but I also want to try and illustrate how this approach can form a nice framework for more complicated deployment scenarios so we’re also going to look at error handling across a deployment pipeline.
Starting with a F# console app, adding the Farmer NuGet package, and the boilerplate Program.fs file first we need to declare our Azure resources – in this case a SQL Server and a database and then bring them together in an ARM template:
let demoDatabase = sqlServer {
name serverName
admin_username "demoAdmin"
enable_azure_firewall
add_databases [
sqlDb { name databaseName ; sku DbSku.Basic }
]
}
let template = arm {
location Location.UKWest
add_resource demoDatabase
output "connection-string" (demoDatabase.ConnectionString databaseName)
}
Pretty straightforward but a couple of things worth noting:
Both serverName and databaseName are simple constants (e.g. let databaseName = “myDatabaseName”) that I’ve created as I’m going to use them a couple of times.
Opening up the database to azure services (enable_azure_firewall) will allow the GitHub Actions Runner to access the database.
On the final line of our arm block we output the connection string for the database so we can use it later.
That’s our Azure resources but how do we apply our SQL scripts to generate our schema? First we’ll need to add the dbup-sqlserver NuGet package and with that in place we’ll first add a Scripts folder to our solution and in my example four scripts:
DbUp keeps track of the last script it ran and applies subsequent scripts – essentially its a forward only ladder of migrations. If you’re adding scripts of your own make sure you mark them as Embedded Resource otherwise DbUp won’t find them. To apply the scripts we simply need some fairly standard DbUp code like that shown below, I’ve placed this in a F# module called DbUpgrade so, as we’ll see in a minute, we can pipe to it quite elegantly:
let tryExecute =
Result.bind (fun (outputs:Map<string,string>) ->
try
let connectionString = outputs.["connection-string"]
let result =
DeployChanges
.To
.SqlDatabase(connectionString)
.WithScriptsEmbeddedInAssembly(Assembly.GetExecutingAssembly())
.LogToConsole()
.Build()
.PerformUpgrade()
match result.Successful with
| true -> Ok outputs
| false -> Error (sprintf "%s: %s" (result.Error.GetType().Name.ToUpper()) result.Error.Message)
with _ -> Error "Unexpected error occurred upgrading database"
)
If you’re not familiar with F# you might wonder what this Result.bind function is. F# has a wrapper type for handling success and error states called options and a bunch of helper functions for their use. One of the neat things about it is it lets you chain lots of functions together with an elegant pattern for handling failure – this is often referred to as Railway Oriented Programming.
We’ve now declared our Azure resources and we’ve got a process for deploying our upgrade scripts and we need to bring it all together and actually execute all this. First lets create our deployment pipeline that first provisions the resources and then upgrades the database:
let deploymentPipeline =
Deploy.tryExecute "demoResourceGroup" [ adminPasswordParameter ]
>> DbUpgrade.tryExecute
If we had additional tasks to run in our pipeline we’d join them together with the >> operator as I’ve done here.
To run the deployment we need to provide an admin passford for SQL server which you can see in this code snippet as sqlServerPasswordParameter and we need to do this securely – so it can’t sit in the source code. Instead as I’m going to be running this from GitHub Actions an obvious place is the Secrets area of GitHub and an easy way to make that available to our deployment console app is through an environment variable in the appropriate action (which we’ll look at later). We can then access this and format it for use with Farmber by adding this line:
let adminPasswordParameter =
Environment.GetEnvironmentVariable("ADMIN_PASSWORD") |> createSqlServerPasswordParameter serverName
Farmer uses a convention approach to a parameter name – I’ve built a little helper function createSqlServerPassword to form that up.
(We could take a number of different approaches to this – ARM parameters for example – I’ve just picked a simple mechanism for this demo)
Finally to invoke all this we add this line at the bottom of our file:
template |> deploymentPipeline |> asGitHubAction
asGitHubAction is another little helper I’ve created that simply returns a 0 on success or prints a message to the console and returns a 1 in the event of an error. This will cause the GitHub Action to fail as we want.
That’s the code side of things done. Our finished Program.cs looks like this:
open System
open Farmer
open Farmer.Builders
open Sql
open Constants
open Helpers
[<EntryPoint>]
let main _ =
let adminPasswordParameter =
Environment.GetEnvironmentVariable("ADMIN_PASSWORD") |> createSqlServerPasswordParameter serverName
let demoDatabase = sqlServer {
name serverName
admin_username "demoAdmin"
enable_azure_firewall
add_databases [
sqlDb { name databaseName ; sku DbSku.Basic }
]
}
let template = arm {
location Location.UKWest
add_resource demoDatabase
output "connection-string" (demoDatabase.ConnectionString databaseName)
}
let deploymentPipeline =
Deploy.tryExecute "demoResourceGroup" [ adminPasswordParameter ]
>> DbUpgrade.tryExecute
template |> deploymentPipeline |> asGitHubAction
All we need to do now is wrap it up in a GitHub Action. I’ve based this action on the stock .NET Core build one – lets take a look at it:
If you’re familiar with GitHub Actions most of this should be fairly self explanatory – there’s nothing special about our deployment code, its a standard .NET Core console app so we begin by building it as we would any other (again this is one of the things I like about Farmer – its just .NET, and if you’re using .NET there’s nothing else required). However after building it we do a couple of things:
To do the deployment Farmer will use the Azure CLI and so we need to login to Azure via that. We do that in the Login via Az module step which is pretty stock and documented on GitHub here. I’ve stored the secret for the service principal in the secrets area of GitHub.
In the final step we run our deployment – again its just a standard console app. You an see in this step the use of the env section – we take a secret we’ve called ADMIN_PASSWORD and set it as an environment variable making it available to our console app.
And that’s it! At this point you’ve got an automated solution that will make sure your Azure SQL database infrastructure and its schema are managed get up to date. Change the configuration of your SQL database and/or add a SQL script and this will kick off and apply the changes for you. If / when you run it for the first time you should see output like this from the build section of the Action:
I think its a simple, but powerful, example of infrastructure as code and the benefits of using an existing language and ecosystem for creating DSLs – you get so much for free by doing so. And if the rest of your codebase is in .NET then with Farmer you can share code, whether that be simple constants and names or implementation, easily across your deployment and runtime environments. Thats a big win. I’m slowly adding it into my Performance for Cyclists project and this approach here is largely lifted from their.
Finally I think its worth emphasising – you don’t need to really know F# to use Farmer and you certainly don’t need to be using it elsewhere in your solution. Its a pretty simple DSL build on top of F# and a fantastic example of how good F# is as a basis for DSLs. I’ve dug a little deeper into the language here to integrate another .NET tool but if all you want to do is generate ARM templates then, as you can see from the Farmer examples on its website, you really don’t need to get into the F# side (though I do encourage you to!).
I’m not sure when it went public but Microsoft now have an alpha of Bicep available on GitHub. Bicep is their attempt to deal with the developer unfriendly horror and rats nest that is ARM templates.
I took a look at it today and came away thoroughly disappointed with what I saw. Before I even started to look at the DSL itself the goals raised a whole bunch of red flags… so lets start there.
And strap in… because this is going to be brutal.
Bicep Goals
Build the best possible language for describing, validating, and deploying infrastructure to Azure. Laudable.
The language should provide a transparent abstraction for the underlying platform. There must be no “onboarding step” to enable it to a new resource type and/or apiVersion in Bicep. Seems important. Azure moves fast and new things are added all the time. On the one hand it would be nice to see each push of a new resource / API to Azure be coupled to some nice Bicep wrapping – but given the breadth and pace of what goes on its probably not realistic.
On the other hand…. this seems dangerous. There’s a lot of intricacy in that stuff and if we just boil down to expressing the ARM inside strings with a bit of sugar round it what have we gained?
Code should be easy to understand at a glance and straightforward to learn, regardless of your experience with other programming languages. Ok. No great quibble here.
Users should be given a lot of freedom to modularize and reuse their code. Reusing code should not require any ‘copy/paste’. Seems weak. I’d like to see this strengthened such that writing new Bicep code shouldn’t require copy/paste – see my point (2) above as these things seem somewhat coupled. Cough. Magic strings. Cough.
Tooling should provide a high level of resource discoverability and validation, and should be developed alongside the compiler rather than added at the end. On the one hand I don’t disagree. On the other… this seems like a bit of a cop out back to magic strings.
Users should have a high level of confidence that their code is ‘syntactically valid’ before deploying. Wait. What? A “high level of confidence”. I don’t want a high level of confidence. I want to know. Ok – if I’m on the bleeding edge and the Azure resource hasn’t been packed into some nice DSL support yet then ok. But if I’m deploying, say, a vanilla App Service I don’t want a high level of confidence – I want to know.
I don’t about you but to me this sounds like it has the hallmarks of another half baked solution (“no its awesome” – random Twitter devrel) that still requires you to remember a bunch of low level details. And probably still relies on strings.
The Tutorial
Next I cautiously cracked open the tutorial… oh god.
Strings galore. Strings for well known identities. The joy. I’ve not installed the tooling but I assume it helps you pick the right string. But really? REALLY? Strings.
I was pretty horrified / disappointed so I continued through hoping this was just the start but as far as I can tell – nope. Strings are a good idea apparently. This is the final example for storage:
Same dogs dinner.
I can absolutely see why you want some form of string support in their. As I noted in the goals if a new API version is released you want to be able to specify it. But there are ways to achieve this that don’t involve this kind of untyped nonsense. In the normal course of events for common things like Storage accounts the only strings in use should be for names.
No wonder they can only give “confidence” things are syntactically correct.
The tutorial finishes with “convert any ARM template to Bicep” – its not hard is it. This is still a thin low value wrapper on top of ARM. We’ve just got rid of the JSON and replaced it with something else. If you don’t add much value then converting between two things is generally straight forward.
I’m struggling not to be unkind… but is their any kind of peer review for this stuff? Do people who understand languages or have a degree of breadth get involved? Do people actually *making* things using this stuff get involved? Because as someone involved in making lots of stuff and running teams making stuff – this misses by miles.
It really doesn’t have to be this way – the community are coming up with better solutions frankly. Bit of a “back to Build” but why the heck didn’t they put some weight behind Farmer – or at least lift some of its ideas (I’m glad they at least acknowledge it). Because here’s a storage account modelled in that:
Here’s a more complex Farmer block for storage and a web app:
Neater right? And typesafe. Its not hard to imagine something inbetween Farmer and Bicep that doesn’t rely on all these strings and bespoke tooling (Farmer is based on F#… so the tooling already exists) but still allows you to dive into things “not in the box”.
Conclusion
Super disappointing. Hopelessly basic. Doesn’t look to solve many problems. Another “requires lots of custom tooling” project. A tiny incremental move on from ARM. Doesn’t seem worth the effort. Better hope the Bicep tooling is good and frequently updated if you plan on using this.
If we’re now “treating ARM as the IL” (which is what it is – despite years of MS pushing back on feedback that ARM is awesome) then this really is a poor effort to build on that. Which is sad because as it comes from Microsoft its likely to become the most commonly used solution. Merit won’t have much to do with it.
If anyone from the Bicep team wants to talk about this – happy to.
There are an awful lot of services available on Azure but I’ve noticed a pattern emerging in a lot of my work around web apps. At their core they often have a similar architecture, deployment in Azure, and process for build and release.
For context a lot my hands on work over the last 3 years has been as a freelancer developing custom systems for people or on my own side projects (most recently https://www.forcyclistsbycyclists.com). In these situations I’ve found productivity to be super important in a few key ways:
There’s a lot to get done, one or two people, and not much time – so being able to crank out a lot of work quickly and to a good level of quality is key.
Adaptability – if its an externally focused green field system there’s a reasonable chance that there’s a degree of uncertainty over what the right feature set is. I generally expect to have to iterate a few times.
I can’t be wasting time repeating myself or undertaking lengthy manual tasks.
Due to this I generally avoid over complicating my early stage deployment with too much separation – but I *do* make sure I understand where my boundaries and apply principles that support the later distribution of a system in the code.
With that out the way… here’s an architecture I’ve used as a good starting point several times now. And while it continues to evolve and I will vary specific decisions based on need its served me well and so I thought I’d share it here.
I realise there are some elements on here that are not “the latest and greatest” however its rarely productive to be on the bleeding edge. It seems likely, for example, that I’ll adopt the Azure SPA support at some point – but there’s not much in it for me doing that now. Similarly I can imagine giving GitHub Actions ago at some point – but what do I really gain by throwing what I know away today. From the experiments I’ve run I gain no productivity. Judging this stuff is something of a fine line but at the risk of banging this drum too hard: far too many people adopt technology because they see it being pushed and talked about on Twitter or dev.to (etc.) by the vendor, by their DevRel folk and by their community (e.g. MVPs) and by those who have jumped early and are quite possibly (likely!) suffering from a bizarre mix of Stockholm Syndrome and sunk cost fallacy “honestly the grass is so much greener over here… I’m so happy I crawled through the barbed wire”.
Rant over. If you’ve got any questions, want to tell me I’m crazy or question my parentage: catch me over on Twitter.
Architecture
Build & Release
I’ve long been a fan of automating at least all the high value parts of build & release. If you’re able to get it up and running quickly it rapidly pays for itself over the lifetime of a project. And one of the benefits of not CV chasing the latest tech is that most of this stuff is movable from project to project. Once you’ve set up a pipeline for a given set of assets and components its pretty easy to use on your next project. Introduce lots of new components… yeah you’ll have lots of figuring out to do. Consistency is underrated in our industry.
So what do I use and why?
Git repository – I was actually an early adopter of Git. Mostly because I was taking my personal laptop into a disconnected environment on a regular basis when it first started to emege and I’m a frequent committer.
In this architecture it holds all the assets required to build & deploy my system other than secrets.
Azure DevOps – I use the pipelines to co-ordinate build & release activities both directly using built in tasks, third party tasks and scripts. Why? At the time I started it was free and “good enough”. I’ve slowly moved over to the YAML pipelines. Slowly.
My builds will output four main assets: an ARM template, Docker container, a built single page application, and SQL migration scripts. These get deployed into a an Azure resource group, Azure container registry, blob storage, and a SQL database respectively.
My migration scripts are applied against a SQL database using DbUp and my ARM templates are generated using Farmer and then used to provision a resource group. I’m fairly new to Farmer but so far its been fantastic – previously I was using Terraform but Farmer just fit a little nicer with my workflow and I like to support the F# community.
Runtime Environment
So what do I actually use to run and host my code?
App Service – I’ve nearly always got an API to host and though I will sometimes use Azure Functions for this I more often use the Web App for Containers support.
Originally I deployed directly into a “plain” App Service but grew really tired with the ongoing “now this is really how you deploy without locked files” fiasco and the final straw was the bungled .NET Core release.
Its just easier and more reliable to deploy a container.
Azure DNS – what it says on the tin! Unless there is a good reason to run it elsewhere I prefer to keep things together, keeps things simple.
Azure CDN – gets you a free SSL cert for your single page app, is fairly inexpensive, and helps with load times.
SQL Database – still, I think, the most flexible general purpose and productive data solution. Sure at scale others might be better. Sure sometimes less structured data is better suited to less structured data sources. But when you’re trying to get stuff done there’s a lot to be said for having an atomic, transactional data store. And if I had a tenner for every distributed / none transactional design I’ve seen that dealt only with the happy path I would be a very very wealthy man.
Oh and “schema-less”. In most cases the question is is the schema explicit or implicit. If its implicit… again a lot of what I’ve seen doesn’t account for much beyodn the happy path.
SQL might not be cool, and maybe I’m boring (but I’ll take boring and gets shit done), but it goes a long way in a simple to reason about manner.
Storage accounts – in many systems you come across small bits of data that are handy to dump into, say, a blob store (poor mans NoSQL right there!) or table store. I generally find myself using it at some point.
Service Bus – the unsung hero of Azure in my opinion. Its reliable. Does what it says on the tin and is easy to work with. Most applications have some background activity, chatter or async events to deal with and service bus is a great way of handling this. I sometimes pair this (and Azure Functions below) with SignalR.
Azure Functions – great for processing the Service Bus, running code on a schedule and generally providing glue for your system. Again I often find myself with at least a handful of these. I often also use Service Bus queues with Functions to provide a “poor mans admin console”. Basically allow me to kick off administrative events by dropping a message on a queue.
Application Insights – easy way of gathering together logs, metrics, telemetry etc. If something does go wrong or your system is doing something strange the query console is a good way of exploring what the root cause might be.
Code
I’m not going to spend too long talking about how I write the system itself (plenty of that on this blog already). In generally I try and keep things loosely coupled and normally start with a modular monolith – easy to reason about, well supported by tooling, minimal complexity but can grow into something more complex when and if that’s needed.
My current tools of choice is end to end F# with Fable and Saturn / Giraffe on top of ASP.Net Core and Fable Remoting on top of all that. I hopped onto functional programming as:
It seemed a better fit for building web applications and APIs.
I’d grown tired with all the C# ceremony.
Collectively we seem to have decided that traditional OO is not that useful – yet we’re working in languages built for that way of working. And I felt I was holding myself back / being held back.
But if you’re looking to be productive – use what you know.
Contact
If you're looking for help with C#, .NET, Azure, Architecture, or would simply value an independent opinion then please get in touch here or over on Twitter.
Recent Comments