While running the experiment I posted about on Sunday I annotated the message I sent on to the event hub with the Azure Functions view of the current time (basically I set a property to DateTime.UtcNow) and, out of curiosity, grouped my results by second long tumbling windows based on that date. This gave me results that were observably different than when doing the same with the enqueue date and time logged by the Event Hub (as an aside there is some interesting information about Event Hubs and clock skew here). My experiments didn’t need massively accurate time tracking as I was really just looking for trends over a long, relative to the clock skew, period of time however I looked at some of the underlying numbers and became suspicious that there was a significant degree of clock skew across my functions.
I reached out to the @AzureFunctions team on Twitter asking how they handled clock sync on the platform and one of the engineers, Chris Anderson, replied confirming what I suspected: there are no guarantees about clock sync on the Azure Functions platform and, further, you should expect the time skew to be large.
That means you can’t really obtain a consistent view of “now” from within an Azure Function. You could go and get it from an external source but that in and of itself is going to introduce other inaccuracies. Essentially you can’t handle dynamic time reliably inside a function with any precision and you’re limited to working with reference points obtained upstream and passed in.
Definitely something to be aware of when designing systems that make use of Azure Functions as it would be easy to use them in scenarios applying timestamps expecting some sense of temporal sequence.
This isn’t, of course, a new problem – dealing with precise and accurate time in a distributed system is always a challenge and requires careful consideration but it does underline the importance of understanding your cloud vendors various runtime environments.