Azure IaaS VMs: The quest to deploy many, faster!

Recently I found myself in the need to deploy a rather large number of IaaS VMs. They all should see each other using a VNet. Well, a very patient man could simply deploy one-at-a-time. But why be patient when you can hack?

I figured I’d try to create one cloud service per VM (and place the VM there). I may test many per service later. But in the meantime, I figured if we have many simultaneous requests to provision we’d likely face errors. And in the cloud, errors are OK – just code expecting them and you’ll be fine.

The strategy is straightforward: Launch many requests and give it some time in between requests. In PowerShell it looks like this:

for([int]$i=1;$i -le 10; $i++)
   Start-Job -ScriptBlock {param ($n) D:\Scripts\OneVMPerService.ps1 -vm $n} -ArgumentList $i
   Start-Sleep -Seconds 10

While(Get-Job -State "Running") {Start-Sleep 5}

My OneVMPerService.ps1 script does two things: First, it creates a cloud service (and if the return code is not ‘Success’, it waits for 5 seconds and then retries) and then it creates the VM. When creating the VM, many bad things can happen with simultaneous requests. The blob holding the image may be locked. Some exclusive work on the VNet may be underway. ChaosMonkey. So I do the same thing, if I’m not successful for whatever reason, I sleep and retry again. This retry is possible by just capturing the context of an Azure operation and checking for its status, like this:

$context = New-AzureService -AffinityGroup $ag -ServiceName $cs
$isSuccess = ($context.OperationStatus -eq 'Succeeded')

Well? Did it work? Yes! Look at these numbers:

# of VMs Mode Time (in seconds) Errors (no. * error code)
1 - 70 -
5 Sequential 360 -
5 Parallel with retries 116 -
10 Parallel with retries- 287 5*400, 7*409, 1*500

Not entirely surprising, and definitely encouraging. I’m considering adding retry policies and a redo log to the approach so it’s more manageable. But, as with everything cloud, scale, parallelize, and code expecting errors. You’ll get the most out of your money that way.

Thanks to Pat Filoteo and Chris Clayton for teaching me about a lot of the things that can possibly go wrong :-)

About these ads

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s