vCloud Air Dissected – pt.3 Understanding DR2C Compute

vCloud Air Disaster Recovery (aka Disaster Recovery to Cloud, aka DR2C) is by far the easiest entry point and the most in demand offering from vCloud Air. However, it also seems to be the most misunderstood service from a compute scoping perspective. Let’s break it down!

I won’t be covering DR2C in its entirety as there are numerous solution briefs, videos, white boards etc, most of which can be found in the links I featured here. My colleague @davehill99 also has a great blog covering everything from DR2C basics to advanced architecture.

DR2C Basics

DR2C is a subscription service built around VPC compute (covered in part 1 of this series) so we know that the same characteristics apply. What’s important to note is that workloads replicated from our on-premises vSphere Replication consumes storage in VCA, but does not reserve any compute. So what does this mean for the consumer?

First off let’s cover the two scenarios which would require access to active compute in DR2C. Unlike a VPC Virtual Data Centre (VDC), DR2C is not intended to run active VM’s outside of Testing or Recovery, defined as follows:

Testing: Launch protected VM’s in an isolated VDC for a period of 7 days. VM data continues to be asynchronously replicated for the entire duration of the test.

Recovery (or planned migration): Launch protected VM’s, accessible in a predefined production topology for 30 days (and beyond if required). VM replication is ceased (on-premises > vCA) for the entire duration that recovery workloads are active. Note: reverse replication and failback is available for DR2C 2.0 customers (vR & vCenter 6.0 and above).

Screen Shot 2015-08-19 at 2.27.01 pm

Scoping DR2C Compute…

Scoping compute for DR2C is a little different from other cloud services as we really need to analyse our procedures for testing and recovery. Let me explain…

DR2C compute is procured in the same blocks as VPC (10GHz/20GB) and scales in exactly the same way. In order to keep the cost of the service low (for the consumer) it is recommended that we only procure as much compute as we need to guarantee. I’ll come back to this in a second.

Screen Shot 2015-08-20 at 7.38.46 am
Note: This figure is conceptual and not intended to show any relationship between compute & storage

The above example shows a VDC capable of supporting 10GHz CPU / 20GB memory / 4TB standard storage. Conceptually let’s say that we need 40Ghz CPU / 80GB memory to fully stand up our replicated VM’s for a test. Using this allocation only allows us to test (or recover) a subset of our total footprint within the core subscription.

This is where temporary add-ons come in. We can procure additional compute resources on a temporary basis (1 week for testing, 1 month for recovery) from My VMware to support any additional compute we will need for both scenarios. This gives us the flexibility to test individual application stacks within the confines of the core subscription, or temporarily extended to full capacity to test (or recover) everything at once.

Screen Shot 2015-08-19 at 2.36.02 pm
Temporary subscription compute add-ons

How much to guarantee?..

The quantity of compute we choose to guarantee is completely dependent on our risk profile and overall Recovery Time Objective (RTO). The more compute we guarantee, the more our monthly cost goes up, much like an insurance policy.

Procuring add-ons will add a small amount of time to our overall recovery, but in reality this is a small inconvenience considering the potential cost savings. A simple rule of thumb would be to protect critical workloads (Tier 1) with guaranteed compute to reduce downtime, but scale up with add-ons for everything else (Tier 2 & 3).

It’s also important to remember that we can utilise active capacity in VPC, VPC OnDemand and DPC to augment our DR strategy. The VMware solution brief on Advanced Architecture for DR2C really helps to understand how different vCA services hang together to form a complete solution.

…and there you have it. Pretty simple once we know the limitations. That said, most of the complexity in DR strategy lies in the network. I’ll save that for the next post.




vCloud Air Dissected – pt.2 Understanding DPC Compute

So we’ve already covered some of the basic elements of scoping vCA compute in part 1. Let’s delve into Dedicated Private Cloud (DPC) to show how the compute differs from the services built on Virtual Private Cloud (VPC, DR2C and OnDemand).

Again, if any of this seems unfamiliar please check out pt.1 Understanding VPC Compute or have a fish around Cloud Academy for some vCA goodness.

DPC compute basics

Unlike the multi-tenanted Virtual Private Cloud service, Dedicated Private Cloud (DPC) assigns physically dedicated host’s to each tenant (as the name might suggest). This has numerous benefits for compute, specifically;

  • Physical isolation for the purpose of security and licensing compliance
  • The ability to split resources across multiple VDCs
  • The ability to control oversubscription at a VDC/VM level

Note: There are other benefits for network, which I will cover in a another post.

From a consumer perspective the services are almost identical, however there are a few additional parameters which can be configured from both the vCA and vCD web portals within the DPC service.

Physical allocation

A great place to start is understanding what we have access to when working with DPC. The physical specification of the hosts is identical to VPC (2x 8 core, x86 Intel processors @2.6GHz and 256GB RAM in AU South at the time of writing). This gives us access to a core subscription of 35Ghz CPU and 240GB RAM (the remaining resources are reserved for the hypervisor).

From a compute standpoint our workloads have access to a single host’s resources, but in reality we actually have two host’s (1x Active / 1x HA) within our DPC environment. High Availability has been a critical component of vSphere for many years now and by no means is it excluded from vCA.

Screen Shot 2015-08-04 at 1.59.21 pm
DPC core subscription = 1x Active + 1x HA hosts

DPC scales in this manner until we reach 7 hosts (1x core & 6x add-ons) where a second HA host is added into our DPC environment. As a rule every 7 hosts in your allocation adds a HA host, capped at 28 which maxes out the underlying cluster of 32 hosts (28x Active + 4x HA).

Screen Shot 2015-08-04 at 2.00.08 pm
DPC core + 6x compute add-ons

So why is this important? Probably the most obvious reason (other than curiosity) is licensing compliance. For the more restrictive software vendors (*cough*Oracle) it is critical to know how many hosts are in our environment as a whole.

Correction: Since I wrote this post the model has changed to horizontally scale HA hosts from 16 hosts onwards (ie. one HA host until you reach 14x active hosts where a second is added).

DPC compute consumption

As DPC compute is dedicated to a single (master) tenant, all compute resources are 100% guaranteed. It is completely up to the account administrator(s) how resources are split over a single or multiple VDC’s within the organisation.

The ability to create multiple VDC’s is a powerful tool as it gives us the choice to run heavily oversubscribed (think test/dev) or with room to stretch (think mission critical) within the same subscription. It’s also useful as a subtenancy construct for service providers (to be covered in a later post).

Screen Shot 2015-08-04 at 6.04.33 pm
VDC resource config

Once we dive into the configuration for an individual VM we can set shares, reservations and limits as a mechanism to balance workloads according to priority when our VDC is in contention. By default the reservation is set to ‘0’ (as pictured below) for both CPU and memory (unlike VPC which automatically reserves 100% of the memory allocation for powered on VM’s).

Screen Shot 2015-08-04 at 6.11.42 pm
VM compute resource config

Sticking with the default configuration (NORMAL, 0, Unlimited) will allow us to run heavily oversubscribed within our VDC with all workloads having equal access to resources. We get far greater control over the individual performance of each VM once we start pulling levers (read: configuring shares, reservations and limits). We’re not going to cover the specifics of resource management in this post, but I recommend reading this blog from @FrankDenneman as a start.


… is not as straight forward as VPC where we can use memory as a measure for sizing the environment (assuming memory is the most common resource for contention). If you are familiar with scoping and building a vSphere environment from scratch then this shouldn’t be too much of a chore.

When you are moving VM’s from an existing vSphere environment to vCA there are a number of tools that can be used to predictively analyse virtual resource utilisation like vRealize Operations Manager or VMware Infrastructure Planner. If not concerned with peak utilisation we can also use a point-in-time capture tool like RVTools. The key is to understand CPU and memory utilisation across a known resource quantity. As long as we know what is currently being consumed, we know how to size the compute for DPC.

For example, if we take a typical vSphere environment with 3 hosts (100GHz CPU / 384GB RAM) with a 15% and 50% utilisation respectively then we can conclude we need 15GHz CPU and 192GB RAM which is within the thresholds of a core subscription (35GHz / 240GB). However if memory utilisation is high, let’s say 75% (288GB) then we need a minimum of a core subscription plus a single compute add-on (70GHz / 480GB).

Before anyone gets fired up, I realise that the above is a drastic oversimplification for anything but the simplest migrations. It’s only intended to show that we don’t need to go to ridiculous lengths to get an indicative scope.

Note: We can use the same methodology for VPC, however we need would need to right-size the workloads before being migrated as vRAM is 100% reserved for powered on VMs.

Simple right?


Author: @Kev_McCloud

vCloud Air Dissected – pt.1 Understanding VPC Compute


In this series of posts I will be dissecting areas of vCloud Air (vCA) that I get grilled on regularly. Before reading on, please note that I am writing these blogs from the vCA Consumer perspective. For the vSphere Admin, interacting at purely the consumption layer can seem unnatural. I’m hoping this series will reveal enough detail to demonstrate VMware’s objectives in designing a global public cloud platform with the vSphere Admin in mind.

I’m going to assume some basic reader understanding of the key differences between Virtual Private Cloud (VPC), VPC OnDemand and Dedicated Private Cloud. There is a wealth of 101 style vCloud Air info covered in Cloud Academy to get you started if you are not already armed with this knowledge. In addition I’ll point you to a great blog from @mreferre who has the perfect vehicular analogy for the basic Virtual Data Centre (VDC) construct. With that out of the way, lets get started.

Screen Shot 2015-07-27 at 3.04.40 pm

VPC compute basics…

One of the areas I get questioned on regularly is how CPU is consumed and scoped for vCloud Air. In this post we are going to focus on CPU consumption within a VPC VDC.

VPC compute is made up of securely shared memory and CPU, procured as blocks of 10GHz vCPU / 20GB vRAM. vRAM is relatively easy to understand as whatever has been allocated (read: configured) to your powered on VMs equates to what is consumed within your VDC. The upside to this characteristic is that vRAM within your VDC is 100% guaranteed which is key to vCA’s multi-tenancy performance. Note: The logical vRAM maximum for a single VM is limited to what is available on a single host, minus the hypervisor overhead (~240GB at the time of writing).

Screen Shot 2015-07-20 at 6.36.23 pm

Core VPC Allocation

vCPU is a little trickier to explain as consumption is not static like vRAM. vCA CPU is procured in aggregated GHz ie. pooled clock cycles across a number of physical hosts. This however does not mean that a single workload’s processing is distributed. Regardless of how many vCPU’s have been configured a VM can only execute on a single host. It’s important to remember that vCA’s underlying hypervisor is vSphere and therefore the same rules apply in vCloud Air.

Going back to my introduction briefly, those who have familiarity with vSphere will want details on guarantees, oversubscription etc. The key to working with multi-tenanted cloud is to accept that low level functions (like the ability to set reservations, limits and shares) are often abstracted from the consumer. In principal this is how vCA ensures equal resources to all tenants. Wouldn’t be fair to have some customers more equal than others, right?

Note: DPC does allow the consumer to configure compute shares and reservations at a VM level however I will cover this in part 2 of this series.

How VPC compute is consumed…

VPC’s current architecture utilises hosts with 2x 8 core Intel x86 processors rated at 2.60GHz (at the time of writing). Contrary to popular belief vCPU is not throttled within vCA, therefore you can run a theoretical max of ~2.6GHz per vCPU. Conceptually, within a 10GHz allocation you could run 3x single vCPU VMs at 100% utilisation before you would potentially run into resource contention ie. 3x 2.6 = 7.8GHz leaving 2.2GHz.
Screen Shot 2015-07-27 at 1.17.14 pm

If we were to launch the fourth VM the expected behaviour is for the scheduler to balance all VMs within the VDC resulting in throttling by the percentage of the overcommitment.

Screen Shot 2015-07-27 at 1.15.55 pm

Note: In the above example this could theoretically be ~100MHz penalty per VM however this would be a point in time snapshot of the potential impact to workloads. The scheduler executes thousands of transactions per second therefore any impact would not be uniform across all workloads. This scenario is pretty unrealistic, but helps me to articulate the limits in this explanation.

If we were using a larger compute allocation (50GHz CPU, 100GB RAM) we could potentially run a single VM with 16x vCPUs running at a maximum clock speed of ~2.4GHz on a single host. The remaining host compute is reserved for hypervisor overhead and this is not included as part of our overall consumption.

Screen Shot 2015-07-26 at 6.27.26 pm

Using this same example, you can assign up to 32 vCPUs to a single VM* (deployed from vCloud Director UI), however vCPUs would be capped at ~1.2MHz when executing simultaneously due to hyper-threading on the underlying host processors. It’s important to note that multithreading characteristics are also dependent on the OS and application and should be taken into consideration when sizing a vCA environment.

*Interestingly, when deploying a new VM from the vCloud Director Web Interface you have the ability to assign up to 64 vCPUs to a single VM, but you will be swiftly issued with a error message upon creation.

vCA CPU Reservation

Definition: A reservation is a guarantee of the specified amount of physical resources regardless of the total number of shares in the environment.

What’s not immediately obvious (and can’t be observed from the basic vCA dashboard) is that a powered on VM automatically reserves 260MHz per vCPU and will scale up its reservation in 260MHz blocks. Theoretically you could power on 38x single vCPU VMs all using less than 260MHz before admission control stopped you from powering on subsequent VMs within you VDC.

Screen Shot 2015-07-26 at 4.37.30 pm

Again, this scenario is unrealistic but would help to explain why your dash may be showing available CPU that you cannot access. It is also important to note that the dash is not realtime (updated every 5 mins) and therefore not a reliable source of data for troubleshooting. Monitoring is a broad topic on it’s own so I’ll have to cover it in a later post.

When it comes to scoping typically most VMware environments are RAM bound and CPU usually clocks in at anywhere between 5-30% utilisation during non-peak periods. With this in mind (and the info in this post) it is pretty simple to put together some basic math to calculate the minimum requirements for a VPC environment (but don’t forget to factor in overhead for peak periods). VPC compute add-ons are also instantaneous when procured from My VMware, so if your math is a little skewed don’t worry, we got you covered.

I hope that this post has been useful in understanding some of the nuances of vCA Compute. Feel free to message me if you want more info on VPC compute or any aspect of vCA in general. -K


Author: @Kev_McCLoud