→ Applies to: VMware ESXi 6.5 and above

Compute Resources for Virtual Machines

• Configuring CPU Resources for Virtual Machines: Users configure virtual CPUs and sockets in a virtual machine in order to provide sufficient processing power to satisfy the performance requirements of the application running on it.
   
• The CPU scheduler in ESXi Hypervisor: In the ESXi hypervisor, virtual CPUs (vCPUs) are scheduled by the VMkernel CPU Scheduler, which is responsible for allocating physical CPU (pCPU) resources to vCPUs. The scheduler operates based on a time-sharing model, where each vCPU is assigned a slice of pCPU time. Scheduling decisions prioritize fairness and efficiency, ensuring that all VMs receive appropriate resources while minimizing contention. Key rules include prioritizing VMs with higher shares or reservations and de-prioritizing those with CPU limits.
   
• Scheduling CPUs of a Virtual Machine: A vCPU is scheduled only if all the required pCPUs (e.g. for a multi-threaded VM) are simultaneously available, making the scheduling of VMs with many vCPUs more challenging.

Best Practices for vCPU-to-pCPU Ratios

• Recommended Ratios: For most environments, VMware suggests maintaining a vCPU-to-pCPU ratio between 1:1 and 1:3. This range is generally safe and unlikely to result in significant performance degradation. Ratios between 1:3 and 1:5 may begin to cause noticeable contention during peak usage, while ratios above 1:5 are highly likely to impact workload performance, particularly for demanding or mission-critical applications.
  
  
   
• Impact of Hyperthreading: ESXi considers hyperthreaded cores as logical CPUs when calculating CPU scheduling. However, hyperthreaded cores only provide about 75% of the performance of a true physical core. Over-subscription on hyperthreaded systems without adjusting for this reduced capacity can lead to inflated vCPU-to-pCPU ratios and unexpected contention.
   
• Avoiding Over-Subscription: Assigning more vCPUs to a VM than necessary can exacerbate CPU Ready issues. Each additional vCPU increases contention for scheduling. Administrators should perform regular workload assessments to ensure VMs are right-sized based on their actual CPU requirements.

CPU Over-Subscription in the ESXi Hypervisor

• What is CPU Over-Subscription? CPU oversubscription hapens when the number of virtual CPUs (vCPUs) assigned across all virtual machines (VMs) significantly exceeds the total physical CPU cores (pCPUs) available on an ESXi host. While this allows greater VM density and better hardware utilization, it also increases reliance on the ESXi CPU scheduler to manage resource allocation efficiently. Overcommitment is safe within limits, but beyond recommended thresholds, it causes performance bottlenecks, particularly during peak workloads.
   
• Role of the Vmkernel CPU Scheduler: The Vmkernel CPU scheduler is responsible for deciding which vCPUs run on which pCPUs at any given time. When vCPUs contend for limited physical CPU resources, the scheduler must delay some vCPUs, resulting in increased wait times (measured as CPU Ready time - %RDY). This delay impacts application responsiveness and system performance, particularly when demand surges.
   
• Balancing Benefits and Risks: Overcommitment helps maximize hardware ROI by running more workloads per host. However, exceeding safe vCPU-to-pCPU ratios can lead to excessive CPU Ready times, degraded application performance, and user dissatisfaction. Proper planning and monitoring are critical to maintain this balance effectively.

Understanding CPU Ready Time (%RDY)

• What is CPU Ready Time? CPU Ready is a metric that measures the percentage of time a vCPU spends waiting for execution on a pCPU. Contrary to initial impressions, higher CPU Ready values indicate worsening performance. According to VMware, CPU Ready time occurs when the scheduler cannot immediately allocate pCPU resources to a vCPU that is ready to execute.
   
• What Causes High CPU Ready? High CPU Ready values are typically caused by oversubscription of CPU resources. Other contributors include setting strict CPU limits, implementing restrictive CPU affinity rules, and inadequate physical resources for workload demand. Fault-tolerant (FT) networks can also cause elevated CPU Ready if bandwidth is insufficient for synchronization between primary and secondary hosts.
   
• Normal vs. Problematic Values: VMware recommends keeping CPU Ready below 5% per vCPU for optimal performance. For example, in vSphere charts, a value of 2173 ms over a 20-second interval equates to 10.87% (2173/20,000), which is double the acceptable threshold. Applications that are CPU-intensive or latency-sensitive may require even tighter thresholds to ensure smooth operation.

• Using esxtop from the CLI: esxtop provides granular insights into CPU Ready times. Launch esxtop on an ESXi host, press c to view CPU statistics, and focus on the %RDY column. A value above 5% per vCPU is indicative of contention. For example, a VM with four vCPUs and an overall %RDY of 20% means each vCPU averages 5% CPU Ready, which is acceptable. Values beyond this require intervention.
   
• Using vSphere Web UI: In the vSphere client, performance charts can also be used to monitor CPU Ready. Navigate to a VM's advanced performance options, select "CPU Ready" as a metric, and review the reported values. Since charts display CPU Ready in milliseconds, divide by the sampling period (e.g., 20,000 ms for 20 seconds) to convert to a percentage.
   
• Host-Level Challenges: Monitoring CPU Ready at the host level aggregates values across all VMs, which can hide individual VM issues. For a precise diagnosis, examine CPU Ready on a per-VM basis. This provides actionable data to identify specific VMs contributing to overall contention.
  
   

Addressing High CPU Ready

• Adjusting VM Sizing: The first step to address high CPU Ready is to ensure that VMs are sized appropriately. Over-provisioning vCPUs not only wastes resources but also increases contention. Evaluate each VM’s actual CPU usage using historical performance data and reduce vCPU counts where feasible. Windows Servers running as (only) DC for example have requirements satisfied with only 2 vCPUs.
   
• Balance Workloads Across Hosts: Balance workloads across your Syneto appliances. Redistribute VMs to hosts with more available CPU capacity.
   
• Long-Term Solutions: Persistent CPU Ready issues may require adding more physical CPU resources. This can be achieved by upgrading hosts with additional CPUs or introducing new hosts into the cluster. Avoid setting CPU limits or affinity rules unless absolutely necessary, as these configurations often exacerbate CPU contention rather than resolve it.