VMware PowerCLI 101 - Part5 - Real time storage IOPS and latency

It is very important to monitor and analyze the performance of storage subsystem components as it direcly affects the application performance. In this article, I will briefly explain how to use PowerCLI to get real time storage IOPS and latency of the following: 

• Virtual disk
• Datastore
• Disk/ LUN 
• Storage adapter
• Storage path

Connect to vCenter server using:
Connect-VIServer <IP address of vCenter>

To understand the list of all available stats for a specific entity, you can use Get-StatType. For example, to list all real time stats for a virtual machine you can use:

Get-StatType -Entity <VM name> -Realtime | sort

Virtual disk

To get real-time IOPS and latency of all virtual disks of a VM named 'lustre01':
Get-Stat -Entity lustre01 -Realtime -MaxSamples 1 -Stat virtualDisk.numberReadAveraged.average,virtualDisk.numberWriteAveraged.average,virtualDisk.totalReadLatency.average,virtualDisk.totalWriteLatency.average | sort Instance,MetricId | select MetricId, Value, Unit, Instance

Datastore

To get real-time IOPS and latency of a datastore (with Uuid: 5bea72bb-5d72ed6a-1d85-246e96792988) from an ESXi host (IP: 192.168.105.10):

Get-Stat -Entity 192.168.105.10 -Stat datastore.numberReadAveraged.average,datastore.numberWriteAveraged.average,datastore.totalReadLatency.average,datastore.totalWriteLatency.average -Realtime -MaxSamples 1 -Instance 5bea72bb-5d72ed6a-1d85-246e96792988 | Select MetricId, Value, Unit, Instance | Sort-Object MetricId

Note: You can get Uuid of a datastore using (Get-Datastore vol01).ExtensionData.Info.Vmfs.Uuid

Refer my article "Real time VMware datastore performance monitoring using PowerShell" for monitoring the real time performance statistics of multiple shared VMFS datastores which are part of a multi-node VMware ESXi cluster.

Disk/ LUN

To get real-time IOPS and latency of a disk (eui.387de1af35b93f6ff0a9bef000000000): 

Get-Stat -Entity 192.168.105.10 -Disk -Realtime -Instance eui.387de1af35b93f6ff0a9bef000000000 -MaxSamples 1 -Stat disk.numberWriteAveraged.average,disk.numberReadAveraged.average,disk.totalWriteLatency.average,disk.totalReadLatency.average | Select MetricId, Value, Unit, Instance

Storage adapter

To get real-time IOPS and latency of a storage adapter: 

Get-Stat -Entity 192.168.105.10 -Realtime -MaxSamples 1 -Stat storageAdapter.totalReadLatency.average, storageAdapter.totalWriteLatency.average, storageAdapter.numberReadAveraged.average, storageAdapter.numberWriteAveraged.average -Instance vmhba64 | Select-Object MetricId, Value, Unit, Instance | Sort-Object MetricId

Storage Path

To get real-time IOPS and latency of a storage path:

Get-Stat -Entity 192.168.105.10 -Realtime -MaxSamples 1 -Stat storagePath.totalReadLatency.average, storagePath.totalWriteLatency.average, storagePath.numberReadAveraged.average, storagePath.numberWriteAveraged.average -Instance fc.300fb123ba76519c:b436362bae5b217-fc.300fb123ba76519c:b436362bae5b217-eui.387de1af35b93f6ff0a9beec00000001 | Select MetricId,Value,Unit,Instance | Sort-Object MetricId

Hope it was useful. Cheers!

Related posts

VMware PowerCLI 101 - Part4 - Snapshots

VMware PowerCLI 101 - Part3 - Basic VM operations

VMware PowerCLI 101 - Part2 - Working with vCenter server

VMware PowerCLI 101 - Part1 - Installing the module and working with stand-alone ESXi host

VMware PowerCLI 101 - Part4 - Snapshots

In this post, I will briefly explain how to make use of PowerCLI when working with virtual machine snapshots.

Take a snapshot of VM:
New-Snapshot -VM "New Virtual Machine" -Name snap1 -Description try1

Revert to a snapshot:
$snap = Get-Snapshot -VM "New Virtual Machine" -Name "snap1"
Set-VM -VM "New Virtual Machine" -Snapshot $snap -WhatIf
Set-VM -VM "New Virtual Machine" -Snapshot $snap 

Delete specific snapshot of a VM:
$snap = Get-Snapshot -VM "New Virtual Machine" -Name "snap1"
Remove-Snapshot -Snapshot $snap -WhatIf
Remove-Snapshot -Snapshot $snap 

Delete all snapshots of a VM:
Get-VM "New Virtual Machine" | Get-Snapshot | Remove-Snapshot -WhatIf
Get-VM "New Virtual Machine" | Get-Snapshot | Remove-Snapshot 

List all VMs with snapshots:
Get-VM | Get-Snapshot | Select-Object VM, Name, Description, SizeGB

List VMs with snapshots older than a week:
Get-VM | Get-Snapshot | Where {$PSItem.Created -lt (Get-Date).AddDays(-7)} | Select-Object VM, Name, Description, Created, SizeGB | Format-Table

Find the parent-child relationship of VM snapshots:
$vm = Get-VM "New Virtual Machine"
get-vm $vm | Get-Snapshot | Select VM,Name,Description,Parent,Children,SizeGB,IsCurrent,Created,Id | sort Created |  Format-Table

Hope it was useful. Cheers!

Related posts:

VMware PowerCLI 101 - Part3 - Basic VM operations

VMware PowerCLI 101 - Part2 - Working with vCenter server

VMware PowerCLI 101 - Part1 - Installing the module and working with stand-alone ESXi host

VMware PowerCLI 101 - Part3 - Basic VM operations

Previous posts of this blog series talked about how to install PowerCLI, connecting to ESXi host and basics of working with the vCenter server. In this article, we will go through basic VM operations.

Get basic VM details:
Get-VM -Name <name of the virtual machine>

To view all properties of a VM object:
Get-VM -Name <name of the virtual machine> | Get-Member

To start a VM:
Get-VM -Name <name of the virtual machine> | Start-VM

To restart a VM:
Get-VM -Name <name of the virtual machine> | Restart-VMGuest

To shut down a VM:
Get-VM -Name <name of the virtual machine> | Shutdown-VMGuest

To delete a VM permanently:
Get-VM -Name <name of the virtual machine> | Remove-VM -DeletePermanently

Let's have a look into the "ExtensionData" property of a virtual machine.

Status related details of a VM:
(Get-VM -Name <name of the virtual machine>).ExtensionData | select GuestHeartbeatStatus,OverallStatus,ConfigStatus

CPU, Memory config details of a VM:
(Get-VM -Name <name of the virtual machine>).ExtensionData.Config.Hardware

CPU, Memory hot add/ remove features:

Layout details of a VM:
(Get-VM -Name <name of the virtual machine>).ExtensionData.LayoutEx.File | sort Key | ft

VM tools status and other guest details:
(Get-VM -Name <name of the virtual machine>).ExtensionData.Guest

Network related info of a VM:

(Get-VM -Name <name of the virtual machine>).ExtensionData.Guest.Net

Hope this was useful. Cheers!

VMware PowerCLI 101 - Part2 - Working with vCenter server

In my previous blog, we discussed how to install PowerCLI module and connect to a stand-alone ESXi host. Now let's start with connecting to the vCenter server.

Connect-VIServer <IP of vCenter server>

Get the list of data centers present under this vCenter server: Get-Datacenter

Get the list of clusters under a datacenter: Get-Datacenter DC01 | Get-Cluster

To verify the status of HA, DRS, and vSAN of a cluster:

(Get-Cluster Cluster01) | select Name,DrsEnabled,DrsAutomationLevel,HAEnabled,HAFailoverLevel,VsanEnabled

Get the list of ESXi hosts part of a specific cluster: 

Get-Datacenter DC01 | Get-Cluster Cluster01 | Get-VMHost

To get the list of VMs hosted on a specific ESXi host:

Get-Datacenter DC01 | Get-Cluster Cluster01 | Get-VMHost 192.168.105.11 | Get-VM

To get a list of powered on and powered off VMs:

(Get-Cluster Cluster01 | Get-VM).where{$PSItem.PowerState -eq "PoweredOn"}

(Get-Cluster Cluster01 | Get-VM).where{$PSItem.PowerState -eq "PoweredOff"}

An efficient way of doing it in a single go is given below:

$vmson, $vmsoff = (Get-Cluster Cluster01 | Get-VM).where({$PSItem.PowerState -eq "PoweredOn"}, 'split')

$vmson will have the list of VMs that are PoweredOn and $vmsoff will have the list of VMs that are PoweredOff.

Cluster level inventory:

Get-Cluster | Get-Inventory

Datastore details of a cluster:

Get-Cluster | Get-Datastore | select Name, FreeSpaceGB, CapacityGB, FileSystemVersion, State

Hope this was useful. Cheers! In the next post, we will talk about performing basic VM operations using PowerCLI.

VMware PowerCLI 101 - Part1 - Installing the module and working with stand-alone ESXi host

This blog post series is for all those who would like to kickstart and learn VMware PowerCLI from the very basic level. Let's start with installing the PowerCLI module.

First, verify whether VMware.PowerCLI module is already installed on your machine.

Get-Module VMware.PowerCLI -ListAvailable

In this case, as shown above, VMware.PowerCLI module is not installed. Now let's find the latest version of the module available from PowerShell gallery.

Find-Module VMware.PowerCLI

11.2.0.12780525 is the latest version that is available in PSGallery.

To install this module use: Install-Module VMware.PowerCLI

Installation of this module may take a couple of minutes as this is the very first time. Once successfully installed you can verify using: Get-Module VMware.PowerCLI -ListAvailable

To list all the available cmdlets: Get-Command -Module VMware*

Now, let's go ahead and connect to an ESXi host.

Connect-VIServer <IP of ESXi server>

Provide the necessary credentials and once connected successfully you will see the below.

To get basic details of the ESXi host: Get-VMHost <IP of ESXi server>

To get more information about a cmdlet you can make use of the PowerShell Help System.

You can find all the properties and methods available for an object using: Get-Member

Example: Get-VMHost 192.168.105.1 | Get-Member

To retrieve specific properties of an ESXi host object: 

$h1 = Get-VMHost 192.168.105.1

Model detail: $h1.Model

Processor type: $h1.ProcessorType

Another very useful property is "ExtensionData". Let's have a detailed look at this property.

This property provides you a lot of info regarding system runtime, hardware health status, etc.

System and hardware health status:

Memory and CPU health status:

Configuration status: $h1.ExtensionData.ConfigStatus

Configuration issues: $h1.ExtensionData.ConfigIssue

Few more useful details that are available under "ExtensionData" are given below.

There is a property called "RebootRequired" which actually shows whether there is any pending reboot for the system.

System uptime and resource utilization are available under "QuickStats" property.

Note:
"OverallCpuUsage" is in MHz, "OverallMemoryUsage" is in MB and "Uptime" is in Seconds.

List of all system capabilities: $h1.ExtensionData.Capability

BIOS version: $h1.ExtensionData.Hardware.BiosInfo

In the next article, we will discuss connecting to the vCenter server using PowerCLI and performing day-to-day operations. Hope this was useful. Cheers!

Related posts:

VMware PowerCLI 101 - Part2

VMware PowerCLI 101 - Part3

PowerShell quick start guide

Cisco switch configuration backup using PowerShell

Note: This blog has moved! New location vineethac.blogspot.com

In this article, I will briefly explain how to back up the running configuration of Cisco switches to a TFTP server location using PowerShell.

Prerequisites:

• A TFTP server should be configured and running.
• PowerShell module Posh-SSH should be installed on the node from which the script is running.

Workflow:

1. Collect credentials to SSH into the switch
   
   $creds = Get-Credential
2. Create a new SSH session to the first switch in the list
   
   $sw_ssh = New-SshSession -ComputerName <Management IP of Cisco switch> -Credential $creds -Force -ConnectionTimeout 300
3. Invoke the command to backup running config to TFTP server over the SSH session
   
   $cmd_backup = "copy running-config tftp://<IP of TFTP server>/config_backup.txt vrf management"
   Invoke-sshcommand -Command $cmd_backup -SSHSession $sw_ssh

You can schedule this PS script using a task scheduler so that the running configuration of switches can be backed up automatically on a daily basis or as per requirements. Hope this was useful. Cheers!

Related article:
Dell EMC switch configuration backup using PowerShell

Working with Cisco Nexus 9K switches using PowerShell

In this article, I will explain briefly about how to work with Cisco NX-API CLI through PowerShell. Before getting into the steps and sample PowerShell code, lets first familiarize with NX-API Developer Sandbox. You can access the sandbox by entering the management IP of your Cisco switch in a web browser. Provide user name and password. Once successfully logged in, you will get a page as shown below. The developer sandbox will convert the normal CLI commands into JSON request and will also provide result/ response in JSON format.

NX-API Developer Sandbox

In the above screenshot, I am using "show hostname" command. The message format is set to JSON and command type is set to cli_show to get a response in JSON. Once done, hit POST. You can see the corresponding request and response in JSON format.

Now, let's try to do the same using PowerShell.

#Step1:

<# 

To fix the connection issues <The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.>

#> 

add-type @"

    using System.Net;

    using System.Security.Cryptography.X509Certificates;

    public class TrustAllCertsPolicy : ICertificatePolicy {

    public bool CheckValidationResult(

        ServicePoint srvPoint, X509Certificate certificate,

        WebRequest request, int certificateProblem) {

        return true;

        }

    }

"@

[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12 -bor [System.Net.SecurityProtocolType]::Tls11

#Step2:

$creds = Get-Credential

#Step3:

$request = @'

{

    "ins_api": {

          "version": "1.0",

          "type": "cli_show",

          "chunk": "0",

          "sid": "1",

          "input": "show hostname",

          "output_format": "json"

    }

}

'@

#Step4:

$result = Invoke-RestMethod -Uri "http://<IP of switch>/ins" -Method post -Credential $creds -ContentType "application/json" -Body $request

Output:

Hope it was useful. Cheers!

Reference:

https://inbound.kelsercorp.com/blog/cisco-nexus-9000-automation

Real time VMware VM resource monitoring using PowerShell

This post is about monitoring resource usage of a list of virtual machines hosted on VMware ESXi clusters using PowerCLI. Output format is given below which gets refreshed automatically every few seconds.

Prerequisites:

• VMware.PowerCLI module should be installed on the node from which you are running the script
• You can verify using: Get-Module -Name VMware.PowerCLI -ListAvailable
• If not installed, you can find the latest version from the PSGallery: Find-Module -Name VMware.PowerCLI
• Install the module: Install-Module -Name VMware.PowerCLI

Note:

• I am using PowerCLI Version 11.0.0.10380590

Latest version of the project and code available at: github.com/vineethac/vmware_vm_monitor

Sample screenshot of output:

Notes:

VMware guidance: CPU Ready time and Co-Stop values per core greater than 5% and 3% respectively could  be a performance concern.

Hope this will be useful for resource monitoring as well as right sizing of VMs. Cheers!

References:

• http://docs.hol.vmware.com/HOL-2012/HOL-INF-04_EN/HOL-INF-04-m1/lessons/Part_1_of_4__Performance_101_-_vSphere_Performance_Basics_-_CPU__C__.html
• http://virtualization.solutions/2018/02/04/performance-monitoring-cpu/

Real time VMware datastore performance monitoring using PowerShell

I had a scenario where we had to monitor the real time performance statistics of multiple shared VMFS datastores which are part of a multi-node VMware ESXi cluster. So this post is about the short script which uses VMware PowerCLI to get read/ write IOPS and latency details of these shared datastores across all nodes in the cluster. The output format is given below which gets refreshed automatically every few seconds.

Prerequisites:

• VMware.PowerCLI module should be installed on the node from which you are running the script
• You can verify using: Get-Module -Name VMware.PowerCLI -ListAvailable
• If not installed, you can find the latest version from the PSGallery: Find-Module -Name VMware.PowerCLI
• Install the module: Install-Module -Name VMware.PowerCLI

Note:

• I am using PowerCLI Version 11.0.0.10380590

Latest version of the project and code available at: github.com/vineethac/datastore_perfmon

Sample screenshot of output:

Hope it was useful. Cheers!

Reference:

https://www.rudimartinsen.com/2017/07/12/vsphere-performance-data-part-3-get-stat/

Working with iDRAC9 Redfish API using PowerShell - Part 2

In this article I will explain briefly about the JSON response from iDRAC and how you can navigate through the Redfish API tree structure to get all the required information. Now, lets have a look at the URIs. 

Query the computer system collection:

$result1 = Invoke-RestMethod -Uri "https://$($idrac_ip)/redfish/v1/Systems" -Credential $Credentials -Method Get -UseBasicParsing -ContentType 'application/json'

Response: 

You can see one member with URI /redfish/v1/Systems/System.Embedded.1

Below is a sample screen shot of JSON output when you try to query the above listed member system. 

You can get some of the basic information straight away from the above JSON response. And these are organized in hierarchy where you can drill down to each object and get the required details. Below diagram shows basic iDRAC Redfish API tree structure.

Example: You can get details/ health status of  storage controller as shown below.

Query:

$result2 = Invoke-RestMethod -Uri "https://$($idrac_ip)/redfish/v1/Systems/ System.Embedded.1/Storage/Controllers/NonRAID.Integrated.1-1" -Credential $Credentials -Method Get -UseBasicParsing -ContentType 'application/json'

Response:

Hope it was useful. Cheers!

References:

Working with iDRAC9 Redfish API using PowerShell - Part 1

iDRAC9 Redfish API reference guide

github.com/dell/iDRAC-Redfish-Scripting

Working with iDRAC9 Redfish API using PowerShell - Part 1

Note: This blog has moved! New location vineethac.blogspot.com

Redfish is a industry standard protocol and specification defined by Distributed Management Task Force (DMTF) for performing systems/ IT infrastructure management actions using RESTful methodology. It is a next generation systems management interface standard which is simple, secure, scalable. Redfish uses JSON data format and transports payload over HTTPS. Initial releases of Redfish focused primarily on systems management and was targeted to be a replacement for IPMI over LAN protocol. Now the capabilities have been extended over the past few years providing a rich set of features and support for network, memory, storage devices etc. Over time the scope of Redfish is being expanded to fit more use cases as the forum is working with several partner organizations. Promoters of this standard include several companies like Broadcom, Cisco, Dell, HP, VMware, Intel, Microsoft etc.

Now, lets have a look at how to connect to iDRAC Redfish API using PowerShell. Redfish provides two authentication methods. Basic authentication and Session-based authentication. Here I will explain basic authentication using username and password for each Redfish API request to iDRAC.

#To fix the connection issues to iDRAC REST API

add-type @"

    using System.Net;

    using System.Security.Cryptography.X509Certificates;

    public class TrustAllCertsPolicy : ICertificatePolicy {

    public bool CheckValidationResult(

        ServicePoint srvPoint, X509Certificate certificate,

        WebRequest request, int certificateProblem) {

        return true;

        }

    }

"@

[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12 -bor [System.Net.SecurityProtocolType]::Tls11

#Get iDRAC creds

$Credentials = Get-Credential -Message "Enter iDRAC Creds"

#URI to get basic system info

$u1 = "https://192.168.10.11/redfish/v1/Systems/System.Embedded.1"

#Using Invoke-RestMethod

$result1 = Invoke-RestMethod -Uri $u1 -Credential $Credentials -Method Get -UseBasicParsing -ContentType 'application/json'

Output:

Hope it was useful. Cheers!

References:
iDRAC9 Redfish API reference guide
github.com/dell/iDRAC-Redfish-Scripting