Showing posts with label ScaleIO. Show all posts
Showing posts with label ScaleIO. Show all posts

Wednesday, 30 May 2018

Creating HTML report of ScaleIO cluster using PowerShell

This post is a reference to a small reporting script for ScaleIO environments. The project will generate a brief HTML report of your ScaleIO Ready Node SDS infrastructure (with AMS - Automated Management Services) by making use of ScaleIO Ready Node AMS REST APIs and PowerShell. The report provides information about MDM cluster state, overall cluster capacity, system objects, alerts, and health state of all disks in the cluster. Here the API is available as part of ScaleIO Ready Node AMS. These AMS REST API allows you to query information and perform actions related to ScaleIO software and ScaleIO Ready Node hardware components. To access the API you need to provide AMS username and password. Responses returned by AMS server are formatted in JSON format.

Project reference: https://github.com/vineethac/sio_report

Use case
: This script can be used/ leveraged as part of daily cluster health/ stats reporting process, or something similar; so that monitoring Engineers or whoever responsible can have a look at it on a daily basis to make sure everything is healthy and working normal. 

Related references:

Hope this was helpful. Cheers!

Tuesday, 20 March 2018

Working with ScaleIO REST API using PowerShell - Part 3

As you all know disk drives are the most important and basic building blocks of a storage system. In order to ensure data availability, the disk drives in your storage system should always stay healthy. Unhealthy drives not only cause risk to your data but also contribute to degraded performance. In this article, I will show how to query and get disk health-related information in a ScaleIO cluster using REST API and PowerShell. 

Please refer my previous posts for API authentication, token generation and 'Invoke-RestMethod'.


#Query all devices in the ScaleIO cluster
$all_devices = (Invoke-RestMethod -Uri "https://192.168.11.15:443/api/types/Device/instances " -Method Get -Headers $ScaleIOAuthHeaders)  

#Select necessary properties and sort
$all_devices = (Invoke-RestMethod -Uri "https://192.168.11.15:443/api/types/Device/instances " -Method Get -Headers $ScaleIOAuthHeaders) | select sdsId, storagePoolId, name, deviceCurrentPathName, errorState, deviceState, ssdEndOfLifeState, temperatureState, aggregatedState  | sort sdsId | ft -AutoSize


#Sample output

Hope this was helpful. Please refer to ScaleIO 2.5 API reference guide for more details. Cheers!

Wednesday, 7 March 2018

Working with ScaleIO REST API using PowerShell - Part 2

In my previous post, I explained how to connect and authenticate with ScaleIO REST API, and use Invoke-RestMethod. In this article, I will explain how to query selected statistics/ properties with POST method. 

First, you have to define the set of properties that you would like to query, and then pass it to "invoke-restmethod". You can get the list of available properties of the respective object from the ScaleIO REST API Reference Guide. 

Example 1: Query and collect MDM cluster information
Here I am collecting the below-mentioned properties of "MDMCluster" object. 

#selected properties to query

$param1 = @'
{
    "properties":["name", "clusterMode", "master", "clusterState", "virtualIps"]
}
'@

#invoke-restmethod with respective uri

$MDM_cluster_stats = (Invoke-RestMethod -Uri "https://192.168.11.15:443/api/instances/System/queryMdmCluster " -Body $param1 -ContentType "application/json" -Headers $ScaleIOAuthHeaders -Method Post)

#organize output in a hash table

$mdm_props = @{

    'Cluster name'      = ($MDM_cluster_stats).name
    'Mode'              = ($MDM_cluster_stats).clusterMode
    'Cluster state'     = ($MDM_cluster_stats).clusterState
    'Master MDM IP'     = ($MDM_cluster_stats.master).managementIPs[0]
    'Cluster VIP01'     = ($MDM_cluster_stats).virtualIps[0]
    'Cluster VIP02'     = ($MDM_cluster_stats).virtualIps[1]
}

Write-Output $mdm_props

#sample output screenshot given below


Example 2: Query and collect overall ScaleIO system capacity details
Here I am collecting some properties of "System Statistics" object. 

#selected properties to query

$param2 = @'
{
    "properties":["maxCapacityInKb", "capacityInUseInKb", "spareCapacityInKb", "failedCapacityInKb", "degradedFailedCapacityInKb"]
}
'@

#invoke-restmethod with respective uri

$system_overall_stats = (Invoke-RestMethod -uri "https://192.168.11.15:443/api/types/System/instances/action/querySelectedStatistics " -Body $param2 -ContentType "application/json" -Headers $ScaleIOAuthHeaders -Method Post)

#organize output in a hash table


$system_capacity_props = @{

    "System max capacity (TB)"                = (($system_overall_stats.maxCapacityInKb)/1024/1024/1024)
    "System capacity in use (TB)"             = (($system_overall_stats.capacityInUseInKb)/1024/1024/1024)
    "System spare capacity (TB)"              = (($system_overall_stats.spareCapacityInKb)/1024/1024/1024)
    "System failed capacity (TB)"             = (($system_overall_stats.failedCapacityInKb)/1024/1024/1024)
    "System degraded failed capacity (TB)"    = (($system_overall_stats.degradedFailedCapacityInKb)/1024/1024/1024)
}

Write-Output $system_capacity_props

#sample output screenshot given below


Hope this was useful. Please refer Dell EMC ScaleIO Ready Node Version 2.5 API Reference Guide for more details.

Wednesday, 28 February 2018

Working with ScaleIO REST API using PowerShell - Part 1

In this article, I will explain briefly how to work with REST API exposed by ScaleIO software using PowerShell to manage/ monitor your ScaleIO system. ScaleIO can be deployed either using AMS (only for ScaleIO Ready Nodes) or without AMS. In case of an AMS deployment, API is available as part of the ScaleIO Ready Node AMS package. The AMS REST API is served from the AMS server which allows querying information and performing several actions related to ScaleIO Ready Node hardware and software components. For a non-AMS ScaleIO deployment, the REST API is served from the ScaleIO Gateway. If you used ScaleIO Installation Manager (IM) to install ScaleIO, the gateway has already been installed and configured with the MDM details. In both cases, the AMS server/ ScaleIO Gateway connects and queries master MDM, reformats the results in a RESTful manner and passes it to a REST client.

Here I am using an AMS managed ScaleIO Ready Node cluster. Access to the API requires a login using AMS username and password. Successful login returns a token, and you can use this token for later authentications for other requests. The responses or results returned by ScaleIO Ready Node AMS are in JSON format. In the following sections, I will explain how to use PowerShell to query ScaleIO REST API and obtain relevant results.  

PowerShell uses Invoke-RestMethod cmdlet which sends HTTP/ HTTPS requests to REST web services that return rich structured data. PowerShell will format the response based on the data type. For JSON PowerShell converts the response content to objects. 

Step 1: Authentication and token generation

#Note: 192.168.11.15 is the IP of my AMS server
$User = Read-Host -Prompt "Please enter ScaleIO AMS username"
$SecurePassword = Read-Host -Prompt "Enter ScaleIO Password for user $user" –AsSecureString
$Credentials = New-Object System.Management.Automation.PSCredential ($user,$Securepassword)

#HTTPS Get request is invoked with credentials for authentication
$Token = Invoke-RestMethod -Uri "https://192.168.11.15:443/api/login" -Method Get -Credential $Credentials

Step 2: Create header with the token
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(':'+$Token))
$global:ScaleIOAuthHeaders = @{'Authorization' = "Basic $auth"
'Content-Type' = "application/json" }

Step 3: Invoke-RestMethod

#Query all alerts in the system
$alerts = (Invoke-RestMethod -Uri "https://192.168.11.15:443/api/types/Alert/instances/" -Method Get -Headers $ScaleIOAuthHeaders)

#Query all SDSs in a PD
$all_SDS = (Invoke-RestMethod -Uri "https://192.168.11.15:443/api/instances/ProtectionDomain::3ce4af5f00000000/relationships/Sds " -Method Get -Headers $ScaleIOAuthHeaders)

#Query a specified PD
$pd01 = (Invoke-RestMethod -Uri "https://192.168.11.15:443/api/instances/ProtectionDomain::3ce4af5f00000000 " -Method Get -Headers $ScaleIOAuthHeaders)
#Query statistics of a specified PD
$pd01_stats = (Invoke-RestMethod -Uri "https://192.168.11.15:443/api/instances/ProtectionDomain::3ce4af5f00000000/relationships/Statistics " -Method Get -Headers $ScaleIOAuthHeaders)

Hope this was useful. Refer to Dell EMC ScaleIO Ready Node Version 2.5 REST API Reference Guide for more details.
In the next article, I will explain how to query selected statistics/ parameters with POST method which will help you to
create custom monitoring scripts for your ScaleIO Ready Node environment.

Note:
If you are getting the below error while trying to create the token during step 1, please add the below lines of code to the
start of your script.

"Invoke-RestMethod : The underlying connection was closed: An unexpected error occurred on a send."

#Solution to above error!
add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
        public bool CheckValidationResult(
            ServicePoint srvPoint, X509Certificate certificate,
            WebRequest request, int certificateProblem) {
            return true;
        }
    }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12 -bor [System.Net.SecurityProtocolType]::Tls11

References:

Thursday, 30 November 2017

Software Defined Storage using ScaleIO

In this article I will explain briefly about ScaleIO and various options that are available to deploy ScaleIO software defined storage (SDS) solution. 

ScaleIO can be considered as a very good option for customers who are moving towards deploying software defined storage  solutions and hyperconverged infrastructure. As ScaleIO software supports multiple hypervisors and operating systems like VMware ESXi, Hyper-V, RHEL, Windows etc. customers with a heterogeneous IT infrastructure gets the most benefit out of it. Apart from that it offers multiple deployment modes like hyperconverged, two layer and mixed mode. I am sure most of you are very much familiar with the term hyperconverged where compute and storage runs together on the same box. You can scale both compute and storage resources together by adding more and more nodes to your cluster. A two layer mode is nothing but a storage only configuration where you can scale the storage resources separately. It is essentially a virtual SAN infrastructure implemented using ScaleIO SDS. A mixed mode scenario will usually occur when transitioning from storage only configuration to hyperconverged.

Now I will just give an overview on how to deploy ScaleIO on VMware and RHEL platforms. ScaleIO has tight integration with VMware and they provide a powershell script and vCenter plugin to simplify the deployment. In case of RHEL platform, you can use Installation Manager (IM) which is a part of ScaleIO Gateway for quick and easy deployment of ScaleIO cluster. Customers have multiple options to consume ScaleIO. They can just buy the ScaleIO software alone and use commodity x86 hardware to build the cluster (not a great idea for production deployments as they have to figure out and use the validated/ qualified hardware and software components to ensure seamless operation and proper support) or they can buy ScaleIO Ready Nodes which are prevalidated, preconfigured and optimized PowerEdge servers to deploy ScaleIO cluster. Apart from that there is another offering VxRack System Flex which is a rack-scale hyperconverged solution built on Dell EMC PowerEdge servers with integrated Cisco networking and ScaleIO software. 

Lets have a look at the major components of ScaleIO. Below figure shows a 5 node hyperconverged ScaleIO cluster running on a highly available VMware platform. The three main components of ScaleIO are:

  • SDC - ScaleIO Data Client
  • SDS - ScaleIO Data Server
  • MDM - Meta Data Manager


In this scenario, all 5 nodes have ESXi installed and clustered. All nodes have local hard disks present in them. And its the responsibility of ScaleIO software to pool all the hard disks from all 5 nodes forming a distributed virtual SAN.

SDC is a light weight driver which is responsible for presenting LUNs provisioned from the ScaleIO system. SDS is responsible for managing local disks present in each node. MDM contains all the metadata required for system operation and configuration changes. It manages the metadata, SDC, SDS, system capacity, device mappings, volumes, data protection, errors/ failures, rebuild and rebalance operations etc. ScaleIO supports 3 node/ 5 node MDM cluster. Above figure shows a 5 node MDM cluster, where there will be 3 manager MDMs and out of which one will be master and two will be slaves and there will be two Tie-Breaker (TB) which helps in deciding master MDM by maintaining a majority in the cluster. In a production environment with 5 or more nodes, it is recommended to use a 5 node MDM cluster as it can tolerate 2 MDM failures.

ScaleIO uses a distributed two way mesh mirror scheme to protect data against disk or node failures. To ensure QoS it has the capability where you can limit bandwidth as well as IOPS for each volume provisioned from a ScaleIO cluster. And regarding scalability a single ScaleIO cluster supports upto 1024 nodes. In very large ScaleIO deployments it is highly recommended to configure separate protection domains and fault sets to minimize the impact of multiple failures at the same time. 

You can download ScaleIO software for free to test and play around in your lab environment.

References:
Dell EMC ScaleIO Basic Architecture
Dell EMC ScaleIO Design Considerations And Best Practices
Dell EMC ScaleIO Ready Node