vSphere ESXi 6.7 Unable to talk between subnets between hosts

Configuration was :

ESXi 6.7 – vSphere handling all the port groups tagged with VLANs

Firewall – CISCO ASA


Problem : New ESXi 6.7 hosts. A virtual machine if on the same host and vSwitch could communicate no problem. However if a virtual machince was communincating with another virtual machine on another host on a different, subnet they were unable to communicate between subnets and hosts. Both virtual machines could ping their local gateways. Firewall, CISCO ASA was just dropping all packets and showing the following error

Error on the Firewall when capturing ping traffic “No source port  on ping “Error (Type 8, Code 0), Denied ICMP type=8, code=0”



Sometimes its the simple tick box on the Firewall / ASA config

“Enable traffic between two or more interfaces which are configured with the same security levels”

All traffic started communicating and the virtual machines could talk between the subnets as per the rules on the Firewall.

VMware vCenter Single Sign-On – Invalid Credentials – Native Platform Error code 1765328360

Logging in to the vCenter Server Appliance fails with the error: Failed to authenticate user


Failed to authenticate principal for tenant vsphere.local 6.5 update1

KB on issue https://kb.vmware.com/s/article/2147174

Logging in to the vCenter Server Appliance Web Client and / or vSphere Client fails with the error:

Failed to authenticate user
/logs/sso/vmware-sts-idmd.log file, you see entries similar to:


  • [YYYY-MM-DDT<time> vsphere.local d5ee8f23-b216-4585-b829-6e4c671d6ede ERROR] [IdentityManager] Failed to authenticate principal [Username@DOMAIN] for tenant [vsphere.local]
    com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: -1765328347][null][null]

Trouble shoot

Login as administrator@vsphere.local

Open VCSA consoleCommand for the appliance

Check VCSA version

vpxd -v

We had : build-8024368

Which log to check :

Become an Expert on vCenter Server appliance Log File Location in 120 seconds

vCenter Server appliance log file location

Connect into vCenter Server appliance using SSH as root and browse towards the directory “/var/log/vmware” to see the list of all logs files of vCenter server appliance 6.5
Browse to the log and open file

To go back down a directory in VCSA

cd ../

to check the date and time on the VCSA

date      Note it is displayed in UTC time zone

They time on ours appears to be out of sync between the vCenter Server machine, and the domain controller


So we then set NTP on the vCenter Server Appliance 6.5


Steps to enable and add NTP servers on the vCenter Server Appliance 6.5

To enable NTP on the vCenter Server Appliance 6.5:

  1. Open a console session to the vCenter Server Appliance and press ALT+F1
  2. Log in using the root user credentials.
  3. Run this command to view the current settings:
  4. Run this command to add an NTP server:
    ntp.server.add –servers ntp_servername
  5. Run this command to verify the NTP server settings:
  6. Get NTP ModeTimesync.getNTP Mode
  7. To remove an NTP server, run this command:
    ntp.server.delete –-servers ntp_servername 



    This issue occurs when:

    • The time is out of Sync with the domain controller the appliance is joined to
    • No NTP time source
    • NTP status down



    VMware vCenter Server 6.5 Update 1 Release Notes



    This blog is funded by AD clicks. See and AD of interest? Click it. 🙂

VMware vRealize – What is vRealize Orchestrator(vRO)?


This is my interpterion of VMware vRealise Orchestrator at a high level. VMware vRealise Orchestrator is a centralised location to create workflows, create actions and define configuration elements to automate tasks. In ddition there is a useful tool built-in which can auto generate documentation based on your workflows created.

so, how does vRO compare to my previous work?

My first impression of vRO, is the tool reiterates the structure I have been putting in place via PowerShell scripts on previous projects and demonstrations.

Previously I would create a simple CSV which contained all the server / virtual machine information

See my previous work on bulk virtual machine deployments:

The logic is :

Within vRealise the procedures are all done in work flows. Its worth noting scriptable tasks are in JavaScript. Scripts can be added to workflows

The scriptable tasks are more interesting. You can predefine properties/variables to use within the script tasks.

Some nice key features :

  • Descriptions area for workflows
  • Version control work flows.
  • Simple Output to System.log

Is there a similar alternative product I’ve used.

Some of vRO features do have similarities to an alternative product I have recently been using called Octopus. Octopus has the options to create processes, add scripts including PowerShell and version control each release and control releases.

Moving on to more advance scriptable tasks in vRO, decisions can then be defined based on If …Else statements to define the output. When scripting within workflows, a simple but effective feature is when a variable is entered in a script, the variable name changes colour.

Out the box options
There are some out the box options such as VMware tools and Virtual Hardware upgrades with vRO workflows.
There are many more advanced topics and work flows. Feel free to share your favourite work flows or developed work flow on twitter @stephenhackers

Note : All opinions are my own.



Add VM Custom Annotation and Create a Report on Annotations

Add Custom Attributes for Notes Annotation

A request to add custom attributes for Virtual Machines when using the fat client. (Web client in 5.1 and 5.5 requires a plugin, see “vsphere-web-client-plugin-for-custom”) 6.0 doesn’t see the attributes in the Web Client, 6.5 does, see the 6.5 KB.

Fields required : Applications, Company Name, Owner, Role, VM Cost

Code Below

Connect-VIServer VC6.test.domain
New-CustomAttribute -Name “Company Name” -TargetType VirtualMachine
New-CustomAttribute -Name “VM Cost” -TargetType VirtualMachine
New-CustomAttribute -Name “Role” -TargetType VirtualMachine
New-CustomAttribute -Name “Owner” -TargetType VirtualMachine
New-CustomAttribute -Name “Applications” -TargetType VirtualMachine
disconnect-VIServer VC6.test.domain -Confirm:$false


Add the details required


Bulk Virtual Machines Deployment and Zero Clicks Part 1

Add additional code code to add annotation in to the bulk script

$companyname = $item.companyname
$applications = $item.applications
$owner = $item.owner
$role = $item.role
$cost = $item.cost

#Get the Specification and set the Nic Mapping
New-OSCustomizationNicMapping -Spec $custspec -IpMode UseStaticIp –Position 1 -IpAddress $ipaddr -SubnetMask $subnet -DefaultGateway $gateway -Dns $pdns,$sdns

#Create VM using Template with the adjusted Customization Specification
New-VM -Name $vmname -Template $template -Datastore $datastore -VMHost $vmhost -ResourcePool $resourcepool | Set-VM -OSCustomizationSpec $custspec -Confirm:$false

#Set the Network Name
Get-VM -Name $vmname | Get-NetworkAdapter | Set-NetworkAdapter -NetworkName $vlan -Confirm:$false

#Set the CPU and Memory
Get-VM -Name $vmname | Set-VM -MemoryGB $ram -NumCPU $cpu -Confirm:$false

#Set some custom attribute fieds
#New-CustomAttribute -Name “VM Cost” -TargetType VirtualMachine
#New-CustomAttribute -Name “Role” -TargetType VirtualMachine
#New-CustomAttribute -Name “Owner” -TargetType VirtualMachine
#New-CustomAttribute -Name “Applications” -TargetType VirtualMachine

#Set annotation value for custom attributes
Set-Annotation -Entity $vmname -CustomAttribute “CompanyName” -Value “$companyname”
Set-Annotation -Entity $vmname -CustomAttribute “Applications” -Value “$applications”
Set-Annotation -Entity $vmname -CustomAttribute “Owner” -Value “$owner”
Set-Annotation -Entity $vmname -CustomAttribute “Role” -Value “$role”
Set-Annotation -Entity $vmname -CustomAttribute “VM Cost” -Value “$cost”



RV Tools can be used to produce an MS Excel file to output a list of virtual machines and custom annotations RV Tools download


Alternative Report function used

Function Code Below
(greg-get-annotations tested successfully in our lab)……………….

function greg-get-annotations {
<# .DESCRIPTION Greg-get-annotations function stores information about annotation fields for vms in given cluster or in all clusters in VC. It stores the result in an arraylist $vms, you can either create a csv report from this object or display it on screen greg-get-annotations |export-csv -NoTypeInformation c:\file1.csv will export it to csv file etc… greg-get-annotations |format-table VMname,Cluster,CreatedOn,Notes will just display on screen a table with annotations that include : vm name, its cluster and field “CreatedOn” and Notes   .PARAMETER clustername Specifies the clustername against wchi report will be built   .EXAMPLE greg-get-annotations -clustername ‘cluster01’|Export-Csv c:\annotation-report.csv Will procude report on vms that resides in ‘cluster01’ and store it in csv file   .EXAMPLE greg-get-annotations -clustername ‘cluster01’|ft * Will procude report on vms that resides in ‘cluster01’ output it to screen   .EXAMPLE greg-get-annotations |Export-Csv c:\annotation-report.csv Will procude report on vms that resides in all clusters and output it to screen   .EXAMPLE greg-get-annotations Without specified -clustername switch, it will do report regarding all clusters in VC   .NOTES AUTHOR: Grzegorz Kulikowski LASTEDIT: 05/30/2011     #>
param ([string]$clustername)
if(!($clustername)){$clusters=Get-Cluster}else{$clusters=Get-Cluster $clustername}
$VMs=New-Object Collections.ArrayList
foreach ($cluster in $clusters)  {
foreach ($vmview in (get-view -ViewType VirtualMachine -SearchRoot $cluster.id)) {
$vm=New-Object PsObject
Add-Member -InputObject $vm -MemberType NoteProperty -Name VMname -Value $vmview.Name
Add-Member -InputObject $vm -MemberType NoteProperty -Name Notes -Value $vmview.Config.Annotation
Add-Member -InputObject $vm -MemberType NoteProperty -Name Cluster -Value $cluster.Name
foreach ($CustomAttribute in $vmview.AvailableField){
Add-Member -InputObject $vm -MemberType NoteProperty -Name $CustomAttribute.Name -Value ($vmview.Summary.CustomValue | ? {$_.Key -eq $CustomAttribute.Key}).value
return $VMs

greg-get-annotations |Export-Csv c:\annotation-report.csv


CSV Out Put



Bulk Virtual Machines Deployment and Zero Clicks Part 1

A recent project revisited deploying virtual machines via PowerCli. Its fair to say this isn’t a new tool but sometimes over looked.

Part 1 /  Part 2

My requirements were to deploy :

  • 100+ virtual machines (within a few hours)
  • domain join all machines
  • license the OS
  • various virtual machine specifications
  • various Windows OS versions.
  • to two different data centers within a linked vCenter setup
  • to resource pools
  • to different data stores
  • to different networks


The constraints:

  • vSphere 6.0 update 2
  • no budget for third party automation tools
  • small window of opportunity to deploy the VMs


On the plus side there was:

o    Loads of available CPU and RAM
o    Large datastores presented
o    Subnets prepared
o    Stretched VLANs across Data Centers


The tools I used to the task

  • Excel (CSV)
  • Notepad++
  • PowerCLi


The CSV file example

# Example Bulk_VMs_Deploy.csv

Template Datastore VMhost Custspec VMname IPaddress Subnet Gateway
2012_Template Storage1 ESXi.domain 2012_Spec test2003VL1
PDNS SDNS ResourcePool RAM CPU VLAN Size Format resource1 2 2 VM Network 10 thin


The Script

# Automate the deployment of customised virtual machines deployed in vSphere 6.0. Tested against u2
# Prereq’s
# 1) Populate the a CSV file called Bulk_VMs_Deploy.csv
# 2) Create a Windows Server template
# 3) Create a customization spec within vSphere for Windows
# 4) Run Bulk_VMs_Deploy.ps1 script via PowerCli as administrator (CSV file must be stored in the same location where the script is run from)
if ( !(Get-Module -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue) ) {

###### IMPORTANT, Check this file path is correct##########
. “C:\Program Files (x86)\VMware\Infrastructure\PowerCLI\Scripts\Initialize-PowerCLIEnvironment.ps1”
Connect-VIServer VC6.test.domain
#connect to a VC. This also works with Linked VC’s
$vmlist = Import-CSV .\Bulk_VMs_Deploy.csv
foreach ($item in $vmlist) {

#set variables to read from CSV
$template = $item.template
$datastore = $item.datastore
$vmhost = $item.vmhost
$custspec = $item.custspec
$vmname = $item.vmname
$ipaddr = $item.ipaddress
$subnet = $item.subnet
$gateway = $item.gateway
$pdns = $item.pdns
$sdns = $item.sdns
$resourcepool = $item.resourcepool
$cpu = $item.cpu
$ram = $item.ram
$vlan = $item.vlan
$size = $item.size
$format = $item.format

#Get the Specification and set the Nic Mapping
New-OSCustomizationNicMapping -Spec $custspec -IpMode UseStaticIp –Position 1 -IpAddress $ipaddr -SubnetMask $subnet -DefaultGateway $gateway -Dns $pdns,$sdns

#Create VM using Template with the adjusted Customization Specification
New-VM -Name $vmname -Template $template -Datastore $datastore -VMHost $vmhost -ResourcePool $resourcepool | Set-VM -OSCustomizationSpec $custspec -Confirm:$false

#Set the Network Name
Get-VM -Name $vmname | Get-NetworkAdapter | Set-NetworkAdapter -NetworkName $vlan -Confirm:$false

#Set the CPU and Memory
Get-VM -Name $vmname | Set-VM -MemoryGB $ram -NumCPU $cpu -Confirm:$false

#Additional Disk
#Get-VM -Name $vmname | New-HardDisk -CapacityGB $size -StorageFormat $format -Confirm:$false

#Remove the NicMapping
Get-OSCustomizationSpec $custspec | Get-OSCustomizationNicMapping | Remove-OSCustomizationNicMapping -Confirm:$false

#PowerOn VM
Start-VM $vmname

#Disconnect from VC.
disconnect-VIServer VC6.test.domain -Confirm:$false



Disclaimer Please take the code and evolve it into a different project? Credit / Tag me on your project Twitter #StephenHackers

Any use of this code is at your own risk. Remember bulk automation jobs require the right resources to be available.

This project & code was based on :
Which progressed to : https://communities.vmware.com/thread/436734

Part 1 /  Part 2

In What Order Do I Upgrade VMware Products and…

In What Order Do I Upgrade VMware Products and What Is Compatible? – VMware Compatibility and Order of Upgrade Best Practices Review

In What Order Do I Upgrade VMware Products and…

After attending VMware TechSummit 2017 last week, I’ve got a ton of takeaways to share over the next few months, but I wanted to get this article out there because it’s about two of the most common requests I hear: product compatibility and the order of product upgrades.

VMware Social Media Advocacy

Learn NSX  Day 6 : NSX and Logical Switches

Replication Modes on Logical Switches

logical-switch options

logical-switch options


Broadcast (BUM)
Hybrid Mode Logical Switch





Sort of utilises both Unicast and Multicast traffic


Unknown Uni-cast replication



Reason to use :
Opposite to Multicast. Separation of the Physical and Logical networks
No PIM or IGMP on physical network. Non-ESXi don’t receive BUM option
Configurable in the Transport Zone (VTEPS Send Uni-cast and can remote proxy in transport zone)

Multicast Replication



Reason to use :
NSX relies on Layer 2 and Layer 3 multicast for physical network for VXLAN encapsulated multi destination is sent to all VTEPS
(page  26) 
Required PIM and L3 multicasting routing
Least amount of bandwidth used on physical network architecture


Logical Switches

Prep work :
Config VXLAN tunnel endpoint (VTEP) VLAN on trunk in physical switches

A good article on logical switching and transport zones was available this page by  Alex Hunt – Logical Switching and Transport Zones 

Spine – Leaf architected networks



A great description by Ethan Banks below :
“In modern data centers, an alternative to the core/aggregation/access layer network topology has emerged known as leaf-spine. In a leaf-spine architecture, a series of leaf switches form the access layer. These switches are fully meshed to a series of spine switches.

Network overlays such as VXLAN are common in highly virtualized, multi-tenant environments such as those found at Infrastructure as a Service providers. Arista Networks is a proponent of layer 3 leaf-spine designs, providing switches that can also act as VXLAN Tunnel Endpoints.” By Ethan Banks


Topics :
Learn NSX – Home
Learn NSX Day 1 : NSX Requirements
Learn NSX Day 2 : NSX Deployment Best Practice
Learn NSX Day 3 : NSX Manager and NSX Controller
Learn NSX Day 4 : NSX Roles
Learn NSX Day 5 : NSX HA, Edge, REST API
Learn NSX  Day 6 : NSX and Logical Switches
Learn NSX Day 7 : Deploying ESXi
Learn NSX Day 8 : vDS (vSphere Distributed Switch)
Learn NSX Day 9 : Load Balancing Feature
Learn NSX Day 10 : Layer 2 Bridging
Learn NSX Day 11 : NSX EDGE
Learn NSX Day 12 : Spoof Guard
Learn NSX Day 13 : Distributed Router and Distributed Logical Firewall
Learn NSX Day 14 : Monitoring
Learn NSX Day 15 : NSX Backups
Learn NSX Day 16 : Useful Commands & Errors


These are notes made during my study of VMware NSX for vSphere. Apologises if any of the detail is incorrect. Hopefully posts under “Learn NSX” help others to start learning about VMware NSX for vSphere.


VMware discussing NSX Notes


  • NSX 201 cross centre metro cluster, data centre migration / DR use case
  • NSX futures – distributed network encryption, management of containers, securing multiple clouds
  • VMware strategy for the cloud native applications and dev ops
  • Deep dive on Photon, lightwave, container management

NSX Discussion
How does NSX protect against bank attach or Twitter hack

NSX micro segmentation (isolation of apps)  ..DFW

3rd party integration like trend , Mcafee .. Steer traffic for layer 7 traffic.. For malware inspection.. Security tag VM.. It tells NSX to automate a policy, possible a quarantine policy

3rd party’s develop the tags ( protects against bank attach or Twitter hack etc ). NSX relies on the 3rd party NSX tags to apply a policy. NSX does have some features for tagging using activity monitoring an VM tools. VM tools can see what is running within the VM.

DMZ anywhere .. How long to provision a VM web facing and secure. NSX can spin it up instantly.

Secure user environment ..

Overlay virtual networking ( abstract )
Stretch across sites
Tunnelling ( change from vLans etc )
Logical layer 2. Packet in envelope. Encapsulate the traffic.

NSX is distributed across hosts rather than all going via a central physical firewall.

Develop cloud – Strategy vRealise but also others such as openstack
Others doing networks , Neutron in theory not as scale able. Challenge is abstraction layer.

Time to setup a PoC!!!

VMware do a NSX 2 day training


Containers – Cloud Native Apps

Contains, Windows 2016 ,G1 -> G2 -> G3 photon
Challenges with containers
Containers sit in the same user space on the same VM. Deploy multiple containers only appear as one VM in vcentre.

Secured By miniOrange