When design the stretched VxRail vSAN cluster, there is an option to utilize Witness Traffic Separation (WTS) to separate the vSAN witness traffic from vSAN vmkernel interfaces. This post is to discuss what is the WTS and how to deploy it.
What is vSAN Witness Traffic Separation (WTS)?
Witness Traffic Separation (WTS) is officially introduced from vSAN 6.5. The WTS is designed to separate VMware vSAN data traffic from witness traffic in two-node vSAN cluster and the stretched cluster.
Without WTS, for vSAN stretched cluster, the vSAN VLAN need to be stretched across data sites and witness site. However, in some scenarios, the VLAN could not be extended outside the Core DC. The WTS is useful when the vSAN VLAN cannot be stretched to the witness site, also this could allow the witness appliance to be deployed in public cloud, such as AWS etc.
It is noted that, when deploying the witness appliance in public cloud, it is important to check the network requirement including latency, firewall requirement and etc.
In VxRail stretched cluster solution, the WTS can be utilized by introducing an additional VMKernel interface in each ESXi data node to carry the witness traffic.
For example, there are two data centers (DC1 and DC2) with four VxRail nodes in each site, by default, each VxRail ESXi have five VMKernel interfaces pre-configured by VxRail as below:
- Vmk0 – External mgmt. -Stretched VLAN
- Vmk1 – iDRAC network -No uplink
- Vmk2 – Internal mgmt. -Stretched VLAN
- Vmk3 – VSAN Data -Stretched VLAN
- Vmk4 – vMotion -Stretched VLAN
To configure WTS, the additional vmk5 need to be added and configured as below:
- DC1 ESXi – vmk5 -Witness Traffic -DC1 Only WTS VLAN(WTS-VLAN1)
- DC2 ESXi – vmk5 -Witness Traffic -DC2 Only WTS VLAN(WTS-VLAN2)
The Witness appliance need to be deployed in Witness site with a vSAN kernel interface configured as below:
- Witness ESXi – vmk0 -Mgmt. Traffic
- Witness ESXi – vmk1 -vSAN Traffic -vSAN VLAN (WITNESS-VLAN)
The vmk5 from each vSAN ESXi data node should be able to communicate with vmk1 in witness server in bi-directions via L3 routing.
The high-level configuration steps for WTS.
Step 1: Build a standard VxRail vSAN cluster via VxRail Wizard. The steps to build a standard cluster and stretched cluster is same in VxRail wizard.
Step 2: Deploy the Witness sever or Witness Appliance in witness site with two vmkernel interfaces. (vmk0 for management and vmk1 for vSAN traffic type.)
Step 3: In VxRail vSAN data nodes in each site, add vmkernel interface in each node for WTS as below:
- In each ESXi data node, add additional VMKernel interface vmk5 in VLAN WTS-VLAN1(DC1) or WTS-VLAN2(DC2) with no traffic type configured.
- In each ESXi data node, use below command to configure and check the new VMkernel interface with traffic type “Witness”
esxcli vsan network ip add -i vmk5 -T=witness esxcli vsan network list
Step 4: In Witness appliance, Check there is VMKernel interface in VLAN WTS-VLAN3 and IP with traffic type “vSAN” configured. Login Witness appliance and use below command to check.
esxcli vsan network list
Step 5: In vCenter, configure vSAN stretched cluster in vCenter vSAN “Fault Domain” configuration page.
Step 6: Add additional Isolation Response Addresses for vSAN cluster High Availability. Choose one or two IP addresses in the stretched vSAN VLAN as the isolation addresses, normally the vSAN VLAN gateway could be used.
https://kb.vmware.com/s/article/1002117
For now, you should have the VxRail Stretched Cluster with WTS configured. Enjoy!
Hi thank you for your blog, in the case of the L3 vSAN VLAN between the datacenters, is it possible to have the WTS between the datacenters and witness?
Hi Mas,
I didn’t see any issue to configure WTS even you use L3 for vSAN traffic.
In ESXi, the new WTS vmkernel interface traffic type will be tagged as “Witness” and ESXi will control the route to use WTS vmkernel interface for “WTS” traffic.
Just make sure the Witness Server vSAN VLAN is a different VLAN than others you are using in your solution. Otherwise you will have routing issue.
Hello,
I have just a little question.
If I understood correctly, we must have 3 other news vlan for that ?
One for the witness appliance vmk1(named witness-vlan), one for the nodes on DC1 (vmk5 named wts-vlan1) and one for the nodes on DC2 (vmk5 named wts-vlan2) ?
The quick answer is YES.
The purpose of WTS is to use separate vmkernel interface for witness traffic, therefore, you need a separate VLAN at each DC for witness traffic. It can be existing VLAN in your org but should be a different from the other VSAN related VLANs used in VxRail.
Do not suggest to use stretched VLAN for WTS for both DC, there might be a gateway problem and might impact the VSAN witness.
For “witness-vlan”, it is always suggested that the witness should be hosted in a remote physical location. Therefore, in majority cases, there is a different VLAN for “witness-vlan” in witness site.
thank you for your reply.
You say :
“Do not suggest to use stretched VLAN for WTS for both DC, there might be a gateway problem and might impact the VSAN witness.”
But if I use 3 news Vlan I must use gateway.
You don’t think it s better use only one new Vlan (in layer 2) ?
Technically, you can use a single stretched VLAN across DC1, DC2 and Witness sites for WTS traffic, but this is not practical in most customer environments. As witness site is always a remote site or even living in public cloud.
To a certain degree, the WTS is designed to allow the witness traffic to go through a separate gateway to go outside of DC. If you would like to use Layer 2, there is no point to configure WTS as the witness traffic by default will use VSAN VLAN, which is layer 2 as well.
However, if you would like to configure WTS and use a new layer2 stretched VLAN across three sites, I didn’t see any technical issue for that.
Hello,
Thank you so much for this post, very informative.
I’d like to ask a question in regards to WTS VLAN, is it mandatory to create a new VLAN for node-WTS trafic? Can we juste use the same VLAN used for Witness vSAN trafic ? If not, and I see you mentioned routing issues, can you elaborate on that?
Thank you
Hi Ayoub,
The WTS (Witness Traffic Seperation) is not a mandatory requirement of VxRail, it is an option component. WTS is designed for the scenario that you cannot stretch the subnet(vlan) across DC1, DC2 and witness site. In this scenario, you need to seperate you witness into different vmkernel interface to make the witness traffic routable from DC1/DC2 to witness site.
If you can use same VLAN acroess DC1,DC and witness site, it means yon can stretch subnet acroess sites. In this case, you don’t need to use WTS, you can stretch you vSAN VLAN acroess three sites and witness will happen in vSAN VLAN.
Hope this helps.
Hello Swanlate,
Thank you for the reply. What I’m having is more like a 2-node deployment (not a stretched cluster, but I guess they’re similar), and the Witness Host will be a VM hosted in the same vCenter as the 2-node VxRail cluster. I guess as you said there would be no need to add another VLAN for WTS, and we can just use the same VLAN used by the Witness.
Hi Ayoub,
Yes, If your environment can support one stretched vSAN VLAN across DC1, DC2 and witness site, you don’t need additional VLANs in DC1 and DC2 ESXi host for witness.