VxRail vSAN Stretched Cluster Witness Traffic Separation Configuration

When design the stretched VxRail vSAN cluster, there is an option to utilize Witness Traffic Separation (WTS) to separate the vSAN witness traffic from vSAN vmkernel interfaces. This post is to discuss what is the WTS and how to deploy it.

What is vSAN Witness Traffic Separation (WTS)?

Witness Traffic Separation (WTS) is officially introduced from vSAN 6.5. The WTS is designed to separate VMware vSAN data traffic from witness traffic in two-node vSAN cluster and the stretched cluster.

Without WTS, for vSAN stretched cluster, the vSAN VLAN need to be stretched across data sites and witness site. However, in some scenarios, the VLAN could not be extended outside the Core DC. The WTS is useful when the vSAN VLAN cannot be stretched to the witness site, also this could allow the witness appliance to be deployed in public cloud, such as AWS etc.

It is noted that, when deploying the witness appliance in public cloud, it is important to check the network requirement including latency, firewall requirement and etc.

In VxRail stretched cluster solution, the WTS can be utilized by introducing an additional VMKernel interface in each ESXi data node to carry the witness traffic.

For example, there are two data centers (DC1 and DC2) with four VxRail nodes in each site, by default, each VxRail ESXi have five VMKernel interfaces pre-configured by VxRail as below:

  • Vmk0     – External mgmt.              -Stretched VLAN
  • Vmk1     – iDRAC network               -No uplink
  • Vmk2     – Internal mgmt.               -Stretched VLAN
  • Vmk3     – VSAN Data                       -Stretched VLAN
  • Vmk4     – vMotion                           -Stretched VLAN

To configure WTS, the additional vmk5 need to be added and configured as below:

  • DC1 ESXi    – vmk5        -Witness Traffic                -DC1 Only WTS VLAN(WTS-VLAN1)
  • DC2 ESXi    – vmk5        -Witness Traffic                -DC2 Only WTS VLAN(WTS-VLAN2)

The Witness appliance need to be deployed in Witness site with a vSAN kernel interface configured as below:

  • Witness ESXi       – vmk0       -Mgmt. Traffic
  • Witness ESXi       – vmk1       -vSAN Traffic          -vSAN VLAN (WITNESS-VLAN)

The vmk5 from each vSAN ESXi data node should be able to communicate with vmk1 in witness server in bi-directions via L3 routing.

The high-level configuration steps for WTS.

Step 1: Build a standard VxRail vSAN cluster via VxRail Wizard. The steps to build a standard cluster and stretched cluster is same in VxRail wizard.

Step 2: Deploy the Witness sever or Witness Appliance in witness site with two vmkernel interfaces. (vmk0 for management and vmk1 for vSAN traffic type.)

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vsan-planning.doc/GUID-05C1737A-5FBA-4AEE-BDB8-3BF5DE569E0A.html

Step 3: In VxRail vSAN data nodes in each site, add vmkernel interface in each node for WTS as below:

  • In each ESXi data node, add additional VMKernel interface vmk5 in VLAN WTS-VLAN1(DC1) or WTS-VLAN2(DC2) with no traffic type configured.
  • In each ESXi data node, use below command to configure and check the new VMkernel interface with traffic type “Witness”
esxcli vsan network ip add -i vmk5 -T=witness
esxcli vsan network list

Step 4: In Witness appliance, Check there is VMKernel interface in VLAN WTS-VLAN3 and IP with traffic type “vSAN” configured. Login Witness appliance and use below command to check.

esxcli vsan network list

Step 5: In vCenter, configure vSAN stretched cluster in vCenter vSAN “Fault Domain” configuration page.

Step 6: Add additional Isolation Response Addresses for vSAN cluster High Availability. Choose one or two IP addresses in the stretched vSAN VLAN as the isolation addresses, normally the vSAN VLAN gateway could be used.

https://kb.vmware.com/s/article/1002117

For now, you should have the VxRail Stretched Cluster with WTS configured. Enjoy!

10 thoughts on “VxRail vSAN Stretched Cluster Witness Traffic Separation Configuration

  1. mas

    Hi thank you for your blog, in the case of the L3 vSAN VLAN between the datacenters, is it possible to have the WTS between the datacenters and witness?

    Like

    Reply
    1. swanlate Post author

      Hi Mas,

      I didn’t see any issue to configure WTS even you use L3 for vSAN traffic.
      In ESXi, the new WTS vmkernel interface traffic type will be tagged as “Witness” and ESXi will control the route to use WTS vmkernel interface for “WTS” traffic.

      Just make sure the Witness Server vSAN VLAN is a different VLAN than others you are using in your solution. Otherwise you will have routing issue.

      Like

      Reply
  2. Alexandre

    Hello,
    I have just a little question.
    If I understood correctly, we must have 3 other news vlan for that ?
    One for the witness appliance vmk1(named witness-vlan), one for the nodes on DC1 (vmk5 named wts-vlan1) and one for the nodes on DC2 (vmk5 named wts-vlan2) ?

    Like

    Reply
    1. swanlate Post author

      The quick answer is YES.

      The purpose of WTS is to use separate vmkernel interface for witness traffic, therefore, you need a separate VLAN at each DC for witness traffic. It can be existing VLAN in your org but should be a different from the other VSAN related VLANs used in VxRail.

      Do not suggest to use stretched VLAN for WTS for both DC, there might be a gateway problem and might impact the VSAN witness.

      For “witness-vlan”, it is always suggested that the witness should be hosted in a remote physical location. Therefore, in majority cases, there is a different VLAN for “witness-vlan” in witness site.

      Like

      Reply
      1. Alex

        thank you for your reply.
        You say :
        “Do not suggest to use stretched VLAN for WTS for both DC, there might be a gateway problem and might impact the VSAN witness.”
        But if I use 3 news Vlan I must use gateway.
        You don’t think it s better use only one new Vlan (in layer 2) ?

        Like

      2. swanlate Post author

        Technically, you can use a single stretched VLAN across DC1, DC2 and Witness sites for WTS traffic, but this is not practical in most customer environments. As witness site is always a remote site or even living in public cloud.

        To a certain degree, the WTS is designed to allow the witness traffic to go through a separate gateway to go outside of DC. If you would like to use Layer 2, there is no point to configure WTS as the witness traffic by default will use VSAN VLAN, which is layer 2 as well.

        However, if you would like to configure WTS and use a new layer2 stretched VLAN across three sites, I didn’t see any technical issue for that.

        Like

  3. Ayoub Houbban

    Hello,
    Thank you so much for this post, very informative.
    I’d like to ask a question in regards to WTS VLAN, is it mandatory to create a new VLAN for node-WTS trafic? Can we juste use the same VLAN used for Witness vSAN trafic ? If not, and I see you mentioned routing issues, can you elaborate on that?

    Thank you

    Like

    Reply
    1. swanlate Post author

      Hi Ayoub,

      The WTS (Witness Traffic Seperation) is not a mandatory requirement of VxRail, it is an option component. WTS is designed for the scenario that you cannot stretch the subnet(vlan) across DC1, DC2 and witness site. In this scenario, you need to seperate you witness into different vmkernel interface to make the witness traffic routable from DC1/DC2 to witness site.

      If you can use same VLAN acroess DC1,DC and witness site, it means yon can stretch subnet acroess sites. In this case, you don’t need to use WTS, you can stretch you vSAN VLAN acroess three sites and witness will happen in vSAN VLAN.

      Hope this helps.

      Like

      Reply
      1. Ayoub Houbban

        Hello Swanlate,

        Thank you for the reply. What I’m having is more like a 2-node deployment (not a stretched cluster, but I guess they’re similar), and the Witness Host will be a VM hosted in the same vCenter as the 2-node VxRail cluster. I guess as you said there would be no need to add another VLAN for WTS, and we can just use the same VLAN used by the Witness.

        Like

      2. swanlate Post author

        Hi Ayoub,

        Yes, If your environment can support one stretched vSAN VLAN across DC1, DC2 and witness site, you don’t need additional VLANs in DC1 and DC2 ESXi host for witness.

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s