Skip to content

Hosted Control Plane and tenant networking

Challenge: running a hosted cluster in a different tenant network segment or VLAN without wide-open access from the tenant segment to the management segment.

Hub cluster must not route into tenant networks

The hub cluster must not have arbitrary Layer-3 addressing or routing into the tenant network segment. The hub may only attach hosted-cluster workloads—for example, KubeVirt VMs on a UDN localnet—to that segment.

Do not use MetalLB on the hub to expose services into a tenant network. That pattern typically requires:

Together, those changes let any workload on the hub reach tenant networks—not only the hosted cluster you intend to isolate.

This guide uses external load balancers in the tenant segment and a dedicated ingress controller shard on the hub instead.

An hosted cluster can devide into two parts: control plane and data plan aka worker nodes. For there parts there different technics to place it into a tenant network:

Exposing hosted control plane into tenant network

... it is fairly hard. The following components must be reachable from workers and clients in/from tenant network and beyond:

  • API Server
  • OAuth
  • Konnectivity
  • Ignition

Here is a summary of common publishing options for these components:

Component/Service Exposing strategy (servicePublishingStrategy) Kubernetes Service type LoadBalancer Route (OpenShift router)
API Server
  • LoadBalancer (recommended; Kubernetes LoadBalancer service)
  • NodePort* (not for production)
    		•  ✅ ❌
    OAuth
  • Route (default)
  • NodePort* (not for production)
    		•  ❌ ✅
    Konnectivity
  • Route (default)
  • LoadBalancer (Kubernetes LoadBalancer service)
  • NodePort* (not for production)
    		•  ✅ ✅
    Ignition
  • Route (default)
  • NodePort* (not for production)
    		•  ❌ ✅

    For this proof of concept, endpoints are exposed as follows:

    • API Server: LoadBalancer (fronted by external api-lb in the tenant segment; see below)
    • OAuth, Konnectivity, Ignition: Route via a dedicated ingress controller shard on the hub, fronted by external ingress-shared-lb with VIPs/DNS in the tenant segment

    Exposing components via a dedicated router shard

    Use a dedicated OpenShift Ingress Controller shard on the hub so only the hosted-cluster control-plane Routes are served by that shard. Tenant clients resolve OAuth, Konnectivity, and Ignition hostnames to ingress-shared-lb, which forwards to the shard’s NodePorts on the management network.

    Place an external load balancer in front of that shard (for example F5 BIG-IP or NetScaler) that can reach the hub’s management network and present stable tenant-facing VIPs or addresses.

    Exposing hosted worker nodes into netant network

    Worker nodes (VM's) of the hosted cluster are straightforward: attach them to the tenant network segment (DHCP or equivalent addressing is required).

    Proof of concept environment overview

    Router between Mgmt and Tenant-A

    VyOS acts as router and firewall between the management and the Tenant-A network. Restrict lateral traffic between the two segments (no full mesh); allow only what you need (for example DNS to resolvers, default route or NAT for internet egress). Hosted-cluster control-plane traffic from tenant nodes should flow to the external load balancer VIPs in the tenant segment (not directly into arbitrary management subnets).

    VyOS config commands 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
    set firewall group address-group ALLOWED-IPS address '10.32.96.1'
set firewall group address-group ALLOWED-IPS address '10.32.96.31'
set firewall group address-group ALLOWED-IPS address '10.32.111.254'
set firewall ipv4 forward filter rule 49 action 'accept'
set firewall ipv4 forward filter rule 49 description 'Allow IPs'
set firewall ipv4 forward filter rule 49 destination group address-group 'ALLOWED-IPS'
set firewall ipv4 forward filter rule 50 action 'drop'
set firewall ipv4 forward filter rule 50 description 'Drop enire coe lab'
set firewall ipv4 forward filter rule 50 destination address '10.32.96.0/20'

set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth1 address '192.168.203.1/24'

set nat source rule 100 outbound-interface name 'eth0'
set nat source rule 100 source address '192.168.203.0/24'
set nat source rule 100 translation address 'masquerade'
set service dhcp-server listen-interface 'eth1'
set service dhcp-server shared-network-name coe-2003 authoritative
set service dhcp-server shared-network-name coe-2003 subnet 192.168.203.0/24 option default-router '192.168.203.1'
set service dhcp-server shared-network-name coe-2003 subnet 192.168.203.0/24 option name-server '10.32.96.1'
set service dhcp-server shared-network-name coe-2003 subnet 192.168.203.0/24 range 1 start '192.168.203.100'
set service dhcp-server shared-network-name coe-2003 subnet 192.168.203.0/24 range 1 stop '192.168.203.200'
set service dhcp-server shared-network-name coe-2003 subnet 192.168.203.0/24 subnet-id '1'
set service ssh
set system host-name 'router-2003'
set system name-server '10.32.96.1'
set system name-server '10.32.96.31'

    Deployment sequence (reference)

    Three external load balancers appear in this write-up; keep their roles distinct:

    Name Role
    ingress-shared-lb Tenant-facing VIPs for OAuth, Konnectivity, Ignition Routes on the hub ingress shard
    api-lb Tenant-facing VIP for the hosted cluster API (APIServer publishing)
    ingress-lb Tenant-facing VIP for hosted cluster application Routes (*.apps…)

    Suggested order:

    1. Hub ingress shard + ingress-shared-lb + DNS for the three control-plane hostnames: OAuth, Konnectivity, and Ignition
    2. Apply HostedCluster and NodePool.
    3. Deploy external load balancer for the Hosted-Cluster API api-lb + API DNS. Based on the NodePorts for the api kubernetes Service, located in hub cluster
    4. Deploy external load balancer for the Hosted-Cluster Ingress ingress-lb + wildcard apps DNS. Based on the NodePorts of the ingress kubernetes service, located in hosted cluster.

    Hub ingress shard + ingress-shared-lb

    Ingress Controller 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
    apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: tenant-a
  namespace: openshift-ingress-operator
spec:
  domain: tenant-a.coe.muc.redhat.com

  endpointPublishingStrategy:
    type: NodePortService
  namespaceSelector:
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values:
          - ingress-test
          - clusters-tenant-a

    1
2
3
    % oc get svc -n openshift-ingress router-nodeport-tenant-a
NAME                       TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                                     AGE
router-nodeport-tenant-a   NodePort   172.30.141.209   <none>        80:32460/TCP,443:32488/TCP,1936:32095/TCP   106s

    The ingress shard load balancer is an RHEL 9 host running HAProxy (external load balancer ingress-shared-lb).

    • Install HAProxy: dnf install haproxy
    • Configure SELinux: setsebool -P haproxy_connect_any 1
    • Apply the example haproxy configuration (update ports to match your NodePort service)
    • Enable and start HAProxy: systemctl enable --now haproxy
    HAProxy config 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
    global
  log         127.0.0.1 local2
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  daemon
defaults
  mode                    http
  log                     global
  option                  dontlognull
  option http-server-close
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000

listen ingress-router-443
  bind *:443
  mode tcp
  balance source
  server ucs-blade-server-5 10.32.96.105:32488 check inter 1s
  server ucs-blade-server-6 10.32.96.106:32488 check inter 1s        
  server ucs-blade-server-7 10.32.96.107:32488 check inter 1s
  server ucs-blade-server-8 10.32.96.108:32488 check inter 1s

listen ingress-router-80
  bind *:80
  mode tcp
  balance source
  server ucs-blade-server-5 10.32.96.105:32460 check inter 1s
  server ucs-blade-server-6 10.32.96.106:32460 check inter 1s        
  server ucs-blade-server-7 10.32.96.107:32460 check inter 1s
  server ucs-blade-server-8 10.32.96.108:32460 check inter 1s

    Add DNS records

    1
2
3
    konnectivity.tenant-a.coe.muc.redhat.com.       IN A 192.168.203.111
oauth.tenant-a.coe.muc.redhat.com.              IN A 192.168.203.111
ignition.tenant-a.coe.muc.redhat.com.           IN A 192.168.203.111

    Apply HostedCluster and NodePool

    HostedCluster
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
    apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  name: 'tenant-a'
  namespace: 'clusters'
  labels:
    "cluster.open-cluster-management.io/clusterset": 'default'
spec:
  configuration:
    ingress:
      appsDomain: apps.tenant-a.coe.muc.redhat.com # (1)
      domain: ''
      loadBalancer:
        platform:
          type: ''
  channel: fast-4.21
  etcd:
    managed:
      storage:
        persistentVolume:
          size: 8Gi
        type: PersistentVolume
    managementType: Managed
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.21.11-multi
  pullSecret:
    name: pullsecret-cluster-tenant-a
  sshKey:
    name: sshkey-cluster-tenant-a
  networking:
    clusterNetwork:
      - cidr: 10.132.0.0/14
    serviceNetwork:
      - cidr: 172.31.0.0/16
    networkType: OVNKubernetes
  controllerAvailabilityPolicy: SingleReplica
  infrastructureAvailabilityPolicy: SingleReplica
  platform:
    type: KubeVirt
    kubevirt:
      baseDomainPassthrough: false
  infraID: 'tenant-a'
  services:
    - service: APIServer
      servicePublishingStrategy:
        type: LoadBalancer
        loadBalancer:
          hostname: api.tenant-a.coe.muc.redhat.com  # (2)
    - service: OAuthServer
      servicePublishingStrategy:
        type: Route
        route:
          hostname: oauth.tenant-a.coe.muc.redhat.com  # (3)
    - service: OIDC
      servicePublishingStrategy:
        type: Route
    - service: Konnectivity
      servicePublishingStrategy:
        type: Route
        route:
          hostname: konnectivity.tenant-a.coe.muc.redhat.com  # (4)
    - service: Ignition
      servicePublishingStrategy:
        type: Route
        route:
          hostname: ignition.tenant-a.coe.muc.redhat.com  # (5)
    1. appsDomain: resolve names under apps.tenant-a.coe.muc.redhat.com to ingress-lb (hosted cluster ingress), not the hub shard.
    2. API server loadBalancer.hostname: resolve to api-lb, which forwards to the APIServer publishing target on the hub.
    3. OAuth route.hostname: resolve to ingress-shared-lb (hub dedicated shard).
    4. Konnectivity route.hostname: resolve to ingress-shared-lb.
    5. Ignition route.hostname: resolve to ingress-shared-lb.
    NodePool
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
    apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
  name: 'tenant-a'
  namespace: 'clusters'
spec:
  arch: amd64
  clusterName: 'tenant-a'
  replicas: 2
  management:
    autoRepair: false
    upgradeType: Replace
  platform:
    type: KubeVirt
    kubevirt:
      compute:
        cores: 2
        memory: 8Gi
      rootVolume:
        type: Persistent
        persistent:
          size: 32Gi
      additionalNetworks:
      - name: default/cudn-localnet1-2003 # (1)
      attachDefaultNetwork: false
  release:
    image: quay.io/openshift-release-dev/ocp-release:4.21.11-multi
    1. Attach NodePool VMs to the tenant segment using a user-defined network (UDN) localnet attachment (default/cudn-localnet1-2003 in this lab).

    Deploy external load balancer for the Hosted-Cluster API (api-lb)

    Use an RHEL 9 virtual machine with HAProxy.

    • Install HAProxy: dnf install haproxy
    • Configure SELinux: setsebool -P haproxy_connect_any 1
    • Apply the example haproxy configuration (update ports to match your environment)
    • Enable and start HAProxy: systemctl enable --now haproxy
    HAProxy config 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
    global
  log         127.0.0.1 local2
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  daemon
defaults
  mode                    http
  log                     global
  option                  dontlognull
  option http-server-close
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000

listen api
  bind *:6443
  mode tcp
  balance source
  server ucs-blade-server-5 10.32.96.105:30918 check inter 1s
  server ucs-blade-server-6 10.32.96.106:30918 check inter 1s        
  server ucs-blade-server-7 10.32.96.107:30918 check inter 1s
  server ucs-blade-server-8 10.32.96.108:30918 check inter 1s

    Add DNS record:

    1
    api.tenant-a.coe.muc.redhat.com.       IN A 192.168.203.<IP of VM>

    Deploy external load balancer for Hosted-Cluster ingress (ingress-lb)

    Use an RHEL 9 virtual machine with HAProxy.

    • Install HAProxy: dnf install haproxy
    • Configure SELinux: setsebool -P haproxy_connect_any 1
    • Apply the example haproxy configuration (update ports to match your environment)
    • Enable and start HAProxy: systemctl enable --now haproxy
    HAProxy config 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
    global
  log         127.0.0.1 local2
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  daemon
defaults
  mode                    http
  log                     global
  option                  dontlognull
  option http-server-close
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000

listen ingress-router-443
  bind *:443
  mode tcp
  balance source
  server tenant-a-gngj5-mfwp6 192.168.203.101:30190 check inter 1s
  server tenant-a-gngj5-rrbmv 192.168.203.102:30190 check inter 1s        

listen ingress-router-80
  bind *:80
  mode tcp
  balance source
  server tenant-a-gngj5-mfwp6 192.168.203.101:30282 check inter 1s
  server tenant-a-gngj5-rrbmv 192.168.203.102:30282 check inter 1s

    Add DNS record:

    1
    *.apps.tenant-a.coe.muc.redhat.com.       IN A 192.168.203.<IP of VM>

    Open topics

    • Disable or constrain cloud provider integration so that Kubernetes LoadBalancer Service requests for the hosted cluster are not satisfied by the hub cluster cloud integration unless that is intentional.
    • WebUI bug: ACM shows https://console-openshift-console.apps.tenant-a.apps.ocp5.stormshift.coe.muc.redhat.com/ for the console, but the URL should be https://console-openshift-console.apps.tenant-a.coe.muc.redhat.com/.
    • Add custom endpoint publishing strategy
    • Find a solution for the NodePort chicken-and-egg problem of the external API load balancer

    Verions

    Tested with:

    Component Version
    OpenShift v4.21.9
    OpenShift Virt v4.21.0
    MCE v2.11.1
    2026-05-26 2026-05-12 Contributors: Robert Bohne