Hello,

I have a little homelab that contains a 3 node k3s cluster which im pretty happy about but i got some questions regarding ingress.

Right now i use nginx as ingress controller and i have the IP of one of the nodes defined under externalIPs. All the nodes are behind the router my ISP gave me so this is nothing special, in this router i configured it to forward port 443 to port 443 of that ip. This all works as excpected im able to access the ingress resources that i want.

But i wanna make some improvements to this setup and im honestly not really sure how i could implement this.

  1. Highly available ingress. When the node which contains the IP of the ingress controller goes down im unable to reach my clusters ingress since my router cant forward the traffic. Whats the best way to configure all 3 nodes to be able to receive ingress traffic? (If needed im able to put it behind something like openwrt or opnsense but not sure if this is needed)
  2. Some ingres resources i only want to expose on my local network. I read online that i can use nginx.ingress.kubernetes.io/whitelist-source-range: 192.168.0.0/24 but this doesn’t work i think because since the ingress doesn’t receive the clients actual ip rather it receives an internal k3s ip. Or is their another way to only allow certain ips to access an ingress resource?

Could someone point my in the right direction for these improvements i wanna make? If you need more information you can always ask!

Thanks for your time and have a great day!

  • boblin@sh.itjust.worksEnglish
    2·
    10 days ago

    To get nginx ingress to use the external clients IP, you can configure the ingress controllers traffic policy. Using the helm chart, I used these values:

    controller:
          service:
            # this has a bunch of downsides, but allows source-ip based access white/deny listing.
            externalTrafficPolicy: Local

    For the ingress IP, I configured metal-lb to receive traffic on a static IP (using IP4AddressPool and L2Advertisement CRDs from metal-lb), which is then used for the port forwarding. I’ve never tested it because I only have a single worker node, but I expect the metal-lb controller will continue receiving traffic to that same static IP if a node goes down.

    • Hercules@lemmy.worldOPEnglish
      2·
      10 days ago

      And does this work for ingress? I searched a little bit around but as far as i understand metallb is for k8s services?

      • boblin@sh.itjust.worksEnglish
        1·
        10 days ago

        Ingress controllers usually use the standard k8s services. In fact metal-lb allows workloads (like the nginx ingress controller) in the cluster to use services of type LoadBalancer, which is the default configuration. This results in an actual IP being made available to your ingress controller.