Linux systems unable to reach out to internet behind pfsense while Windows work fine.
I am trying to set up my #homelab for #forensics and I have encountered a very weird issue. I am running an internal pfsense #firewall for my environment. On this firewall there are 3 interfaces: WAN and two Isolated Labs.
On the “WAN” interface, both #linux and windows systems are able to access the internet without any problems.
But on the other two lans - named Infrastructure and Lab - the Linux systems are not able to connect to the internet after booting, while the systems are able to communicate with each other. In order to get them working I need to go the respective interface and make any change there and save the page, practically refreshing the settings on the interface. After this all currently online linux systems on that interface are working as expected and connectivity is restored. As you can imagine this is a huge pain, especially when I am testing things and I need to turn on various systems at different times. If i put the systems on the WAN interface they face no issue at all, so what could be the solution here for fixing connectivity for linux systems on the other interfaces?
edit: it issue is both on pfsense 2.7.2 and 2.8.0
#dfir #proxmox #dfir #homelab #firewall #pfsense #networking #network
Check your route table and arp cache (for gw). Are you using dhcp?
@Gobo gw? I have tried with and without dhcp and the issue persists
Gateway. Does the system properly know how to get outside of its subnet.
@Gobo yes the gateway is configured properly. If ti wasnt the system would not have internet after a firewall refresh
Then I’d say get a pcap from the pfsense interface and see what’s going on. Is the systems Mac still in the fw arp table? . If you think it’s a system fw problem, try disabling the local fw and see if things work
What do you mean “go to the respective interface and make any change”? In the firewall or client? What changes when you do this, IP address or routing table or gateway or what?
Its hard to determine exactly where the issue is coming from, so some more information about the setup would be helpful.
- Is pfsense on a physical machine with 3 physical network interfaces, or is it virtualized?
- If you bring down the network interface on the linux box after you do the trick to make the internet work, then bring it back up (as opposed to fully restarting the machine), does the same issue still happen?
- You mention that you have windows and linux connected to the WAN interface? That doesn’t sound right. In a typical set up, you would have WAN connected to your modem, and LAN connected to your devices. You might want to double check your firewall rules and routing table in pfsense to make sure everything is configured correctly.
In my experience, networking is very prone to small configuration errors, and setting up a homelab is going to force you to learn some great networking skills one way or another :)
- pfsense is virtualized on a Proxmox VM
- If i bring down the network interface and back up it loses connectivity once more
- The firewall is for and internal network so the WAN is my home network. I have an ubuntu server acting as a NAS and another ubuntu server for Jellyfin. Both servers are on the same proxmox instance, just on the WAN interfaces on the pfSense so I can access them from all the computers on the local network.
ok, so to recap just to make sure I have your setup correctly: You Have a Proxmox node with 3 VMs on it: Pfsense, an Ubuntu VM, and another Ubuntu VM. The Proxmox Node has a WAN network (likely a Linux Bridge type interface called vmbr0 or something similar), and a LAN network (another bridge, but with no associated physical interface on the Proxmox host). The last step can differ depending on how exactly you set it up, but the basic idea of having two “networks” is the same. EDIT: You will have two LAN networks as per your original post Your pfsense VM has three Network Devices under the hardware tab, one connected to your WAN network, and the other two connected to your LAN networks.
In Pfsense, you have the appropriate interfaces configured and firewall rules allowing devices on the LAN interface to connect out of the WAN interface (this step in particular has a lot of moving parts)
On each of your Ubuntu VMs, there is just a single Network Device under their hardware tabs, connected to the LAN network. (although right now you say they are connected to the WAN network so you can connect to them from other computers on your home network). Once this is set up correctly, move them back to the LAN network, and maybe set up a rule that allows connections from a single device on your home network. If allowing any device on your home network to connect to your homelab is what you want, then it doesn’t make too much sense having a firewall if you aren’t going to use it ;).
Does this match with your current setup?
@Chewt
The node has others vms as well, some of them are windows vms but they are not discussed as they do not face any problems connecting to the internet no matter what interface they are connected on. The Linux VMs that are on the WAN network which is the one that has the linux bridge that is connected to a port and those systems do not have any connectivity issues.The other two lans also have a linux bridge that is not bound to any port and the Linux systems need the interface refresh in order to access the internet. Windows systems on those networks are connected to the internet with no issues.
yes the pfsense hardware is as you describe it. (picture attached)
For the last paragraph, I need to have some devices that will be isolated from my WAN (home network) because I want to create a cybersecuirty lab and I do not want anything to “escape” to my home devices.
Plus this teaches me how to manage a network with proper segmentation etc.
ok, if you’re fine with doing a little more troubleshooting, I have a few more things to try.
You said before that the systems are able to communicate with each other on the LAN network, just not connect out via WAN. Does this include the pfsense box? That is to say are you able to
ping 10.10.10.1(assuming that’s the IP of the Infra interface on pfsense) from one of you Ubuntu VM’s when it is on the Infra network? Also what is the output of both commandsip -br aandip route? What I’m am curious about here is maybe the default gateway is not being correctly set when new devices connect to the pfsense. If this ends up being the case, it might be an issue with DHCP (or if you aren’t using DHCP then just the configuration on each Ubuntu VM). I suspect that Windows may be doing some fancy things to figure out on its own some network settings without them being explicitly set.@Chewt
yes they can still ping my firewall while not connected to the internet.
Here are two screenshots from one of the ubuntu vms on the home network and one from the kali vm on the infra network
Well that doesn’t really indicate any problems with the default gateway. My last bit of troubleshooting advice is to traceroute and see if packets get stuck at the router. And check the firewall logs in pfsense to see if you can find anything there.
Hopefully it ends up being a relatively painless problem to fix!
The way your describing this isn’t quite clear. I think you have something crossed. There are a series of videos by Lawrence Systems on YouTube where he walks through basic to complicated PFsense configurations. I think it’s worth watching. What your trying to do is setup two vlans I think? They probably don’t have access to the WAN which you need to allow as this is not the default.
It seems complicated but making firewall rules makes sense once you get the hang of it. The best part of home labbing is learning though it usually requires breaking some stuff in the meantime. All part of the journey. You’ll get there, have fun.
