Wondering if anyone else is seeing the following. We have a HA pair of 501Es on 6.4.4. We started receiving reports today that our users were unable to connect with a -30 error to our SSL VPN. That seems to indicate no DHCP address is available. Our VPN profile is configured to allow only one connection at a time for each user and we are using a pool of ~250IPs for less than 150 users. DHCP timeout is 7200s. When we checked our gate, we saw multiple users were consuming two addresses (see attached image). Wondering if this is another Y2K22 type issue? We can manually go in and release the unused IP, but odd that this just started happening on first full work day of 2022. We’ve been on 6.4.4 for at least 6 months without this issue. Any thoughts or ideas are appreciated. Will post update once we determine root cause.
Happens to us as well. 600D running 6.2.6. We were told to upgrade to the latest three months ago. We’re doing it this quarter.
It’s a known bug, check with TAC in which version it is resolved for you.
6.4.8 fixes some cert issues that were in up to 6.4.7, so I would suggest 6.4.8… I have a number of devices on it and it’s been stable.
TAC responded with the following:
Thank you for your reply.
From the logs shared, I see missing indexes in command output “exe vpn sslvpn list” which is matching with bugID#745499 and is resolved in versions 6.4.9:1932, 7.0.2:0211
6.4.9 is tentatively scheduled to be released between Feb 08, 2022 - Feb 10, 2022; these dates are subject to change.
7.0.2 is available to download from support site.
We also have a work-around to delete the missing indexes using below command if the issue resurfaces or you may increase the SSLVPN IP pool.
#exe vpn sslvpn del-tunnel
For example:
Primary # execute vpn sslvpn list
SSL VPN Login Users:
Index User Group Auth Type Timeout Auth-Timeout From HTTP in/out HTTPS in/out
.
.
71 jsmith SSL_VPN_FULL 2(1) 7189 35768 73.225.218.123 0/0 0/0
73 jsmith SSL_VPN_FULL 2(1) 7191 35863 66.235.42.123 0/0 0/0
Here 72 index is missing and so we may delete missing index with command “exe vpn sslvpn del-tunnel 72”
Feel free to reach out to me if you have further questions.
So at least there is a CLI command to purge the orphaned IPs in a fairly easy fashion.
Will upgrade to 6.4.8 and monitor 6.4.9 stability before moving to that.
We upgraded to 6.4.4 to 6.4.6 and finally 6.4.8. Process was easy enough. We haven’t seen the VPN DHCP issue since. But we have a new problem now.
We had an inbound policy to our EMS for FortiClient Telemetry. We had the EMS cert configured for full inspection on the Fortigate so that we can detect encrypted inbound attacks to ports 10443 and 8013. After we upgraded to 6.4.8, our ISPs SDWAN routers (upstream of our HA pair of 501Es) started malfunctioning. We didn’t lose connectivity but we received a ton of spurious up/down alerts.
After a week and a half of trouble shooting we determined the upgrade broke the inbound inspection (was in flow mode) for FortiClient to EMS traffic. Before upgrade primary response was server-rst for ftnt_action events when viewing this policy. After the upgrade, this changed to client-rst and the number of events/sessions tripled. This caused a memory DoS on the ISP routers approx. every 24hrs.
So we removed the security policies (IPS and App Monitoring) and the session number/response returned to normal. Very frustrating that what should be critical and well developed product features like this can be so buggy. TAC responded that 6.4.9 is tentatively scheduled to be released between Apr 12, 2022 - Apr 14, 2022(subject to change). This has already been pushed back from Feb.
Consider IPsec VPN for faster performance.
It’s critical to keep your devices up to date. 644 has known vulns relating to SSL VPN. Please track psirts on at least a monthly basis and upgrade soonest otherwise you put your company and its data at risk.
Well, for starters, SSL-VPN doesn’t use DHCP, so I’d be curious to know where exactly you’re setting up DHCP timeout for it.
But there are bugs revolving around the IPs not being released from the Fortigate’s IP pool, leading to exhaustion.
I spoke to tac about this just last week and they said the fix should be in 6.4.9 when it comes out. In the meantime we set the dhcp scope to /16
We are in the process of doing that. Perhaps this was just an abnormally high number of connections this AM and the duplicates do prune over time, but not quick enough for todays larger than number of connection attempts? I just find it odd that we are only seeing it now after being on 6.4.4 for as long as we have been. Thanks for feedback so far. I appreciate it!
Thanks. Always good to hear what others are using.
Which ones. We are only allowing TLS 1.3 with strong crypto. I do watch these but wasn’t aware of any specifically impacting our build/configuration. We have a lot of Geo Blocks and hardening enabled with FAC MFA, so I think we are good at the moment, but I do agree that overall we have lingered too long on this build. We want to get on 7.0.X but were waiting for a few revs higher to miss as many bugs as possible. Perhaps waiting too long.
I can’t locate a PSIRT mailing list. Is RSS the only option? I do subscribe to the weekly threat report but that appears to cover external threats and not FortiOS issues. If you have a mailing list subscription URL would be appreciated.
Timeout is for the VPN session, not DHCP lease, sorry. It was interesting this AM, I see 45 sessions. Most are showing single address, but 5 of those are assigned two addresses. Such an odd issue. Maybe caused by a rapid connect/disconnect/reconnect?
Considered that. Will see how it goes today. Yesterday we ran a Splunk report that showed us the most recent IPs assigned to each session and then “deleted” those not indicated as being used from the GUI. Not the most efficient process but it worked.
Before deciding which version to upgrade to make sure to check Psirt advisory at Fortiguard.
https://www.fortiguard.com/psirt?product=FortiClientEMS
Security wise it seems to be advisable to upgrade to version 6.4.7 or 7.0.2, else you might end up with a version with critical vulnerabilities.
Yup. With record case numbers, betting your whole population is remote today, and being the first “real” work days of the year, everyone’s working at the same time this week… probably been just below the threshold up to this point.
As for update, highly recommend it. There’s some nasty vulns in 6.4.4, fixed in 6.4.7 & 6.4.8 respectively. There was another in Analyzer and Manager (CVSS of 10.0) back before v6.4.6 I think it was(?). Anyways, no time better than the present to plan a soon patching — now that you have the business’ attention.
Yes - we had the same issue, it appeared after almost a year. You are not affected all the time as it depends on the number of current users that are connected to the VPN and how many IPs have been assigned to them, however as soon as the pool is exhausted you will receive the error message on FortiClient. I haven’t found any logs or commands to check how many IPs are assigned/left from the pool or in general a workaround - your only option is to start cleaning before the issue appears or upgrade the firewall to permanently resolve this.
Did some checking. Glad we aren’t using FortiWeb. Yikes. Yeah, this one is bad. Thanks for reminder.