Recently had to try and explain why a FTPS configuration was not working over an otherwise open private WAN. Issue was the two stateful firewalls at each end. Since writing this post I have shown it / e-mailed it to three other people to try and help them understand their own encrypted FTP issues.
So because it seems helpful I’ll add it here (IP Guru’s will get annoyed by the very simplistic language used, but it could save you time if you get asked in future!)
FTP connections require more than one channel of information, there is the control channel, TCP Port 21 and then single / multiple data transfer channels for PUT/GET/DIR commands for sending/receiving/listing data etc. The TCP or UDP ports that the data transfer channels use are negotiated between the server and client via the control channel once the user has logged in successfully.
A modern firewall works by looking at outbound connections from an internal network and ‘tracking’ them, that is, it keeps a record of internal host A trying to contact internet webserver W1 on port 21, then only allows traffic from the internet if it is from Webserver W1 on port 21 sending data to internal host A. In this way, any communication channel that has not originated inside the customers network will not be allowed into the network from the internet (or <NETWORK NAME REMOVED> in this case). Clearly this posed a problem for the FTP protocol, as the original FTP specification mandated that the FTP SERVER would decide on the data transfer channel ports and try connecting BACK to the client on these new ports, of course that did not work, as the firewall at the client end has no record of the client connecting outbound on these data transfer channel ports and so drops the connection.
To get around this issue (when security became important on the internet and people started deploying firewalls) the FTP standards created a new ‘PASSIVE’ mode, this mode just allows the data transfer channels to be created FROM the client to the server, allowing the firewall to see the outbound connection and therefore allow return data traffic from the server. This works fine, unless BOTH server AND client are behind firewalls, at this point, neither ACTIVE or PASSIVE mode solves the problem, there will always be one firewall that drops the connection because it hasn’t seen the computer behind it initiate the connection first.
To solve this, most firewalls (including ours here and the one at <SITE NAME REMOVED>) have ‘FTP Helpers’ built in, these pieces of code inspect the data passed between server and client in the FTP control channel (Port 21) and therefore see the negotiation between the systems over what data channels to use, because they see which ports the systems are getting ready to use for FTP data channels, the firewalls can dynamically open the needed ports, expectantly waiting for the connection and then close the ports again when the control channel disconnects (because if there is no control channel there is no user).
This works perfectly, however if you need ENCRYPTION on your FTP transfer, due to the nature of the data you are transferring, then, both control and data channels are encrypted from client to server and back again with TLS or SSL encryption.The firewall becomes blind to the data it needs to ‘help’ the FTP connection, as the control channel appears to the firewall as nothing but encrypted jibberish, therefore the FTP helper in the firewalls cannot work out what ports are being negotiated.
This is why you can log in successfully, but anything that requires listing, sending or retrieving data fails, as the data channel cannot be set up because the firewalls are not expecting the connections. The only resolution to this is FTP Clear Control Channel mode, this (as the name suggests) only uses encryption for the transfer channels and leaves the control channel in plain text so that firewalls along the path can deal with the connection correctly.
It is support for FTP Clear Control Channel mode that I wanted to log onto the server and check for, but after some reading into FileZilla server, it appears this is not supported.
It is for this reason that both our site AND <SITE NAME REMOVED> are to blame, purely because they both operate firewalls.
This is not an issue that can be resolved without doing one of the following:
– Running a FTP server that supports CCC
– Removing one of the Firewalls
– Removing encryption
– Permanently opening up a range of ports from/to both machines and then configuring both server and client to always use these ports for data channels. This would also mean only that pair of systems specifically configured for this server could successfully use FTP in this manner.