Update: July 26th 2011 Putty 0.61 has been released. My patch is incorporated in this release, and is listed as part of the release note.
From time to time, I would create a SOCKS proxy with SSH to secure my FTP or HTTP transfers. PuTTY is by far my favorite SSH client in Windows because it is free, lightweight, and supports advanced feature such as SSH connection over HTTP proxy.
Last week, when I was sending some data over to a FTP server through a PuTTY’s SOCKS proxy, I noticed something strange. Filezilla (FTP client ) indicated that all file transfers are 100% completed, yet the uploaded files were almost always incomplete. I retried the same upload numerous times, but would always result in almost completed file.
Here's the original file size and uploaded file size. Notice that they are not quite the same.
After trying different software combinations, I eventually figured out that the bug is originated from PuTTY. And some internet research shows that the bug has been around since 2003!
The Why and the How
I was bored, so I decided to download the source code of the PuTTY nightly snapshot and started a debug session. (Yes, real programmers debug FOSS after dinner for entertainment.)
Since I was completely new to the source code and SSH2 protocol, the bug certainly wasn’t obvious. I spent most of the first night setting up breakpoints and tracepoints to understand the logic path.
By the second night, I got a good handle on the way the sockets are created, and started reading the Connection Layer (RFC 4254) of the SSH2 protocol.
By the third night, I found the reason for the port-forwarding data corruption.
When PuTTY forwards a port, it sets up a TCP server at the local endpoint to receive the traffic, and then it sets up a TCP client to forward to traffic to the SSH server.
In terms of implementation, this is a three-step process for PuTTY.
1. PuTTY receives data from the local endpoint. In my case, it is Filezilla.
2. PuTTY buffers up the data in its own queue – the buffer chain.
3. Then PuTTY will send the data to the SSH server, with respect to the SSH2 sliding window protocol. The SSH2 window size is small – only 16 kB (no idea why). Therefore, PuTTY would often fill the window quickly, and buffer up the data in the buffer chain. The data will be only sent as soon as the SSH server opens up the window through a SSH2_MSG_CHANNEL_WINDOW_ADJUST message.
Unfortunately, step three has a bug. When the local endpoint closes its connection to PuTTY, PuTTY would immediately close its channel to the SSH server without fully sending all its data. So the last set of channel adjustment message are ignored, and led to data being stuck in the buffer chain.
Because the channel is closed prematurely, the last set of window adjustment messages are ignored.
I modified PuTTY’s SSH2 channel closing sequence to keep the channel open until the buffer chain is completely emptied. For debugging, I also added some events in PuTTY’s event log to indicate the number of bytes PuTTY’s trying to send before closing the channel.
PuTTY event log to record the shut down scenario.
This modification works like a charm for me. I tested hundred of GB of small file uploads, and not a single file was corrupted.
I submitted the patch to PuTTY development team, and it was accepted by Simon Tatham – the development lead. The patch was checked into the PuTTY’s trunk directory as rev. 8971, and is built into the PuTTY nightly snapshot.
svn://svn.tartarus.org/sgt/putty rev. 8791
You can download my patch here if you are interested in the details.
PuTTY’s source code is clean and self explanatory. It is written by good programmers.
Now I have something to brag about in front of my nerdy co-workers and friends.
Read Full Post »