I was given a credit card reader/device and was asked to determine if I could modify a custom-made PoS (point of sale) app, to interact with the CC reader. It sounded like an easy task. I was also told that all of the users were on Citrix / Terminal Server Client (RDP) sessions and the CC reader would be connected to the local terminal (next to the user) via a USB cable (COM3/COM4). Interesting, but still, no biggie.
I downloaded the SDK from CC reader’s manufacturer and had things working on my PC pretty quickly. The device plugged & played on COM4. The System.IO.Ports.SerialPort library in .NET worked nicely. Lookin’ good.
I was able to borrow a Windows 2003 Server from another team in my department. I uploaded my demo software, set up my RDP connection to route the serial port (see pictures:  RDP “more settings”  RDP Serial Ports).
The server recognized the serial port (COM4). I was able to run my demo and connect from my remote session (running on the server) and make calls to the Ingenico over Serial (connected locally). The COM port traffic routed pretty nicely. All was good.
Last detail: I did my initial tests on Windows 2003 server, however the final PoS software would be running on Windows Server 2008 R2. No biggie, right? Both of those server OSs should behave the same, but I thought I would try it on Windows Server 2008 R2, just to be sure that there were no surprises.
Our server guys spun-up a 2008 R2 (Terminal Server / Citrix) VM for me. I connected and ran my demo. Whoa. Wait a sec. It won’t connect. What is going on here?!
I Bing/Googled it and found that this is actually a common problem. The usual solution: give up. Yes, I know. No, I’m not making it up. Yes, I did read links from the 2nd and 3rd results pages. I’m telling you: nobody had a good solution. It was a little disheartening.
Giving-up, just didn’t seem like a reasonable option to me. There must be some other solution. Somebody must have found a solution to this!
Unravelling the Problem
For starters, I needed to understand the problem. Why did this work in Server 2003, but not 2008 R2? If Microsoft blocked this, they would have removed the settings. However, all of the settings are there. Some of my online research even showed conversations between people and Microsoft Support, where they tried for a week or two and then they simply gave up and went with another device or different connection means (some devices support IP connections in lieu of USB/Serial).
First and foremost: I was very fortunate to have tried it on Win2003 first. This proved that the problem was not:
• The device drivers (from Ingenico)
• The .NET library System.IO.Ports.SerialPort (or adjacent/supporting objects)
• RDP (remote desktop protocol)
• Network config (layer 3 traffic, firewalls, etc) – probably
• My PC (generally speaking)
These all worked. So I could eliminate them from my research (for now). The obstacle seemed to be isolated within Server 2008 R2 itself.
My best bet was to look at how Citrix / TS / RDP was intercepting the serial port traffic and routing it.
To route the serial traffic, a RDP server virtualizes the ports and presents them. As a RDP user, you get to see the physical ports (on the back of the server) and virtual ports (on my remote machine) all together, with no apparent difference. As a result, a program will have difficulty distinguishing between the serial ports that are on the physical machine and the ones that have been virtualized (routed from my machine, to the server and back-again). That is a good thing. Probably. Unless of course, you need to use COM1 or COM2 and the server actually has a physical COM1 or COM2 port. Then you have a device collision and the abstraction will seem rather un-friendly. However, for me, there was no conflict. It should all work nicely.
After digging around in the machine and investigating the hardware configs, here is what I found. In server 2003, when I connect via RDP, the OS creates a virtual hardware device for my (local) serial port, COM4.
On my local machine (windows 7), I found the Serial port configs in the registry:
In contrast, on Server 2003 and 2008, it looks like this:
So, I can see how my COM ports have been virtualized on the server. However, it seems very odd that it is mapping the names of my (remote) ports to itself: “COM4”=”COM4”. This implied that must be another config to find.
On Server 2003, I found this additional config in the registry:
“Port Description”=”S62312UA1311PVC: COM4”
“Client Device Name”=\\;COM4:3\\tsclient\\COM4
However, Server 2008R2 does not create these HKLM entries. That seemed like a useful clue.
The most interesting part of this reg entry is the last line, “Client Device Name”. You might notice that it sort-of looks like a network path to a file share. This is actually how the OS has abstracted COM ports. It treats them like file shares. So think about that for a minute and see if you can think about security risks that could come from software that opens a COM port and starts writing to it. What if, instead of telling it to open a valid COM port, you do something evil, like providing the name of an actual file on the C drive? If there are no safety checks, then things could get ugly. I think Microsoft must have seen hackers exploiting this back-door, and so they closed and locked-it. It would have been better if they put security measures on the COM port interface, just like they do with the file system. Probably most sys-admins would just overlook it. After all, most people really don’t want to do serial communications unless they really have to.
Security is nice, but now what can be done?
Well, I found some nice demo code to test the availability of some COM ports. What I found is that common names like “COM3” actually gets translated by the software (.NET) into “\\.\\COM3”. Same for the other COM ports. However, the “\\.” means that these are ports on the local machine. The ports on the remote machine are called “\\tsclient\\COM4”. I was pretty excited to figure that out.
When I tried to open the com port via the .NET serial IO commands, I got an error. It did not like any COM port whose name did not start with “COM”. That was bad. Again, I think this is meant to block possible exploits against physical files.
Fortunately, Microsoft has actually posted the source code for the .NET libraries. http://referencesource.microsoft.com/#System/sys/system/io/ports/SerialPort.cs (Pretty cool, right?) In the file System.IO.Ports.SerialPort (SerialPort.cs), on line 417 -420, you can see that they intentionally detect something like this and reject it. Damn! So, to get things to work I cannot use the standard Serial IO libraries that are built-into .NET. This (pretty much) leaves 2 choices and neither is good:
1. Make my own library to wrap the COM port (or use a pre-written library from someone else)
2. Somehow figure out how to map COM3 and COM4 to the TSclient ports, within the Server 2008 R2 registry or HAL.
And so, I stand at the edge of this grand canyon.
The option to rewrite (or at least alter) the libraries written by Microsoft, sounds a bit, um, crazy. However, the alternative is to discover how to do something that might not be possible. The fact that I read a few articles by people who worked through this with Microsoft support, only to give-up at the end, makes me think that this approach might not be achievable. Perhaps it is too lofty to think that I can solve something that Microsoft Support cannot?!
Follow-up: Luckily the vendor who made the device, has a SDK for making calls to it. This SDK does not use the .NET Serial IO libraries. They use their own Serial communication libraries, written in C++ (part of RBA SDK). So, when I put my program on the server, I was able to open a connection to \\TSClient\\COM4, with no problem. My project has been a success and works fine with serial over RDP on Windows Server 2008R2.
Follow-up #2: This solution also works with Citrix, but the com ports are named like \\Client\\COM4. This type of traffic also needs to be enabled on Citrix server configs. It is usually turned-off or blocked. I believe you also need the latest client (4.3 or above).