We use some utility encryption routines in C for various purposes. When I first tried to migrate these routines from HP-UX to Red Hat Linux, I ran into a problem right away: the linker couldn’t find the libcl.a library.
There is no library by that name in Linux, and I had no idea what the linker wanted to find there. As an experiment, I edited the make file to remove the reference to –lcl. Now the linker complained that it couldn’t find the functions setkey() and encrypt().
A little research showed that, in Linux, those functions reside in the library libcrypt.a. I edited the make file again, replacing the reference to –lcl with a reference to –lcrypt.
(Under HP-UX, the libcl.a library includes not only the encryption functions but also a bunch of math functions. Under Linux, the math functions presumably reside in the library libmath.a. However I haven’t needed any of them so far, so I’m guessing.)
Now the linker was happy, but I wasn’t. Whenever I tried to encrypt anything, all I got was a string of zeros. Um, that’s not encryption. At best, it’s a really, really, really bad hash function.
At that point I fell down the rabbit hole of debugging – grepping the source, studying the code, inserting displays, Googling for clues, and trying experiments. Here’s what I found out.
The setkey() and encrypt() functions apply DES encryption to blocks of 64 bits at a time. That’s bits, not bytes. However, you don’t pass the bits as 8-character arrays. You pass them as 64-character arrays, one character for each bit. It’s the job of your application to translate back and forth between these bit arrays and whatever the data should really look like.
One reason for this arrangement is probably that not all machines use 8-bit bytes. The C language requires that a byte contain at least eight bits, but it can have more. I’ve heard of CPUs that use 9-bit bytes, and others that use 64-bit bytes. If you really want to be perverse, you could build a machine with 37-bit bytes. It may be easier for an encryption standard to support exotic architectures if it breaks everything down to the bit level.
In the HP-UX version of these functions, the bits in these arrays can be encoded as the ASCII characters ‘1’ and ‘0’. I haven’t found any documentation of that fact, but that’s how our programs were coded, and they seem to work okay.
In the Linux version of these functions, the bits in these bit arrays evidently must be encoded as the binary values 0×00 and 0×01. I haven’t found any documentation of that fact either. The man pages on both systems are ambiguous.
The best evidence is that, after I rewrote our code to pass the bit arrays with binary values instead of ASCII characters, the encryption started working on Linux. I can encrypt something and then decrypt it, and get the same thing I started with. Furthermore I can encrypt something on HP-UX, decrypt it on Linux, and get the right answer.
Out of curiosity, I tried the Linux version on HP-UX – and it worked! Evidently the HP-UX implementation of setkey() and encrypt() can accept either character values or binary values in their bit arrays, but the Linux version only accepts binary values. Once I converted the code to use binary values, I could use it on both platforms without a bunch of ugly #ifdefs cluttering up the code.
