Expanding Netmon to aid in debugging

A walk-through of creating a Netmon parser in the context of a real case

This is an archive post of content I wrote for the NTDebugging Blog on MSDN. Since the MSDN blogs are being retired, I’m transferring my posts here so they aren’t lost. The post has been back-dated to its original publication date.

As is obvious to frequent readers of our blog, our team logs a lot of time in our debugger of choice (for some, windbg, for others, kd).  However, a debugger is not always the best tool for the job, and sometimes even the best tool has limitations. I found this to especially true when working a few Internet Printing Protocol (IPP) cases recently. 

Probably the biggest challenge of many IPP cases is the mixed environments you usually find IPP running in.  The benefit customers see in IPP over other print providers is it works natively or with minimal configuration on Windows, Mac, and Linux clients.  This makes it popular in places like college computer labs, where there isn’t one standard client system.   Unfortunately, this also means that we can’t really debug both sides of the communication when something goes wrong.

In a recent case, a customer was having problems printing from their Windows Vista clients to a Linux Common Unix Printing System (CUPS) IPP server.  If the CUPS server was set to allow anonymous connections, everything worked perfectly. When the administrator enabled authentication, he found that most print jobs failed to print.  After a bit more testing, he found that small jobs (perhaps a page of printed text) worked fine, but larger, multi-page documents failed.

For situations like this, I prefer to use a network trace to get a feeling for where the problem is occurring.  The problem was – IPP wasn’t one of the protocols built in to Netmon (and I find Wireshark’s IPP parser to not always work well – especially with Windows clients/servers).  I decided that the amount of time it would take to decode the traffic by hand could be better spent creating a Netmon IPP parser that I could use every time I ran into one of these issues.

One of the great things about Netmon is you can view the source of every included parser.  This was a big help, as I hadn’t written a parser before.  [Note: all steps noted are written using Netmon 3.4.There might be slight differences in NM3.3.]  To do this, open Netmon and click the Parsers tab. Under Object View, expand parser files and double click any of the .npl files.  The source will appear on the right. 

The language for Netmon parsers is similar to C++, with a limited set of statements.  These are all documented in the Netmon help file, but the ones I found useful are described below.  To begin, I started by defining a number of tables.  The basic idea of a table is to provide a way to convert a value to a string.  For example, one field in an IPP packet is the status of a printer, which is represented by an integer.  In order to allow Netmon to show printer states in a readable form, I created a table to convert the values as seen in Figure 1 below.

Table IPPPrinterState //2911-4.4.11
		case 3 : "idle";
		case 4 : "processing";
		case 5 : "stopped";
		default : "Unknown Code";

Figure 1: Netmon Table

Each table is defined with the Table keyword, followed by a name for the table.  (It may optionally be followed by a list of parameters, which I’ll use later.  In this case, I added a comment that specified which RFC and section this information comes from.)  A table consists of a switch statement with a case for each value, and a default for all other cases, much like other programming languages.  I created tables like IPPPrinterState for each field that could be represented in an IPP packet from information I found in each of IPP’s RFCs.

Once the tables were complete, I moved on to creating the Protocol portion of the parser.  This section of the code provides the logic that iterates through a packet and calls the tables for the appropriate data.  This section starts with either the RegisterBefore or RegisterAfter keyword.  These are used to determine how your parser is called.  Essentially, Netmon takes all of the parsers it has, and compiles them into one continuous binary (.npb) and registration tells Netmon where your parser fits.  For my case, I used the following registration code. 

[ RegisterAfter (HTTPPayloadData.OCSP, Ipp, Property.HTTPContentType.Contains("application/ipp")) ]

This tells Netmon that, when compiling the parser, it should insert my code right after the code for the OCSP protocol in its HTTPPayloadData parser, my protocol should be called IPP, and it should enter my code path if the HTTP Payload is of content type “application/ipp”.  This allows my parser to work a bit differently than the Wireshark IPP parser – Wireshark uses a port number (631) to identify IPP traffic, whereas my code looks at HTTP content types.  The advantage of this, for me, is that Windows servers use port 80 for IPP by default, not 631, so in cases with a Windows IPP server, this parser should correctly identify the packets.  You may be wondering how or why I chose to register after OCSP.  Basically, I knew I needed my code to be registered in the section of code where HTTP does its payload content type processing.  So I opened up HTTP’s parser, and searched for the content type analysis. OCSP was the first protocol I found in HTTP’s content type logic, so I used that as the place to insert my protocol.

After the registration comes the Protocol statement. I chose the following.

Protocol IPP = FormatString("Status/OpCode = %s", IPPOperationsSupported(OpID))

This names my protocol IPP and specifies that I want the description of the protocol to display the IPP status code.  This way, a user doesn’t need to drill down to find out if this is a print job or a printer status request.  You’ll notice FormatString is a function in Netmon that is similar to printf.  In this case, I am passing a variable (OpID, which is defined lower in my code) to my IPPOperationsSupported table to determine what this OpCode means.  Before I had a parser, I would need to look up the operations supported values in the IPP RFC for each packet I examined.

Next is the body of the protocol.  Basically, this consists of a series of fields (like variables) that define how a packet is laid out.  Creating a field is similar to declaring a variable in C++. You start by choosing a data type that matches the size of the data in the packet and provide a name for that field.  For example, Figure 2 shown below contains the first seven lines of my Protocol.

struct Version = FormatString("%i.%i", Major, Minor)
	INT8 Major;
	INT8 Minor;
INT16 OpId = FormatString("%s (0x%X)", IPPOperationsSupported(this), this);
INT32 RequestId;

Figure 2: Code in the protocol block

The IPP specification states that all packets begin with two 8-bit values, the first value specifies the major protocol version in use, and the second value specifies the minor.  In this case, I wrapped both in a struct so Netmon will display them as “Version: 1.0”, instead of separately as “Major: 1” “Minor: 0” on two lines.  After the version is a 16-bit field that specifies the operation requested (for example, print-job or get-printer-state).  I choose to display this value by looking it up in the IPPOperationsSupported table, then printing it as the string, followed by the hex value (e.g. “Get-Printer-Attributes (0xB)”).  The ‘this’ keyword simply uses the value of the current field, which in this case is the OpId.  Even though Netmon parses through the packet sequentially, this kind of use of a Field before its value is retrieved is allowed.  Finally, I set the RequestId field, which is a 32-bit int value.  Since this field is just a transaction ID for this conversation, I don’t need to do any formatting to it.

After that, things got a little more complicated.  IPP’s structure allows for a variable number of attribute groups, each of which can contain a variable number of attributes.  For example, in response to the request “Get-Printer-Attributes” from the client, the server responds with the Printer Attributes group, which contains a number of attributes like printer-state, queued-job-count, and so on.  First, I needed to deal with the attribute groups in a loop until I’d read each one.  IPP specifies that the end of the attribute groups is specified with the value of 0x03, so I wrote a while loop to create attribute groups until FrameData[FrameOffset] is equal to 3 (See Figure 3 below).  FrameData and FrameOffset are special values provided by Netmon.  FrameData is an array of the entire contents of the frame, and FrameOffset is Netmon’s current location in the packet.  I use this instead of declaring a field here because referencing FrameData[FrameOffset] does not advance the parser frame location.  This is important because I want to consume that value further down.

Inside that loop, I declared another struct that contains an attribute group.  Much like the Protocol IPP line above, we reference a field here that will be declared lower down.  This line does not advance the FrameOffset, since we don’t declare a field here.  The first line of this struct is the field declaration line that finally consumes the attribute group tag.  Below that is another While loop to process all attributes in the attribute group.  IPP differentiates between attributes and attribute groups by making all attribute group identifiers smaller than 0x10, and all attribute identifiers 0x10 or higher.  I use this as the condition for my loop.  Finally, I declare an Attribute struct inside this loop.  This struct is displayed after looking up how to properly print based on the Attribute Name and Value in the AttribDisplayTable. 

IPP declares attributes as an 8-bit type identifier (int, bool, string, etc.), a 16-bit int specifying the attribute name’s length, the name (a string), a 16-bit in value length, and a value.  Since I want to look up the value in various tables, depending on the Attribute Name, I store the Attribute Name as “AttName” in a property.  This way, I can continue to reference it while processing continues.  Properties are declared in brackets just above the field they will store.  In my case, I prepend the ‘Post.’ evaluation keyword to the property name.  This instructs Netmon to use the end result of the next line as its value, but before advancing the FrameOffset.  I do this again for the actual value, which I call Val.  If I did not use the Post evaluation keyword, Val would contain the unsigned int32 value of printer state, instead of the formatted string result I get by looking up printer state in its table.

While [FrameData[FrameOffset] != 0x03]
    struct AttributeGroup = FormatString("%s", IPPTags(TagGroup))
        INT8 TagGroup = FormatString("%s (0x%X)", IPPTags(this), this); 
        While [FrameData[FrameOffset] >= 0x10]
            struct Attribute = AttribDisplayTable(AttName, Val)
                INT8 Type = FormatString("%s (0x%X)", IPPTags(this), this);
                INT16 NameLength;
                AsciiString(NameLength) AttributeName;    
                INT16 ValueLength;
                    case "printer-state" : 
                    UINT32 PrinterState = FormatString("%s (0x%X)", IPPPrinterState(this),this);

Figure 3: Loops in protocol block

My case statements continue like printer-state for all possible attributes of IPP.  At the very end of the protocol block, after I’ve closed my switch, structs, and whiles, I have one more line, which consumes any data remaining in the packet.  This would contain document data if the packet was a print job, and is required so all the packet data is consumed before Netmon moves on to the next packet.  That line is:
BLOB (FrameLength – FrameOffset) Data;

As you can see, it is a binary blob data type, set to the size of the frame, less our current location.

Finally, after my Protocol block, I needed to define my own data type.  IPP defines its own data format to specify printer uptime, so I created a data type for it as shown below in Figure 4.

//Uptime format spec
    Size = 4;
    DisplayFormat = (this != 0) ? FormatString("%i days %02i:%02i:%02i (%i seconds)",
        (this/86400), //Days
        (this-((this/86400)*86400))/3600, //Hours
        (this-(((this/86400)*86400)+(((this-((this/86400)*86400))/3600)*3600)))/60, //Minutes
        (this%60), //Seconds
        this) : "0"

Figure 4: Custom data type

The first line of Figure 4 specifies this will be a data type composed of numeric data named UPTIME.  Size specifies how many bytes the type uses.  DisplayFormat is what Netmon displays for this type. In this case, I use the x ? y : z syntax.  Netmon doesn’t have if/then/else keywords, but instead uses this ternary operator.  I use a special case for 0 since it seems to be a common return value in the traces I’ve looked at, and having ‘Uptime: 0 days 00:00:00 (0 seconds)’ seemed excessive.

Figures 5 and 6 below show what the result looks like in Netmon.

Figure 5: Frame Summary
Figure 5: Frame Summary
Figure 6: Frame Detail
Figure 6: Frame Detail

So what did the trace show?  Windows attempts to send IPP requests with no authentication first, then if it receives an access denied, retries with authentication.  This is by design, as the IPP server replies with the authentication types it supports in the access denied message.  In the case of print jobs that are too large to fit in a single packet, IPP’s spec allows servers to either issue the access denied message as soon as it receives the first packet, or after it has received the entire job.  It turns out that the IPP Print Provider on Windows was designed to send the entire job before listening for a response, so it missed the access denied message that CUPS sent after it received the first packet.  See http://support.microsoft.com/kb/976988/ for related information.  Want a copy of the IPP parser? It will be included in a future release of the Netmon Parser Update.

I hope this post have given you a better idea of how Netmon works, how IPP works, and helps if you ever need to write a parser for your protocol.

-Matt Burrough

Case Study – Software Restriction Policies and Large EXE Files

This is an archive post of content I wrote for the NTDebugging Blog on MSDN. Since the MSDN blogs are being retired, I’m transferring my posts here so they aren’t lost. The post has been back-dated to its original publication date.

Recently I received a debug request for a customer having problems running large executables.  On their systems, they could run most EXEs without any problems, but they had two programs that were over 1.8 GB which, when run, would display the following error:

Error message stating "Windows cannot access the specified device, path, or file. You may not have the appropriate permissions to access the item."
Error message stating “Windows cannot access the specified device, path, or file. You may not have the appropriate permissions to access the item.”

If they tried to run them in a command prompt, they received the message “Access is denied.”  Both attempts were made with an administrator account and in neither case were the processes created.  Through testing, they found that the programs worked if they were scheduled to run as System and also worked when run in safe mode as an administrator. 

When the case was brought to my attention, it was noted that when the failing executables were run, the following appeared in process monitor logs:

Process Monitor Log showing Invalid Parameter result when trying to start exe.
Process Monitor Log showing Invalid Parameter result when trying to start exe.

The engineer did not see this when one of the problematic EXEs was run (successfully) on his test machine.  The customer provided a VM image of their system which we set up in HyperV with a named pipe kernel debugger.  I then started kernel debugging to find the cause of the INVALID PARAMETER error, hoping that resolving it would fix the issue.

To start, I looked at the call stack within process monitor for the invalid parameter:

Call stack of the process start in ProcMon
Call stack of the process start in ProcMon

The problem is this isn’t exactly where we return invalid parameter.  Looking at the source code for Fltmgr, it doesn’t return invalid parameter – this was just where the error was caught in procmon.   This call stack did provide some ideas for good starting places to debug, however.  First, I looked up the hex value for STATUS_INVALID_PARAMETER in ntstatus.h – 0xC000000D.  Knowing this, I decided to set a breakpoint on nt!IofCallDriver and ran the program.   Once the debugger broke in, I planned to use wt -oR.  This will trace through the calls displaying the return values next to each call.  From there, I would just need to find 0xc000000d on the return column.  Unfortunately, I had forgotten was wt does not display return codes in kernel debugging, only when debugging user mode.

With wt not an option, I decided to use a combination of debugger commands to approximate the output of wt. I knew the return value I was looking for, and I was also confident that I would find that code in the EAX register after the problem occurred.  As such, I needed to write a loop that would walk through the instructions until it found 0xC000000D in EAX.  The debugger provides two main options for walking instructions: p and tp (Step) will execute a single instruction and display the register values.  If the instruction is a call, it will not enter that function, but just display the results after that subroutine has been executed.  t (Trace) also executes a single instruction, but it will enter into the function and will display each instruction.  

In this case I wanted trace so I could see which function was returning the invalid parameter status.  Tracing though that many instructions/functions would take a long time, but there are some variations on t (and p) that can help.  tc (or pc)will execute instructions until a call statement is reached, where it will break and show the register values.  tt (or pt) will execute instructions until a return instruction is reached.  tct (or pct) will run until either a call or return is reached.  In this case, I opted for tct

Knowing that I would use tct, I had to find a way to execute tct statements until EAX was the value I was looking for. This can be accomplished with the z (While) debugger command.  The syntax is pretty easy, it’s just z(expression) and it works just like a do-while loop.  Putting it all together, I used this command in the debugger:

tct; z(eax!=0xc000000d)

I then waited for the debugger to break in so I could see where this status was being thrown.  Regrettably, the code called ended up going in to some recursion which made my while loop take way too long.  To resolve this, I set a new breakpoint just before we entered the recursion, reran the program, used p to step past the call then ran the tct loop.

This time I was quickly brought to the code I was looking for.  As soon as it broke in, I ran k to view the callstack:

kd> k
ChildEBP RetAddr  
b9541a3c f7b7fab9 Ntfs!NtfsCommonDeviceControl+0x40
b9541aa0 f7b8b02f Ntfs!NtfsFsdDispatchSwitch+0xe4
b9541bbc 8081df85 Ntfs!NtfsFsdDispatchWait+0x1c
b9541bd0 f7876d28 nt!IofCallDriver+0x45
b9541bfc 8081df85 fltmgr!FltpDispatch+0x152
b9541c10 f7876d28 nt!IofCallDriver+0x45
b9541c3c 8081df85 fltmgr!FltpDispatch+0x152
b9541c50 808f5437 nt!IofCallDriver+0x45
b9541c64 808f61bf nt!IopSynchronousServiceTail+0x10b
b9541d00 808eed08 nt!IopXxxControlFile+0x5e5
b9541d34 808897bc nt!NtDeviceIoControlFile+0x2a
b9541d34 7c82860c nt!KiFastCallEntry+0xfc
0012e960 7c826fe9 ntdll!KiFastSystemCallRet
0012e964 77e416f9 ntdll!NtDeviceIoControlFile+0xc
0012e9c8 77e6738d kernel32!DeviceIoControl+0x137
0012ec44 77e67032 kernel32!GetVolumeNameForRoot+0x16d
0012ec68 77e67782 kernel32!BasepGetVolumeNameForVolumeMountPoint+0x73
0012ecd0 7d20b01d kernel32!GetVolumePathNameW+0x1c7
0012ef18 7d20ae2c ADVAPI32!CodeAuthzFullyQualifyFilename+0xbc
0012eff8 7d20b33f ADVAPI32!__CodeAuthzpIdentifyOneCodeAuthzLevel+0x19f
0012f07c 77e6df9e ADVAPI32!SaferIdentifyLevel+0x163
0012f278 77e6ce03 kernel32!BasepCheckWinSaferRestrictions+0x60c
0012fa90 77e424b0 kernel32!CreateProcessInternalW+0xc0e
0012fac8 4ad0256f kernel32!CreateProcessW+0x2c
0012fc24 4ad01a2b cmd!ExecPgm+0x221
0012fc58 4ad019b3 cmd!ECWork+0x84
0012fc70 4ad03c58 cmd!ExtCom+0x40
0012fe9c 4ad01447 cmd!FindFixAndRun+0xa9
0012fee0 4ad0c30b cmd!Dispatch+0x137
0012ff44 4ad07786 cmd!main+0x216
0012ffc0 77e6f23b cmd!mainCRTStartup+0x12f
0012fff0 00000000 kernel32!BaseProcessStart+0x23

If we look at the assembly around Ntfs!NtfsCommonDeviceControl+0x40, we see that mov 0xC000000D in to esi, then mov esi to eax if our return from NtfsDecodeFileObject isn’t 4:

f7b7faf9 e8e904fdff      call    Ntfs!NtfsDecodeFileObject (f7b4ffe7)
f7b7fafe 83f804          cmp     eax,4
f7b7fb01 0f848873ffff    je      Ntfs!NtfsCommonDeviceControl+0x54 (f7b76e8f)

f7b7fafe 83f804          cmp     eax,4
f7b7fb01 0f848873ffff    je      Ntfs!NtfsCommonDeviceControl+0x54 (f7b76e8f)
f7b7fb07 be0d0000c0      mov     esi,0C000000Dh
f7b7fb0c 56              push    esi
f7b7fb0d 53              push    ebx
f7b7fb0e 57              push    edi
f7b7fb0f e83506fdff      call    Ntfs!NtfsCompleteRequest (f7b50149)
f7b7fb14 8bc6            mov     eax,esi

I looked at the source code for these functions, and it didn’t make sense that a failure here would cause the problems we were seeing; especially specific to large executables.  Out of curiosity I ran notepad on the VM while logging in perfmon and found that it too displayed INVALID PARAMETER, but the program stated and ran correctly:

Invalid Parameter result for app that works
Invalid Parameter result for app that works

Since this wasn’t the problem, I stopped reviewing the code and decided on a new approach.  We knew that when running the EXE in a command prompt we received an Access is denied message.  At that point it made sense to switch to user mode debugging and take a look at the cmd.exe process that was trying to launch install.exe.

Doing user mode debugging in a VM can be a bit of a challenge, especially if you are trying to minimize changes to the VM (and in my case, avoid putting any symbols on the customer’s VM image).  Since I already had a kernel debugger attached, one option would be to run ntsd.exe (debugger provided in the Debugging Tools for Windows) on the VM with the -p switch specifying the PID of the cmd.exe process I wanted to debug and -d switch which forwards the i/o of ntsd to the kernel debugger.  The problem with this approach is the kernel debugger just becomes a front end for issuing commands and seeing the output from ntsd.  That means all symbol resolution is still done on the target system running ntsd. 

I didn’t want give the customer VM Internet or corporate network access, so I instead opted to run dbgsrv.exe on the VM.  Running “dbgsrv -t tcp:port=9999” tells the debug server to listen on TCP port 9999 for debugger connections.  Then, on the HyperV server computer I could just run windbg -premote tcp:server=(IP of VM),port=9999 -p (PID of cmd on VM) to debug it. 

I suspected that we may be calling CreateProcess but it was failing, so I set a breakpoint kernel32!CreateProcessW.  Sure enough, it hit when I tried to run install.exe in the command prompt:

0:000> k
ChildEBP RetAddr  
0012fac8 4ad0256f kernel32!CreateProcessW
0012fc24 4ad01a2b cmd!ExecPgm+0x221
0012fc58 4ad019b3 cmd!ECWork+0x84
0012fc70 4ad03c58 cmd!ExtCom+0x40
0012fe9c 4ad01447 cmd!FindFixAndRun+0xa9
0012fee0 4ad0c30b cmd!Dispatch+0x137
0012ff44 4ad07786 cmd!main+0x216
0012ffc0 77e6f23b cmd!mainCRTStartup+0x12f
0012fff0 00000000 kernel32!BaseProcessStart+0x23

This time I could use wt -oR since this was a usermode debug.  Looking in ntstatus.h again, the code for STATUS_ACCESS_DENIED is 0xC0000022.  Running wt can take a very long time, so I used the -l switch, which limits the number of levels deep it will display.  This would be something like using tct as I did above until you were a few calls deep then using pct.  Using wt -l 3 -oR gave me the following:

  575   291 [  1]   kernel32!CreateProcessInternalW
   35     0 [  2]     kernel32!BasepCheckWinSaferRestrictions
   25     0 [  3]       ntdll!RtlEnterCriticalSection eax = 0
   48    25 [  2]     kernel32!BasepCheckWinSaferRestrictions
    1     0 [  3]       ntdll!NtOpenThreadToken
    3     0 [  3]       ntdll!ZwOpenThreadToken eax = ffffffff`c000007c
   57    29 [  2]     kernel32!BasepCheckWinSaferRestrictions
    1     0 [  3]       ntdll!ZwOpenProcessToken
    3     0 [  3]       ntdll!NtOpenProcessToken eax = 0
  116    33 [  2]     kernel32!BasepCheckWinSaferRestrictions
  113     0 [  3]       ADVAPI32!SaferIdentifyLevel eax = 0
  130   146 [  2]     kernel32!BasepCheckWinSaferRestrictions
    4     0 [  3]       ntdll!ZwClose eax = 0
  132   150 [  2]     kernel32!BasepCheckWinSaferRestrictions
   22     0 [  3]       ntdll!RtlLeaveCriticalSection eax = 0
  138   172 [  2]     kernel32!BasepCheckWinSaferRestrictions
    5     0 [  3]       kernel32!__security_check_cookie eax = ffffffff`c0000022

Now we are getting close!  I set a new breakpoint for kernel32!BasepCheckWinSaferRestrictions and reran the test.  This gave me the following line:

   63     0 [  3]       ADVAPI32!__CodeAuthzpCheckIdentityHashRules eax = ffffffff`c0000022

One last run with a new breakpoint at ADVAPI32!__CodeAuthzpCheckIdentityHashRules and I found what I was looking for:

   58   218 [  1]   ADVAPI32!__CodeAuthzpEnsureMapped eax = ffffffff`c0000022

The depth is shown in brackets.  As this call was 1 deep from __CodeAuthzpCheckIdentityHashRules and I was using 3 as my maximum depth in wt, I knew this is where the access denied was coming from.  I reviewed the source code and found that this is the code that performs Software Restriction Policy checking.  Specifically, we are attempting to map the executable into memory to perform hash checking on it.  Since there isn’t 1.8 GB of contiguous available memory, it failed.  Looking at the VM, I discovered that the customer had implemented a number of software restriction policies.  As a test, I removed their restrictions on the VM, and the programs ran successfully.  A search of the KB revealed that a hotfix was published for this problem: 973825.  Installing the hotfix in the article also resolved the issue with their policies intact.

-Matt Burrough

Unlocking some puzzles requires building a better key… board

This is an archive post of content I wrote for the NTDebugging Blog on MSDN. Since the MSDN blogs are being retired, I’m transferring my posts here so they aren’t lost. The post has been back-dated to its original publication date.

Hi, this is Matt from the Windows Performance team. Sometimes we are presented with
problems that defy our usual troubleshooting and require a creative approach. In a recent
case, we needed a way to test the responsiveness of an application as text was typed into its
fields. Initially, we tested the program using a script that used the SendKeys method to time
entry time. Unfortunately, these tests aren’t completely realistic, since the script can be
affected by the processor utilization on the system, and the script can’t create hardware
interrupts like a keyboard does. Realizing that only real keyboard input would be a valid test,
and that the rate of typing needed to be reproduced exactly for each test, I set about building
an automated keyboard.

First, I found an old PS/2 style keyboard that hadn’t been used in years and opened it up.
Luckily, it was old enough to use all through-hole components, which made it easier to modify.
The main component I cared about was the keyboard encoder, which was a COP943C. A search
online turned up a datasheet for the keyboard encoder with a sample circuit design that looked
very similar to this keyboard. The document shows there are a couple of steps to determining
which pins need to be shorted to generate a particular key. First, each key has an ID number
that is shown in figure one of the PDF (figure one below). After finding the proper ID, a table is
consulted to determine the row and column pins used to create that key code (figure two
below). Finally, those row and column numbers are translated into physical pins on the
encoder using the schematic diagram (figure three below). For example, the letter ‘a’ is
number 31. The matrix shows 31 is made with the L5 (column 6) pin and C6 (row 3) pin. The
pin out shows this to be physical pins 14 and 19. When tested, shorting these pins creates an

See the Video:

Matt explaining the automatic keyboard
Key Codes
Figure 1: Key Codes [i]
Key Code Matrix
Figure 2: Key Code Matrix [i]
Encoder Pin Out
Figure 3: Encoder Pin Out [i]

Now that we know how the keyboard circuit works, we need a method to generate key
“presses.” For this, I found a board I assembled a year or two ago using a PCB and components
from moderndevice.com. The board is an Arduino clone that is based on Atmel’s ATmega168
microcontroller. One of the great things about using an Arduino is their IDE, which allows for C
programming with a number of pre-defined functions to make development quick. Also, the
boot loader is already taken care of, which makes the work easier.

Wiring the board to the keyboard was straightforward. Figure 4 shows how to control a relay
with an Arduino, and triggering a keyboard is rather similar. A resistor is placed between a
digital out pin of the Arduino and the base pin of a transistor. The collector then goes to one
pin of the keyboard encoder needed to type the letter desired, and the emitter goes to the
other pin.

Arduino-controlled Relay
Figure 4: Arduino-controlled Relay [ii]

In order to save on solder joints, I decided to chain together the transistors, which affected the
key selection. Additionally, because I wanted to leave the encoder in the original circuit and
some of the pins were blocked by other components (resistors, capacitors), specific pins were
selected. Figure 5 shows the layout of the transistors. These were soldered to a prototyping
board with hook up wire to connect back to a breadboard with resistors and the Arduino and
hook up wire soldered directly to the pins of the keyboard encoder.

Transistor Layout
Figure 5: Transistor Layout

These pins selected allowed characters a, s, z, space, and enter to be typed. All that remained
was to write some software to trigger the transistors. The code first sets the digital pins to
output and logic low, turns on a LED to show it is working, then waits 3 minutes to allow time
for the PC to boot and application in question to be launched. The LED then goes out for a five second
warning, and then the loop sequence begins. The loop turns on the LED, types “as z”
followed by enter, then turns off the LED and sleeps for 2.5 seconds before starting again.

// Sample code to drive keyboard encoder
// Matt Burrough
// September, 2008

int ledPin = 13; // Use digital pin 13 for a status LED
int sPin = 3; // Connect pin 3 to the transistor connected to the s leads
int aPin = 4; // Pin 4 is for a
int zPin = 5; // Pin 5 is z
int enterPin = 6; // Pin 6 is enter
int spacePin = 7; // Pin 7 is space
int holdKey = 30; // Milliseconds to "hold" each key down
int betweenKeys = 50; // Milliseconds to wait between key presses

void setup() { // Initial setup code (runs at power-on)
    setupPin(ledPin); // Set up each pin with function below
    digitalWrite(ledPin, HIGH); // Turn on the LED to show the board is on
    delay(180000); // Wait 3 minutes to allow time for PC to boot
    digitalWrite(ledPin, LOW); // Turn off the LED
    delay(5000); // Wait 5 seconds

void loop() {
    digitalWrite(ledPin, HIGH); // Turn the LED on
    typeKey(aPin); // Type keys
    digitalWrite(ledPin, LOW); // Turn the LED off
    delay(2500); //Pause 2.5 seconds

void setupPin(int pin) { // Used to set up pins...
    pinMode(pin, OUTPUT); // Set the digital pin as output
    digitalWrite(pin, LOW); // Turn off the pin

void typeKey(int pin) { // Type a key...
    digitalWrite(pin, HIGH); // "Press down" on a key
    delay(holdKey); // Hold down the key
    digitalWrite(pin, LOW); // "Release" the key
    delay(betweenKeys); // Pause between keys

Figure 6: Code Sample

That’s how I made an automated keyboard. I hope that you’ve found this post interesting; I’ll
leave you with a photo of the finished product.

Assembled Circuit
Figure 7: Assembled Circuit


ⁱⁱ http://www.arduino.cc/playground/uploads/Learning/relays.pdf