Reverse engineering fglrx and HDMI audio
So this time instead of just improving HDMI audio support I've decided to also describe that RE process. I hope it may help some hacker who would like to help but doesn't know where to start.
First of all, I assume we want to include out work in Linux kernel, so we're interested only in clean room reverse engineering. It means we can't disassemble binary, but we are allowed to observe closed source driver behaviour. There are two methods left in this case: reading registers values or watching regs operations.
Reading registers
This is a simpler method that doesn't require any extra hacks or knowledge of closed driver architecture. It's based on just reading registers (all or some subset) and figuring out their meaning. It's usually a good idea to compare registers values from different moments: for example before connecting HDMI monitor and after. To verify your guesses, you can try modifying registers manually. For AMD/ATI GPUs there is a great tool radeontool/avivotool.
The problem with this method is you can't really watch the whole process (of enabling HDMI audio). You just see a state before and after, order of changes remains a mystery. It's also hard to figure out which registers are related to the investigated feature. And there is the worst case: indirect access. If there are so called "index" and "data" registers, it's pretty impossible to track changes performed by closed source driver. You can find tons of hardware with subsets of registers that are accessed by a 2 main registers. In such case writing values to 3 sequential registers is performed by: 1) Writing "1" to index reg and "0xbaad" to data reg. 2) Writing "2" to index reg and "0xdead" to data reg. 3) Writing "3" to index reg and "0xc0de" to data reg.
Luckily HDMI audio on AMD cards is handled in quite simple way. It doesn't require specific order of regs operations and doesn't use any indirect access. That's why we were able to RE it that way. Unfortunately fixing some bugs and adding support for the newest cards isn't possible using this method.
Tracing regs operations
The best idea is to track all operations performed by closed driver and analyze them. First of all you need a tool for tracking. In case of kernel modules it can be handled using "mmiotrace" and in case of user space drivers it's probably the best idea to use gdb. HDMI audio in fglrx is handled by DDX driver (you can verify that by removing fglrx.ko and still using fglrx DDX), so we need some gdb trick. Unfortunately I can't explain how to write gdb script tracing registers operations. I use the one created by Jerome Glisse and I didn't really have to worry about that. I guess you have to find proper functions names ("strings" may be your friend) and find out which CPU registers are holding interesting arguments like register and value (in case of writing).
So for HDMI reverse engineering I'm using fglrx.gdb, just note it's 64b specific (see CPU registers). Running this script results in xorg-trace.txt being created (with all registers operations logged). Having such a dump you just have to find a HDMI setup related operations, understand them and re-implement in your driver.
Some real-life example
OK, I've described the process, but there is nothing better than some real example. I've using fglrx.gdb to track fglrx's operations on my HD6320. I've found HDMI registers ops in the xorg-trace.txt and there is some part of them:
RREG32(0x000120dc); -> 0x00000000
WREG32(0x000120dc, 0x24414000);
RREG32(0x000120e0); -> 0x00000000
WREG32(0x000120e0, 0x00001000);
RREG32(0x000120e4); -> 0x00000000
WREG32(0x000120e4, 0x28488000);
RREG32(0x000120e8); -> 0x00000000
WREG32(0x000120e8, 0x00001880);
RREG32(0x000120ec); -> 0x00000000
WREG32(0x000120ec, 0x24414000);
RREG32(0x000120f0); -> 0x00000000
WREG32(0x000120f0, 0x00001800);
RREG32(0x00012104); -> 0x10000000
WREG32(0x00012104, 0x00100000);
RREG32(0x00012108); -> 0x00000000
WREG32(0x00012108, 0x00200000);
RREG32(0x00012120); -> 0x00000000
WREG32(0x00012120, 0x00876543);
Doesn't look too friendly, does it? Let me manually add registers names we received from AMD:
RREG32(0x000120dc); -> 0x00000000 HDMI_ACR_32_0
WREG32(0x000120dc, 0x24414000); HDMI_ACR_32_0
RREG32(0x000120e0); -> 0x00000000 HDMI_ACR_32_1
WREG32(0x000120e0, 0x00001000); HDMI_ACR_32_1
RREG32(0x000120e4); -> 0x00000000 HDMI_ACR_44_0
WREG32(0x000120e4, 0x28488000); HDMI_ACR_44_0
RREG32(0x000120e8); -> 0x00000000 HDMI_ACR_44_1
WREG32(0x000120e8, 0x00001880); HDMI_ACR_44_1
RREG32(0x000120ec); -> 0x00000000 HDMI_ACR_48_0
WREG32(0x000120ec, 0x24414000); HDMI_ACR_48_0
RREG32(0x000120f0); -> 0x00000000 HDMI_ACR_48_1
WREG32(0x000120f0, 0x00001800); HDMI_ACR_48_1
RREG32(0x00012104); -> 0x10000000 AFMT_60958_0
WREG32(0x00012104, 0x00100000); AFMT_60958_0
RREG32(0x00012108); -> 0x00000000 AFMT_60958_1
WREG32(0x00012108, 0x00200000); AFMT_60958_1
RREG32(0x00012120); -> 0x00000000 AFMT_60958_2
WREG32(0x00012120, 0x00876543); AFMT_60958_2
Hope it looks better now. So you can see that after writing ACR registers fglrx start writing various values to the AFMT_60958_* regs. If you take a closer look at evergreend.h you can ever understand what that values mean. This 0x10000000 is actually a AFMT_60958_CS_CHANNEL_NUMBER_L(1) and 0x00200000 is AFMT_60958_CS_CHANNEL_NUMBER_R(2). So this code simply initialized channels with some sane order.
There is nothing left but just implementing the same for open source driver. How I've handled that? Take a look at this simple
[PATCH 6/6] drm/radeon/evergreen: write default channel numbers