Is it possible to use the dock connector with custom hardware?
If you design the hardware properly, sure. The dock cable has USB, line out, a serial port, etc.
The reason i'm asking is because I got an idea to solder the end of an ipod cable to a NES controller for use with ROMS.
Theres no support for USB or serial on the ipod yet, so you won't be able to do this unless you get those working first.