Not sure that the word "can't" is makes sense. I know of a number of non-GPL products which use GPL code primarily as loadable libraries
Everything in rockbox runs within a single process in a unified memory space. We don't have the ability to side step the GPL like PC applications can.
I would assume that GPL code could likewise use non-GPL code as long as the entry points were known.
Just because the entry points are known doesn't mean you can ignore license terms. Generally when applications use GPL software without being GPL (e.g. lame), they do so by running a GPL program and then capturing its output.
But my last question is what determines the size of the sdhc card that can be used? Or is there a single limitation imposed by either hardware or firmware?
The SDHC driver determines if a card works or not.