I don't think I've ever seen an SoC capable of directly executing its firmware off the SD card without loading it into RAM first.
Given the lineage of the Clip Sport series, it's more likely that there is still some sort small flash chip onboard, solely used for firmware storage.. and with the SD card removed, I would expect the device to boot up. Since you have yours apart, can you post photos or other details about the various chips it has? I've only found photos of older models in the series, not the Clip Sport Go.
Still, there is nearly no documentation to be had on this series of SoCs, and due to their severe resource constraints there hasn't been much interest in figuring out how to build or otherwise modify its firmware. HEre is one of the few things I found:
https://github.com/nfd/atj2127decrypt