I wasn't sure of the organization of the parameters in mixed mode, so I ran a quick simulation. I created 3 diff pairs, with slight differences. I extracted the S parameters up to 10GHz and converted the results into mixed mode. Here are the results.
The green curves are all the SDD insertion loss curves. The red curves are the SCD conversions. This shows that the common and differential quadrants for 12 ports are the same as for 4 ports. I hope this helps.
Oh, that makes sense. Four 6x6 sub-matrices. For some reason I was hung up on looking for a mapping to 4x4 sub-matrices, thanks.