The online help for the Touchstone Viewer has a very good description of mixed mode S parameter. I suggest starting by reading that section. It starts on page 1301 of the PDF version of the manual with HyperLynx SI version 9.4.2.
One thing the people often overlook is that a port has 2 terminals. Thinking about the connection of 2 terminals helps to understand differential signals. Standard S parameters are measured with one terminal on a signal and the other terminal on a reference (usually called ground) very nearby. This is also called single-ended signaling. A differential signal or S parameter is measured with the 2 terminals on adjacent signal nets. We want to see the difference between the signals on 2 nets. A true differential signal is centered on 0V and the two signals are exactly equal in magnitude and opposite in polarity. Any signal that is common to the 2 signals creates an offset of the observed differential signal.
Consider a 4-port S-parameter set. These ports are measurements of single-ended ports. When you convert these S parameters to mixed mode, it changes the data to show 2 differential ports (2 signal points on one end and 2 signal points on the other end). The magic is that it considers the 2 ports with differential SIGNALS and also with common SIGNALS. The upper left quadrant of the matrix is the 2x2 differential-to-differential signal relation. The lower right quadrant is the common-to-common signal relation. The other two quadrants show signal conversion between differential and common signals. This is important in quality and EMI analysis, but many people don't know what to do with this data. The most common consideration is just the differential signal relations in the upper left quadrant. This 2x2 matrix is the differential mode S parameters. There you have standard (4x4 matrix), mixed (4x4 matrix) and differential (2x2 matrix) mode formats.
Since SerDes lanes are generally differential pairs, you want to view their S parameters is differential mode. You can use mixed mode if you want to evaluate common signal behavior or mode conversion.
I don't know why the viewer doesn't allow TDR view of differential S parameters, but you can see that view using standard S parameters. Select the standard S parameter file, and switch to time domain view. Notice that you can change the Plot Type to Mixed Mode and the Mode to Differential.
Then you can see the TDR of the differential pair.
Between this note and the User Guide, you have a lot of data to digest. Let us know if you have more specific questions.