The coordinate for cell instances is the bottom-leftmost significant layer for the instance (not the cell itself) after rotation, mirroring, etc. Significant means a layer that is used by Calibre--for example, a device layer or connect layer. The coordinate is in the context of the containing cell.
For a device, the coordinate is the bottommost left corner of the seed (or device) layer. The coordinate is in the context of the containing cell.
Here is a graphic to illustrate both cases:
For additional information on $X and $Y, with added information on the $T (transform) parameter, refer to TechNote mg6973.