Hi - it looks like it should work. Yes, the environment variable is the correct way to specify the custom menu command file, there is not another invocation method. I would make sure that there are no spaces after the "\" line continuation characters in your Tcl file - that messes up the Tcl parsing.
Are you looking at the very end of the Calibre menu?
I see this message in the CIW, shortly after messages for loading the Calibre Skill Interface:
// Calibre Skill Interface * ...
Successfully read menu customization file: ...
You can also set the environment variable MGC_CALIBRE_ECHO_VIEWER_MENU_CMDS to get the commands echoed, but I don't think that will help if the menu is not loaded.
Hope this helps,