Declare a public mode variable.
Set mode = on in the button click subroutine.
Set mode = off a the end of the mousedrag function
Check if mode is on at the beginning of the mousedrag function; exit if mode = off.
Or some variation of that.
In this way, the mode is only on when you click the button. As soon as the click and drag action is done, it turns itself off until the button is clicked again.
There may be some better way of doing it by setting the mouseDrag commandlistener to nothing. You might want to experiment with that as well. There may be a performance benefit depending on the persistence of the mouseDrag listener. Perhaps activate the commandlistener in the buttonclick routine then kill it at the end of the mousedrag function.