Since the VM is accessible all fine via SSH, it is not a problem with the VM in general. As only a VM reboot is required to restore the console connectivity, should not be a problem with the host. When the issue occurs, instead of refreshing the browser, can you close the console window and open a new one? Does this work? Also, can you install one Windows VM and check if the issue occurs on the Windows VM too?
Cheers,
Supreet