I don't think its the hardware, you are 99% there mate!
Looking at the video, there is an obvious formatting error in the last modeline, perhaps there are others I can't see, so I suggest you carefully select, then copy and paste my working modelines into your Winmodelines window and do the reboot when it asks for it.
Code:
Modeline "664x377_25 10.1kHz 50.0Hz" 8.100 664 680 752 800 377 379 385 405 interlace -hsync -vsync
Modeline "560x220_25 6.0kHz 25.0Hz" 3.750 560 568 584 624 220 221 233 240 -hsync -vsync
Modeline "656x383_25 11.0kHz 50.0Hz" 8.820 656 665 744 800 383 384 392 441 interlace -hsync +vsync
Modeline "640x540_25 15.6kHz 50.0Hz" 13.620 640 694 758 872 540 556 562 625 interlace -hsync -vsync
Modeline "570x456_30 15.7kHz 59.9Hz" 12.080 570 621 678 768 456 472 478 525 interlace -hsync -vsync
I must say that the way you are switching between standards is exactly the way I do it.
There is a final "OK" dialogue which doesn't appear on the video because it's on the screen that is now blank, I don't click that and it allows the "Esc" key to revert the screen back to normal as you have descovered.
Peter Scott uses the Windows resolution slider to switch to the system he wants, but that doesn't work too well for me for some reason.