VLC is great at what it does, but even it's developers would tell you it is not really a Blu-ray player - it s just capable of playing some streams on a disc.
The best soft player I am aware of is PowerDVD Ultra - we use this as well as recommend it for PC users ( Macgo Bluray player Pro is our player of choice on Mac OS) as it handles everything we have ever thrown at it, and the Macgo people have amongst the finest software support I have ever experienced - 3 days from filing a bug report to a fixed version available to install!! Point is, a media player is one thing, a licensed Blu-ray player is quite another - almost all software players are using unlicensed codecs or libraries somewhere (which is why they are free) and may not necessarily be properly spec compliant for DVD or Blu-ray. You don't say what platform you are using but I am assuming Windows. They are best effort - especially Blu-ray with it's mandatory AACS - and some discs don't work well, and with Blu-ray the menu structures won't work properly either.
These are both licensed technologies and no free player will handle these discs properly on demand. It is not, and never will be, a DVD player or a Blu-ray player. VLC is very complex and is a media player that can also handle DVD and some Blu-ray. #2 - what MPEG-2 decoder is being used by VLC? If you delve into it's settings there is an amazing amount of tweakability available under the hood - try to discover what codec (MPEG-2, obviously, but what one is important) it is using & if it is available try setting a different one & see if it helps. Don't go too fast either - that is even worse, so as long as you let IMGBurn do the work by setting to 2x and allowing it to sort things out you will be fine. Best of all, it will not let you burn at incorrect speeds - years ago it was accepted wisdom to burn at single speed, but no longer - these days it is best to burn at the minimum speed supported by the combination of your media and burner, and IMGBurn will tell you this information & use it - if you set it to a slow speed such as 2x, it will not actually write at this rate if the media & burner combo is set to 4x, so instead of allowing a bad burn it will always reset to the minimum for the combination. It is also a good idea to use a tool such as IMGBurn for all disc image creation & burning as it just does it, well, properly. #1 - what is the make of the blank media you are using? Blank discs are very far from all being created equally and better quality blank media will yield better results. Various possibilities come to mind here, given the facts as I understand them being: