You aren’t the only one! The biggest thing is that you have to understand that you can do this without the instrument. Once you get that through your head it makes it easier to do on the violin. Pretend like you are holding a violin and now go ahead and place your fingers down on your fake fingerboard. That took a lot less effort than before right? This is the same effort level you should have when you actually have the instrument in your hand. Most likely you are pressing too hard, and if you have any sort of bad angle technique with your fingers (you are reaching for notes instead of just placing fingers), you will most likely keep restricting your vibrato. It is great that you know that this is causing your vibrato to suffer, now it is just a matter of working on it!
Another thing you could do is working just doing pizzicato, and try to see how light you can place your fingers down on the fingerboard while still getting a clean pluck noise. Plucking/pizzicato is when you use your finger to pick the string instead of the bow. Try using the very tip of the finger (right before the nail) instead of the meat of the finger That also helps in establishing minimal pressure which will lead to better vibrato and shifting technique.