It is generally excepted that you have up to 15 seconds to get the attention of your audience. So if your website is taking longer to load than 15 seconds it has failed before it has even started.
The speed a page loads is of course, in part, down to the users connection. Something you have no control over. But you do have control over how your website is hosted. So, what is hosting? It is a computer which is generally part of a very large network of computers in a data center. The difference between your computer and a 'server' in a data center is the way it is connected to the internet. There is nothing stopping you using your home computer to host a website, but the result would be very slow speeds especially if more than a couple of people were visiting the site. Most internet offerings where they say 'up to 20mb' or whatever are for download speeds. The bit you need if you're going to host a website is the upload speed, generally a fraction of the download speed.
Now a hosting company will try to maximise the return on each of it's servers. To do that, in the case of the budget solutions, is to put as many websites onto each as they can on each. Now lets try something, on your home or office computer see how many apps you can open and use at the same time. Play some music, a video too, how far can you go before your computer slows down, maybe even crash? As I mentioned before a server is just a computer. If there are a lot of websites on it that means there may be lots of visitors. Each website is like an app using the computer processor and RAM. It will get slower and slower. If any of those websites you are sharing the server with gets very popular, it will be all the sites on the server that suffer.
So, in conclusion, if you think you are saving money on a low cost hosting solution, you are almost certainly not.