This is so broad and education on the subject is so sparse its hard to get people upto speed. I almost feel like people do the opposite and spread mis-information (not in regards to this thread, more like random gaming outlets when gamers finally learn what a network is.).
Lets explain a few things first.
This is my internet speed represented by a speed test to a local server very close to my location and CONNECTED to my provider.
I pay for 500/500
This is the same speed test across the country to LA for sake of argument, same provider.
Your first thought is that distance matters and you would only be kind of right.
What really matters here is transit.
I would probably pull close to 500/500 (accounting for a little loss) but that is if I had a 3200 mile fiber run to the router in LA. (and other engineering stuff we wont get into like the optics required to make that even happen)
No in this case we are relying on transit. Let's take a look. In this case I know that the speedtest server I used specifically was lax.host.speedtest.net. So lets trace that and see who we touched. Now this isn't definitive, alot of times core and edge routers are set to ignore ICMP requests. In these cases they show up as asterisks *. This should still suffice for our example. (we won't get into MTR here)
In this case we can see that it left my local POP at hop 3 and then arrived at miami in hop 4. Following that we see the various *'s as mentioned before this is either peering or frontiers own backhaul to CA.
Moments later we see the first Edge router respond indicating it has hit a POP that will now delegate traffic to its final destination. This happens in hop 9 when it finally lands in the greater LA network.
The thing is That server, that data center. Those routers, those different providers, all of them will add latency and speed decreases (though not mutually exclusive) depending on congestion or just inefficiency.
Thats a astronauts view of networking hardware and route awareness.
The next is generally software, or physical.
Youtube for example maintains a buffer. Thats why when first clicking on a video you get a tremendous amount of bandwidth usage until youtubes CDN determines that you have a significant buffer for you video. At this point it will decrease the amount of data transmit to only the level needed to maintain the buffer. This makes it easier on CDN and provider networks. It helps ease congestion and over all network performance. Not to mention what if you stop it? If a 10 hour video is 6GB and you stop watching after 20min youtube does not want to transmit all 6GB of data. Remember transit is expensive!
Finally software regulation (not in the context of QoS) affects many platforms. MMOs, online only, and online connected games and video services. This is not unique to youtube.
Generally lastly is physical elements. From your home modem getting hot and clocking down, to noise on the line itself or maybe just the "last mile" to your house having cables in bad shape all of this including things like HDD reads from the source (like streaming a movie from a home server.) or CPU usage while decoding, can all affect how internet usage changes.
If you think that was long winded I held myself back considerably. In short what you are experiencing is not necessarily alarming but should you actually believe it an issue reboot your router/modem or talk to your ISP and explain the problem.