Time more than difficulty.  Takes an expert - which most soldiers weren't - a minimum of 15 seconds to reload a unrifled musket, call it 3 rounds a minute for the experienced, average soldier.  (A rifled musket takes longer so call it 1.5 rounds/minute on average.)

Infantry were rigorously trained to hold their fire until there was a high chance of inflicting damage - with a smooth bore if a soldier actually hit what he was actually aiming at it was Pure Luck - with volley fire (throw enough lead downrange and you are sure to hit ... something?) and by that time they were too close to try and reload ... so it was on-in with the bayonet¹.  The Brits got so good at this they kept trying it up through World War One.  (Against machine guns and barbed wire.  (It didn't work so good.))

The usual canard bayonets were used to repel cavalry is nonsense.  To repel a frontal cavalry charge all infantry has to do is stand there.  Horses don't like running into a solid mass and will shy away.  Flank attacks can work which is why they were put on the flanks during a battle.  And which is why the British developed the British Square as a battle formation, if you don't have flanks they can't be attacked.

---------------------------------------------------------------------------------------------

¹  Despite what you see in movies most of the time the infantry on the defense heroically ran away when faced with a determined bayonet charge from close range.

She believed in nothing; only her skepticism kept her from being an atheist. -- Jean-Paul Sartre