I think if one bothers about speed at all, then only when passengers can  rely on frequent trains. I mean, it makes little difference if a travel takes 3 or 5 hours when you would leave at 8 o'clock but the only trains are at 6 and 18 o'clock.

Regarding two single track, use the other for passing vs. one double-track operation, let me demonstrate that the difference is significant with the following virtual train scheme I generated.

This is a 50-mile line between two major cities, with 12 sections. The borders of the sections are considered both crossover points and stations for local trains. I put paths of twice-hourly passenger trains with 50 mph travel speed and once-hourly medium-high-speed trains with 133.3 mph top speed on it in both directions. (For simplicity, different gradient lines represent acceleration/braking for the fast trains, I didn't bother to resolve that for the local trains.)

The second line for each path represents a 2.5-minute buffer for lateness. Then: the orange fields show time-distance zones when trains can pass each other, while the grey zones represent a train travelling 'on the wrong track'. In the first mode of operation, all the grey squares must be kept free, and there is room left for only one long non-stop freight train with 50 mph per direction. In a double-track operation, you only have to watch out for the fast trains and their crossing of same-directional trains, room for around four freight train paths in both directions.

_{*Lunatic*, n.
One whose delusions are out of fashion.}