Besides fast apertures/lenses, there is also fast film.
The term high speed is also used for the same meanings.
I always interpreted the 'fast' designation as meaning the operator could 'shoot fast' or 'shoot fast things', meaning to capture fast moving action better than with the slow lens, aperture, or film. Fast film allowed you to use a faster shutter speed.
Kodak, for many years, offered a film whose official name (at least in the USA) was "High Speed Ektachrome"**. Zeiss sold a series of f/1.2 cinema lenses called 'Super Speed Lenses'.
Apertures are more correctly be referred to as large/small, film as higher/lower sensitivity.
In spoken speech it is easier and faster(ha ha) to say 'fast lens' than 'large aperture lens', 'fast film' than 'high sensitivity film'.
**High speed meant ASA(ISO) 160, pushable to 400.