The practice of saying “o’clock” is simply a remnant of simpler times when clocks weren’t very prevalent and people told time by a variety of means, depending on where they were and what references were available.
Generally, of course, the Sun was used as a reference point, with solar time being slightly different than clock time.
Clocks divide the time evenly, whereas, by solar time, hour lengths vary somewhat based on a variety of factors, like what season it is.
Thus, to distinguish the fact that one was referencing a clock’s time, rather than something like a sundial, as early as the fourteenth century one would say something like, “It is six of the clock,” which later got slurred down to “six o’clock” sometime around the sixteenth or seventeenth centuries. In those centuries, it was also somewhat common to just drop the “o’” altogether and just say something like “six clock.”
Using the form of “o’clock” particularly increased in popularity around the eighteenth century when it became common to do a similar slurring in the names of many things such as “Will-o’-the wisp” from “Will of the wisp” (stemming from a legend of an evil blacksmith named Will Smith, with “wisp” meaning “torch”) and “Jack-o’-lantern” from “Jack of the lantern” (which originally just meant “man of the lantern” with “Jack,” at the time, being the generic “any man” name.
Later, either this or the Irish legend of “Stingy Jack” got this name transferred to referring to carved pumpkins with lit candles inside).
While today with clocks being ubiquitous and few people, if anybody, telling direct time by the Sun, it isn’t necessary in most cases to specify we are referencing time from clocks, but the practice of saying “o’clock” has stuck around anyway.