To return to the old adage, a diamond is both hard, scientifically defined, and brittle. The steel of a hammer (hardness 5 or 6) will not scratch a diamond, but it can break it.
"Toughness" measures a material's resistance to blows. For gemstones, these measures are usually given in descriptive terms. It turns out that most gemstones would be considered "fragile". There are a few exceptions. Cryptocrystalline quartz, like chalcedony, has "hard" toughness. Jade, whether it's jadeite or nephrite, has "very strong" toughness.
What does the wearability of gemstones mean?
Assuming you don't regularly hit your jewelry with a hammer, scratches are a more common risk. Think about the number of times you put your hands in your pockets, handbags, glove compartments, and desk drawers. Now think about what brushes against a ring when you do these things.
The hardness of gemstones greatly contributes to the degree of wear of a gemstone. We often talk about wear or sometimes "durability". An opal with a hardness of 6 will quickly be covered with fine scratches and lose its polish if it is worn daily as a ring stone. Remember that quartz (hardness 7) is a component of dust and one of the most common minerals on Earth. Just wiping the dust off a softer material can scratch it. Scratches can be tiny, even microscopic, but they accumulate and become visible over time. In contrast, a ruby with a hardness of 9 will remain shiny and lustrous for years because it is harder than most abrasive particles that contribute to wear.