There are a number of excellent resources around the UK and further afield. This page links to a few of them, please link to others in the comments
UK:
Natural History Museum, London. Physical and digital collection. Mineralogy collections | Natural History Museum
Corsi Collection of Decorative stones, University of Oxford. Physical and digital collection. Corsi Collection of Decorative Stones – Stones
John Watson Building Stones Collection. University of Cambridge. Physical collection. John Watson Building Stones Collection | Sedgwick Museum of Earth Sciences
Devonshire Marble Industry Torquay museum. Physical collection. Devonshire Marble Industry.
BGS Nottingham for the largest UK collection of geoscience samples, collection currently being digitised National Geological Repository – British Geological Survey
Europe
USA
Walker Zanger Building Stone Libarary at University of Pennsyvania. Physical and online collection Browse Stones | Walker Zanger Natural Building Stone Library – Architectural Conservation Laboratory