Thanks for the reply !
I went to a physiotherapist but it wasn't very helpful for the problem. She just released all of the trigger points I had around the shoulder and trapezius.
I'm going to try an orthopedic/sport doctor as you suggested and see if it can give me a better understanding of the problem .
I am also in agreement when you talk about the ligaments, it feels maybe they don't hold the end of the bone in position. I hope I didn't rip them off but I feel no pain at all.
Anyway, I'll give my feedback if might help other people !

PS : Also, my clavicle looks like it is a bit more "out" under the neck than the other one. That said, my two clavicles are very dissimilar in shape (my torso is also very asymetric) so this might not be useful evidence.