I wouldn't say it's definitely muscular from what you've said, the pain when arching sounds like it could be tight facet joints (spinal joints) that are painful on close-down. If you imagine your spine joints roughly sitting like this normally = , when you bend backwards the back of it closes down and gives you something like this < . Rubbish I know but best i can do and hope it makes sence!! You say you sit at a desk, its not uncommon for a few of these to tighten up and cause a bit of pain, which can in turn can cause muscle or other pain. That's one possible answer. Another is that you have simple low back pain which is extremely common, but if this is the case it's been shown to be quite important for recovery if you don't avoid things just because they bring on a bit of pain - pain isn't always a sign that you're doing more damage.
If you want exercises to try then you could start with a bit of lumber flexion stretches (bending your spine the opposite way to what you say hurts). An example would be standing and touching your toes, or lying on your back and pulling your knees tight into your chest. If you imagine it, both of these will bend the bottom of your spine the other way and open up the back of the joints, kind of like this > . Could also try some core strengthening exercises which have been shown to help with LBP.
This said, these are only good if that is indeed whats causing your pain, and theres now way anyone can know for definite if they don't see you in person so i'd still recommend going to see someone in real life if you consider it bad enough that you've searched for and joined a physio forum.