Hi there and thanks for your very detailed thread.
It's always nice to receive a fresh view on someones history and yours does seem like a regular one we see in the clinic. It is very normal, so something to be aware of.
It appears from your history that what you are exhibiting is a overactivity (increased resting tone) in the lumbar musculature. We need not point out a specific muscle as that is not really relevant to the treatment approach. What we do know is that at times the lumbar extensor musculature is becoming overactive and that this contributes to discomfort/pain. Historically releasing of the tissues does ease the pain temporarily but it returns within a few days. And your soft mattress is another factor where you notice the discomfort.
So we need to ask, why is the lumbar musculature overactive? The painful response itself is perhaps normal so that is not primary issue, the issue is why your system behaving in this way. Most likely this is due to it having to overplay its supportive mechanism for the underlying skeletal structures (your spine). It could well be that you have a small Spondylolisthesis in the lumbar area, or perhaps an unstable facet joint on one side. Often larger Spondylolistheses can be felt in a physical examination however smaller, subtle defects might only be picked up on x-ray. When you have an issue like this repeated flexion of the spine (bending forward/knees to chest/plough in yoga etc), or stretching into lumbar flexion gives relief and 'feels good' for the lumbar region. Conversely repeated extension or overloading of the extensor muscles feels bad (think squats and deadlifts). We don't really need any invasive investigations if we can find a pattern that feels good as we know what we need to work on to get the extensors to 'relax' or reduce their hyperactivity/resting tone.
Most of the low back is stabilised by the abdominals muscles, especially (and most importantly) those that work in rotation - so think the obliques and transverse abdominus (don't think the rectus abdominus that is worked in the classic 'crunch' style sit-up). If your job is largely sedentary then for sure this is the area to work on. The more normalised resting tone and strength you can get into those rotational support muscles then more the lumbar extensor muscles will be able to 'let go'.
Yoga might be a good way forward in the first instance to give you a chance to work on the breath and the letting go whilst awakening the supportive muscles through movement. I tend to like this approach rather than a list of specific exercises. After all we can't (shouldn't) provide those without visual guidance - so an online forum is not the ideal place. There are some Pilates floor exercises like the rollovers with and without rotation that could be of use too, as would most pilates Studio equipment based classes or Gyrotonic equipment based classes. These later come at more cost though and are not as widely available so it depends where you are and your circumstances.
I think your mattress is just letting you know that your spine is not happy to assume the position that it would take once your muscles are more relaxed. Ideally it should be no problem so perhaps it can be used as a subjective feedback for if things are improving over time. Don't jump to a new one right now, see if you can make your spine work with the one you have. And if it's just too soft then sure, make a change.
Let us know how you get on. For now consider the idea of flexion movements, not passively, but as part of an active movement to develop connection and strength through movement. Passive stretches on their own will have no long term benefit.