Sounds to me like a Patello-femoral tracking dysfunction.
the main form of treatment would generally be to strengthen the quadriceps muscle group specifically VMO (vastus medialis obliqus)
It may be that basically you have always been strong which has meant you haven't had issues and then you stop doing gym / sport and the outer aspect of your quads is strong and tight and the medial aspect is weak. it can also be related to footwear and feet issues but usually if thats the case it will first happen when your a teenager and is usually fixed with orthotics (could be a place to look if strengthening doesn't work)
You can get braces that aim to hold the knee cap stable, they are usually fairly low profile things with a hole in the middle and velcro straps so that you attempt to position the knee cap properly. You can also try strapping the patella (if you google patella strapping i'm sure you'll find a diagram)
As far as quads exercises go basically any exercsies are generally good provided they do not cause your pain - straight leg raising with toes turned slightly out, cycling, leg extensions (low or no weight lots of reps) with toes out and if you can with a ball between your knees, leg press again low weight and again a ball between your knees...... the list goes on and on.