The patella is a unique type of bone in that it is part of the quadriceps muscle - the big muscle in front of your thigh that straightens your knee. It is designed to improve the mechanics of the muscle as it moves around the trochea of the femur. This is basically a pulley arrangement where the inside of the patella and the muscle is like the cable around the fulcrum which is the femoral condyle. While I don't know what exactly has happened in the case of your fracture it is likely that the rehabilitation is about remodelling the knee cap to slide around the condyle of the femur. The fracture will have disrupted the smooth cartilage surface of the patella and you have to try to get the the surface has lost its smoothness and now your job is to strengthen the quadriceps muscle and also improve the flexibility of the muscle and to improve the smoothness of the patella where is contacts the femur.
In addition to the surface problem the tissues around the front of the knee may have probably contracted due to the immobilisation and the inflammation. These tissues have to be gradually mobilised and stretched out so that you can properly bend your knee.
This is where the pain comes in. You have to apply the appropriate forces to cause the beneficial remodelling, but it is going to be hard work. However there is no virtue in suffering the pain.
The swelling is probably due to increased fluid in the knee joint due to inflammation. One source of the pain is the inflammation. If you can reduce the swelling by such techniques as ice packs this may help. The ice also temporarily numbs the area so this can be a useful modality to combine with your quadriceps exercises that no doubt you have been given to do. Another modality is a battery powered electrical device called TENS that can modulate the pain while you exercise. Your physio can advise you on the use of TENS
As I said - no virtue to the pain so I would get some advice about managing the pain Non steroidal antiinflammatory drugs maybe one option to talk over with your doctor. but there are other options. I would get some professional advice about this however.
The physio has put the hard word on you about getting on with the rehabilitative exercise. They are most likely right and delaying the exercise is only going to prolong things and may result in a worse outcome. However getting some pain management is a great idea. You really do have to get going on it but there are no medals for being in more pain than you need to be.