Minor ankle fracture (lateral mallelous) is not healing

On 4/10 I fell and fractured my distal fibula. Went to the ER And they placed a splint on my foot and told me to go nonweightbearing. I saw the orthopedist on 4/15, I was instructed to wear a CAM Boot for four weeks, weight bear as tolerated and return to the doctor, at that time he originally hoped to have me start wearing shoes again, he took another x-ray on 5/15, and he instructed me to leave the CAM boot on for atleast another 10 days and start therapy ASAP to wean me out of the boot. I've been going to physical therapy now for two weeks. So far we haven't removed the boot at all. The physical therapist just finally got to review the x-rays today, and he said he could still see a fracture. He wants me to return to the doctor to get another x-ray, and doesn't want to remove the boot until he sees that is healed. I just got the chance to look at the x-rays myself, and it doesn't look like there is any healing it all to me.

I feel like the doctor keeps leading me on, at my first visit, he told me that it was a very minor fracture, and they would treat it like an ankle sprain and I should be able to be back into my regular shoes after 5 weeks. Then at my second appointment, he tells me I am healing fine (which seems like a lie looking at my X-rays) and that after 10 days i should be out of the boot and then on no restrictions after 3 weeks. How can this be that there has been no healing in such a small fracture (and it appears more pronounced now)? thank god my physical therapist insisted on looking at the X-rays.

Here is my xray (the left was taken 4/15 and the right was taken on 5/15):
Attachment 1242