Yes it is ok to remove slab after 4 days (usually - depends on swelling) - sometimes i cast the next day after a fracture but you have to make sure the swelling is stable (won't increase) or they aren't too swollen - then the cast will become loose. You will be fine - this is such a normal fracture by the sounds of it