If your fruit is also often dry, with brown scaly patches and early fruit drop, this is blossom-end rot.
It's due to lack of calcium â€“ either because it's lacking in the soil or there has been drought or irregular watering so the tree couldn't take up enough nutrients to support the growth of the fruit.
It is a sign of malnutrition rather than disease and is easily treated â€“ although there's nothing to be done for fruit already afflicted.
The addition of lime (a handful or two per tree) around the drip line would be an effective stand-alone remedy, but you might as well cover all bases by applying citrus food too.
Citrus are among the greediest plants and are always starving for one thing or another.