Hello Mark,
By your description i think you're right, you have a mite infestation.
Mites thrive on hot dry conditions, which probably your case, also an uncontrolled increase in population would indicate the absence of predatory mite.
I imagine by your description that you have only one olive tree, probably in a garden?
If this is the case I would recommend you avoiding any dangerous pesticides, you can use summer oil (
Horticultural oil - Wikipedia, the free encyclopedia) to control the mite population, applying it particularly in the areas where the mite concentrates. But pay attention these treatments also will kill the predatory mite, so in a few weeks/mouths the mite population will build up again.
You can also try and introduce some predatory mite in your olive tree, like is described here:
Achieving Biological Control of European Red Mite in Northeast Apples
Hope I've helped.
Cheers,