A desert island is an island with no people living on it. It does not mean an island covered in sand, and it does not refer to a hot desert climate.
In this expression, desert is connected to deserted, which means empty or abandoned. The idea is simple: a desert island is an island that has been left alone, or has no inhabitants.
This can be confusing because the spelling matches the noun desert, meaning a dry region like the Sahara. But in desert island, the meaning is different.
- desert island: an uninhabited island
- deserted street: a street with no people on it
- desert as a noun: a dry, sandy or rocky region
For example, you might say, After the storm, they were stranded on a desert island. You would not use the phrase to mean a beachy island full of people and resorts.
So if you picture a palm tree and empty shores, the key idea is not the sand. It is the absence of people.