Whisk together the egg yolks and sugar until pale in color. Set aside.
Combine the milk and heavy cream in a saucepan and heat over medium heat until just steaming.
Pour about 1/4 cup of the warm cream mixture into the eggs and whisk, bringing up the temperature of the eggs.
Add the tempered egg mixture to the saucepan and cook over medium heat, stirring often, until slightly thickened, about 5 minutes. Remove from heat.
Add in the vanilla and grate in the nutmeg.
Chill in the fridge until completely chilled, about two hours.
To serve, add brandy or rum to taste if using and ladle into glasses.
Top with a fresh grating of nutmeg.