Bring several cups of water to a simmer in a medium pan to create a waterbath. Place the egg yolks in a medium bowl and place over the pan of simmering water.
Whisk in 1/3 cup of sugar. Continue whisking until the mixture looks pale yellow and has thickened, about 4-5 minutes.
Add the milk, brandy, rum and nutmeg to the egg mixture and whisk to combine. Pour mixture into a large bowl and set aside.
In the bowl of a mixer, fitted with a whisk attachment, whisk egg whites and remaining sugar to a soft peak stage. They should hold their shape but are not stiff.
Fold the egg whites into the egg yolk mixture. Whisk heavy cream in a medium bowl until it thickens and fold into the egg mixture.
Refrigerate thoroughly, at least 1 hour, and whisk just before serving.