Lightly grease a 9x13-inch baking dish. Arrange the cubed brioche evenly inside the dish.
Whisk together the eggs, whole milk, heavy cream, granulated sugar, brown sugar, vanilla, cinnamon, nutmeg, and salt until completely smooth.
Pour the custard evenly over the bread cubes, gently pressing them down so every piece absorbs the mixture. Cover and refrigerate for at least 6 hours or overnight.
Combine the flour, brown sugar, cinnamon, and salt in a bowl. Cut the cold butter into the mixture until coarse crumbs form, then refrigerate until ready to use.
When ready to bake, remove the casserole from the refrigerator while the oven preheats to 350°F. Sprinkle the crumble topping evenly over the soaked bread.
Bake until the center is set, the topping is golden brown, and the edges are lightly crisp.
Allow the casserole to rest for about 10 minutes before serving. Dust with powdered sugar and drizzle with warm maple syrup if desired.