Preheat your oven to 400°F (200°C) and line a baking sheet with parchment paper.

Roll out the puff pastry on a lightly floured surface, then cut it into 2x3 inch rectangles.

Bake the pastry for 15-20 minutes until golden brown; let them cool completely on a wire rack.

Heat the milk in a saucepan over medium heat until it is steaming but not boiling.

Whisk together the sugar, cornstarch, and egg yolks in a separate bowl until smooth.

Pour the hot milk into the egg mixture gradually, whisking continuously to temper it; then return this mixture to the saucepan.

Cook the mixture over medium heat, stirring constantly, until it thickens and bubbles, then remove it from heat.

Stir in the butter and vanilla, and allow it to cool down.

Assemble the pastries by layering the cooled puff pastry with the pastry cream, finishing with a pastry layer on top.

Chill the assembled pastries in the refrigerator for at least 30 minutes.

Prepare the glaze by mixing the powdered sugar, milk, and vanilla until smooth.

Drizzle the glaze over the chilled pastries and garnish as desired.

Cut into portions and serve chilled.