Preheat oven to 350°F (175°C). Grease and flour a 9-inch square baking pan. Set aside.

Prepare the Streusel: In a small bowl, whisk together flour, brown sugar, slivered almonds, and cinnamon. Cut in the cold butter using a pastry blender or your fingertips until the mixture resembles coarse crumbs. Place in the refrigerator while preparing the batter.

Prepare the Batter: In a medium bowl, whisk together the 2 cups of flour, baking powder, and salt. In a separate large bowl, cream the softened butter and sugar until light and fluffy. Beat in the eggs one at a time, then stir in the vanilla extract.

Alternate adding the dry ingredients and the buttermilk to the butter mixture, beginning and ending with the dry ingredients (3 additions of dry, 2 additions of buttermilk). Mix only until just combined. Do not overmix.

Assemble the Cake: Spread half of the batter evenly into the prepared pan. Dollop spoonfuls of the apricot jam over the batter, and sprinkle half of the 1/4 cup slivered almonds (if using for the swirl). Top with the remaining batter, then spread the remaining jam on top.

Topping and Baking: Sprinkle the chilled streusel mixture evenly over the top layer of jam. Bake for 40-45 minutes, or until a wooden skewer inserted into the center comes out clean.

Cool completely on a wire rack before slicing and serving.