Alright, After some more debugging, I found something strange.
My parent height was already set to 100%, so I tried changing that to 500px, yet the problem was still there.
So I tried heightSpec on fixed and set the height property to 500, the problem was still there.
After commenting every other property in my config, the problem was still there.
And then, in a Hail Mary attempted, I tried setting my config with the input config in the HTML instead of the typescript viewchild property.
Issue solved!
Do you guys have any explanation as of why this matters? I am setting the config based on the viewChild property in the afterViewInit because of performance reasons (as is described in the docs)
You can find a reproduction of the bug in the attachments.