WPF leaks memory when using separate STA Thread

Recently, I was asked to tackle down a memory leak problem in WPF application. The problem was so not obvious and so mysterious, that triggered bigger research on a topic.