I found this article helpful, when I has a similar question: Memory Scaling on Haswell
. I know it's talking about Haswell, and not Skylake, but the conclusions are likely to still hold true.
[The] importance of order is such that:
1. Amount of memory
2. Number of sticks of memory
3. Placement of those sticks in the motherboard
4. The MHz of the memory
5. If XMP/AMP is enabled
6. The subtimings of the memory
But it probably depends on your workload - if you need over 16GB RAM then obviously 20GB of slower RAM will be better than 16GB of faster RAM. But if that's not the case, then the opposite's true.
Assuming you're not frequently using over 16GB RAM I'd just get another 4GB stick.