The overall loading speed of a landing page will indeed depend on the size of it, the "bigger" the LP the more time its gonna take to load it.
Your loading speeds are more than fine according to my experience, 220ms for example is a very good result, 650ms is still a good one too.
I'm pretty sure the higher values were happening when testing from locations further from your hosting, so thats normal too.
Affiliates tend to give loading speed too much weight, expecting big increases in performance with higher loading speeds and Im sorry but thats not the case.
You definitely need your lander to load very fast, absolutely no point to argue about this, but going from very fast to super fast wont make a big difference.
In other words, if you improve the loading speed from 350 to 250ms, the difference in performance will most likely be none.
On the other hand, taking it down from 2 seconds to 500ms should mean a significant change for the better.