Hi guys I have been testing campaigns lately and I noticed something weird.
So i'm refreshing my creatives all the time to get good CTR, and I get roughly the same amounts of landing page clicks to go to the submit page.
How come that in the first few weeks the conversions were great, then suddenly I get the same CTR, same number of clicks but way way less conversions?
Thanks for your help!
There are several possible reasons here. Since you mentioned that you are refreshing the ADs, we can exclude the ad burnout. Let me give you a few possible options:
1. Competition changed on the source, so you are losing some of the good clicks to your competitors. This is usually also connected with changes in volume, but it doesnt have to be since the large push networks have plenty of publishers. So you might have lost some of the good placements and are getting more traffic from the poor ones. Try to breakdown the stats by placements to see whether you lost some of the well converting ones and there are some new with higher volume.
2. The source might have lost some good pubs, the data would look similar to what I mentioned in the above point ... placements that used to convert would be gone.
3. Something with the offer... it could be a tech problem or some optimization on their end due to low lead quality.
4. The new ads could be worse. CTR is an important but it's not the only metric to look at. You can have the best CTR ads but in case they don't convert, it's not worth much. Maybe the new ads are not compatible with the LP you are using. Try to throw more LPs into rotation.
5. PUSH traffic is generally known for rather unstable behavior, so the performance might return after a few days. Try to pause the campaign and restart later or make a new one (clone) to start fresh and have the algo treat it like a new one. This helps quite often.
There might be more reasons, let's see if someone else stops by to share their opinion. @erikgyepes maybe? 
There are ton of things that could happen and go wrong and @matuloo pretty much sum up the all possible scenarios, so I would probably just repeat everything. 
However, for how long period is this happening and what kind of conversion flow do you run (CC, SOI) and how big is the geo?
It's quite easy to burn through smaller geos, but you say you change creatives, so eventually the new creatives might not be as good which Matej pretty much said above.
Maybe go back to the creatives that were working first time? If you run them a while ago, they might be "fresh" by now again as the people on the list constantly change.
Also take a look at the placements and make sure you are still getting the same good traffic, some sources constantly add ton of new placements that are just eating up the budget with ridiculously low CTRs on the landers, checkout if you have any of those.

Ok, this is happening a lot, placements come and go... publishers also change networks often (though I'm not sure how often this is the case with push traffic), that's why it's a good idea to work with multiple sources and constantly search for new well converting zones/placements.
Don't forget that once you run an ad multiple times after some time it wears down a bit. You saturate your target audience with the same ads so in essence there isn't much to do about it. Don't forget that you run on the same source and the same users possibly. It has a huge factor in the reduced conversions you are getting. As Erik said. add some more traffic sources so you will always have a backup
Good luck mate!
If possible, rotate a similar offer on the same flow. (not sure which vertical you're running, but in many verticals you can easily find an offer that is 99% similar to the offer you're already running on a separate network)
Sometimes you will find that offer 1's CR drops like crazy, while offer 2's CR stays stable. (I nearly always run parallel offers on the same flow, and you'd be surprised how often this happens!)