Campaign automation with Python (50)
Welcome to my first follow along ever 
I don´t know how regularly I update it because it´s more of a fun project and not mandatory for my work but I do my best to update as often as possible.
Before I start let me first tell you a bit about my motivation to run the FA.
Main reason is that I really would like to see more action in our programming and coding section.
Of course it´s no easy task for me to post new content there when I am no coder myself but for many years already I want to learn coding.
Not for big projects or so, just for myself because I enjoy it to program own stuff, expecially when it´s helpful then 
I don´t have a coding background at all and even today I am not able to make a simple website from scratch but when you work with affiliate stuff for 15 years then of course you still learn a bit of this and a bit of that.
This basic knowlegde and basic understanding of coding logic was enough that I started working on my own cloaker in 2009 or 2010.
It was a very basic and 100% PHP based cloaker and I improved it many times in the following years.
I learned all I needed for it along the way, learning by doing.
That way I didn´t have to learn things that I don´t need and could learn exactly what was needed in my situation.
I also took some other scripts and changed them a good bit so that they work for my stuff.
Nonetheless I always wanted to learn more about coding but never got deeper in it.
My main goal is to work on automation scripts to support my lazyness and to make my daily affiliate life easier.
This is when I thought about different programming languages.
Usually my decision would be PHP because there I know a bit already.
But for the stuff I want to do I thought that Python probably is better for it because it has many good libraries for AI, machine learning and other scientific stuff already and from what I read it´s very good for automation anyway and also easy to learn.
Somehow I still didn´t find the motivation to finally start working on some stuff.
Recently I learned that @jeremie and @bennimen have some coding background and have scripts for their own affiliate stuff and this got me a bit more motivated.
It seems the good ol´ @jaybot also know a little bit about coding or even Python.
Right now my biggest obstacle is to find the best way to start at all.
I use Sublime to write the code and then run it in terminal.
But I would prefer to use an IDE where I can write, run and debug all in one interface.
I tested VS Code and Pycharm, on VS code I somehow can´t get the integrated terminal working so I can´t run and debug my code there.
Pycharm works ok so far, I tested the community version.
@bennimen also recommended Jupyter Notebooks and he uses it himself but before I get used to it I think I better try to get used to a real IDE 
Will try to find the best way for me but until then I just use Sublime and terminal.
Next thing is about all the terms and stuff, what is needed and how I use it, interpreters, functions, modules, libraries.
What is included in Python already, what do I have to download/install?
Basically all these pain everyone has when we start something new and I am confident that the fun starts when the basics are clear
Alright, thanks for still reading after the long prologue.
Now let´s start with the project itself.
My plan is to get some automation for my campaigns running so that I have more time for other things.
For blacklisting placements I already use TheOptimizer so what I would need is to have some automation for my tracker stuff = landing pages and offers.
To keep it as simple as possible in the beginning I will start with direct linking campaigns where I test many offers at once.
The
I guess it gets a bit more tricky when I have to update the campaigns then later so that I have to post data back through the API.
Special thanks to @bennimen for sending me a script for reference so that I can check the code a bit.
What I did so far is to grab the all-time stats for a specific campaign group in
So this little code snippet is the beginning of my journey to tracker automation
import json
import requests
response = requests.get("http://mydomain.com/index.php?page=Campaigns&user_group=all&status=2&group=40&traffic_source=all&date=9&timezone=+2:00&api_key=##my_api_key##")
db = json.loads(response.text)
items = []
for item in db:
items.append(item['name'])
for x in items:
print(x)
Then I will make the script to loop through all these campaigns stats and pause offers based on simple rules.
Will update with some fresh code later
Following! Looking forward to contributing and give back some...
You know, the requests.get comes with json method which automatically converts the response into a json object.. so your code can look like this instead:
import requests
response = requests.get("...")
for item in response.json():
print(item['name'])
Haha awesome.
I feel like I need to change my name to benniemen now too.
One thing I noticed with my scripts was that I couldn't run multiple scripts calling the
If I'm manually running my code, I try to time it during times that I know my auto scheduled code isn't running.
With that in mind, there could possibly be some interference between your code and TheOptimizer. But there also might not.
Sent from Maui using Tapatalk
Originally Posted by diplomat
Thanks for that
Originally Posted by bennimen
Originally Posted by bennimen
Well, right now I am just playing aroud a bit with some code, trying to get my feet wet.
I am also still thinking about the best way for me to write and execute the scripts.
With VS Code the problem is that I somehow can´t get a linter running and the integrated terminal also won´t work.
Spent too much time on it already so VS Code is checked off.
Now I have Pycharm and Thonny.
Pycharm seems to be very good although I probbaly won´t use all features but what I don´t like is that the community version doesn´t support SQL becaues I am pretty sure I will need SQL functionality rather sooner than later.
Thonny is free and looks very basic but somehow I like it.
You can write the code there and run it inthe integrated shell and what I really like is that you can add an additional window where you can see the variables with their values.
I think I will stick with Thonny or Pycharm, seems they both have what I need and are good to work with.
It also seems that I slowly start to understand what the different environments are.
Anyway, will update when I have some more code to show
Ok, little update for today.
I created a directory "offerstats" where the script creates a *.csv file per campaign.
Then the script writes all offers to these files, one offer per line.
I had one little issue that sometimes I only received a "status error" for a campaign instead of the offers.
I thought that maybe it has to do with parallel access to the API when TheOptimizer also calls the API at the same time.
This is why I added the "while" loop to grab the offer stats again when there is the word "status" in the response string.
Now it works pretty good so far.
At the beginning of each run the script first deletes all *.csv fies in the offerstats folder and creates the files then with new data again.
And here´s the updated code
import requests
import os
import glob
fileList = glob.glob('offerstats/*.csv')
for offer in fileList:
os.remove(offer)
# Get all time stats for specific campaign group
stats = requests.get("http://mydomain.com/index.php?page=Campaigns&user_group=all&status=2&group=40&traffic_source=all&date=9&timezone=+2:00&api_key=##my_api_key##")
# Get campaign IDs
for item in stats.json():
campid = item['id']
# Get offer stats from the campaign, run again in case of error
offerstats = requests.get(f"http://mydomain.com/index.php?page=Stats&camp_id={campid}&group1=3&group2=1&group3=1&date=9&api_key=##my_api_key##")
for offer in offerstats.json():
while "status" in offer:
offerstats = requests.get(f"http://mydomain.com/index.php?page=Stats&camp_id={campid}&group1=3&group2=1&group3=1&date=9&api_key=##my_api_key##")
offer = offerstats.json()
f = open(f"offerstats/{campid}.csv", "a")
f.write(str(offer) + "\n")
I guess it´s a good time to get used to some new Python modules
This while loop is going to "spam" the API and may get you banned if you abuse it.
requests has a retry class that is designed for that:https://stackoverflow.com/questions/...s-library?rq=1
You may also want to limit your fetch rate:
https://pypi.org/project/ratelimit/
Originally Posted by jeremie
Hey twinax,
Checkout https://editor.swagger.io/
Using GenerateClient, you can quickly generate API libraries that generates full objects and factories, as well as establishing connections and making requests.
I used it for funnelflux and propellorads api to automate bidding and camp management based on data.
The advantage of having a client with actual objects and getters/setters lets you work quickly on the IDE ($campaign->getId, $campaign->getRate etc..) and also automate the validation of fields, data types etc..
Originally Posted by jeremie
Just a note with what twinaxe is doing, since he's calling his tracker on his own self hosted server, the api access most likely won't get banned. I don't think
They do have a temporary account suspension if the wrong api key is called too many times though.
A ban could possibly and likely result for calling the api of another service outside of their acceptable rate limits though, maybe like affiliate network's or traffic network's.
Sent from Maui using Tapatalk
Originally Posted by mywebexperts
For now it looks pretty complicated for me but I will check it when I all the API stuff later.
Originally Posted by bennimen
But I changed the script a bit again.
I removed the while loop because I tested the script several times in a row and the status error issue din´t happen again.
Yesterday I already checked Pandas to then store the data in Pandas.
When this is done I will try to come up with an idea for the rules.
Didn´t abandon the FA, just had some other things to do and still need to figure out how to use Pandas and Numpy and trying to come uo with a good and easy system to pause offers
IMO what's a downside in TheOptimizer is that they don't work with every traffic source, this feels abit limiting to me.
Have you thought about also creating an optimization automatic program that will work with all traffic sources? allowing you to automate also non popular networks? ones which traffic might be cheaper, less competitive, potentially more volume?
lol, this thread is nice. @twinaxe If you are still working on this pythonic tool hit me up on PM
Originally Posted by roiter123
What's number of traffic sources that are missing, and you would like to use?
Coding this kind of tool for STM-use is doable, it wouldn't be THAT much advanced as paid solution but could be enough for your needs.
Originally Posted by plutus
Having something like that would be amazing bro.
It´s been some time but now I started working again on the project.
Stay tuned for updates in the next days.
If you need any help, hit me up on Skype!
Love this FA idea!
Originally Posted by ScottyG
For now it doesn't look that bad, only had a hard time today to deal with the JSON response from the API.
In the end it was my fault because I made it too complicated.
That costed lots of time and nerves but at least now I understand what are lists and what are dictionaries and how to work with it.
The script grabs now campaign IDs from the JSON response, adds them to a list, loops through the list to get LP Infos from each campaign.
Will do the same for offers in these campaigns.
Next part will be to apply rules and let the script do some calculations, then pause landing pages and offers should they meet the conditions for it.
I hope that can be done in 1-2 days but first I need to get the other
I already have plans for further automation, next step will be to pause campaigns based on rules.
A little bit like The Optimizer yet much more basic but therefore very specific for these campaigns.
When this is done I plan to add a feature to take profitable campaigns and create them for other ad formats (for example take a profitable push campaign and run it on pops and in-page push as well).
These new campaigns will then also be automated.
Will probably need to prepare a little creative database for it but it all should be possible somehow
Should I really get it done I can also add more trafficsources for scaling.
The end goal then is almost 100% automation of creation, testing, optimization and scaling of campaigns.
This could also be a real game changer for click2call offers where carrier and country performance can change very fast.
With an automated system it would be much more effective to only run the high performers.
But now I first want to get the offer/landing page part done, then I can think about the rest
Originally Posted by twinaxe
For now it doesn't look that bad, only had a hard time today to deal with the JSON response from the API.
In the end it was my fault because I made it too complicated.
That costed lots of time and nerves but at least now I understand what are lists and what are dictionaries and how to work with it.
The script grabs now campaign IDs from the JSON response, adds them to a list, loops through the list to get LP Infos from each campaign.
Will do the same for offers in these campaigns.
Next part will be to apply rules and let the script do some calculations, then pause landing pages and offers should they meet the conditions for it.
I hope that can be done in 1-2 days but first I need to get the other
I already have plans for further automation, next step will be to pause campaigns based on rules.
A little bit like The Optimizer yet much more basic but therefore very specific for these campaigns.
When this is done I plan to add a feature to take profitable campaigns and create them for other ad formats (for example take a profitable push campaign and run it on pops and in-page push as well).
These new campaigns will then also be automated.
Will probably need to prepare a little creative database for it but it all should be possible somehow
Should I really get it done I can also add more trafficsources for scaling.
The end goal then is almost 100% automation of creation, testing, optimization and scaling of campaigns.
This could also be a real game changer for click2call offers where carrier and country performance can change very fast.
With an automated system it would be much more effective to only run the high performers.
But now I first want to get the offer/landing page part done, then I can think about the rest
Highly recommend you check out Firebase for a database and use the realtime if possible!
I like about it that it also works with JSON.
Why do you recommend Firebase?
What's the advantage over MongoDB?
I wanted to avoid a cloud solution.
Originally Posted by twinaxe
Also you store JSON directly. Plus GET requests can be 7ms instead of 2s.
Mainly I created a snippet in VSCode using an online snippet generator that allowed me to press a hotkey and instantly fill out the path I want the data stored and press tab and everything is written for you.
If you don't really care about how fast you write code, MongoDB is a solid option as well.
Speed isn't that important, I am the only one using the script and few ms more or less won't make a difference.
But good to know about Firebase, will kepp it in mind for the future.
Btw, what kind of stuff are you coding and what languages?
Originally Posted by twinaxe
My preferred stack is React/VueJS (TypeScript) - Node JS (TypeScript) - Firebase for most web apps.
Got an interesting idea for a new app but don't have enough funds to properly develop and launch it at the moment.
Other than that, I work with drones using AI to discover mineral and oil deposits at my 9-5.
Was pure PHP and I had absolutely zero knowledge about that stuff.
I just had the idea, started working on it and used it for 7 years or so, always improved it in that time.
I also thought about NodeJS some time ago but in the end I decided to go with Python.
As I already said in the opening list, I would really enjoy it to see more activity in our coding section
Originally Posted by twinaxe
Anyway. I like mongodb as it will take basically anything as an object and stick it in its database. It’s fast and works great for js node stuff.
I have never used firebase, but I know clickfunnels uses them (try ripping and cleaning a CF funnel and find out) so it definitely works for large and small projects.
Both should work fine.
Yesterday I had to deal with some of my "favorite" tasks, APIs and databases.
It was pretty frustrating at some points where I just didn´t know what went wrong and how to solve it but I am like a pitbull, when I get my teeth into such stuff I don´t release until the problem is solved.
So I stayed on my PC till 3am, did a lot of trial and error and finally I got it done.
Here´s what the script does so far:
- Get all tracker campaigns from a specific group
- Filter the campaigns for a specific keyword
- Loop through the campaigns to get info: ID, name, country, revenue, trafficsource campaign ID
- Add these info to the database. If the campaign is in list already it updates the values, if it´s not in the list already it inserts
- Connect to the trafficsource API to get cost for the campaigns
- Based on cost and revenue it calculates profit and ROI
- Add cost, profit and ROI to database
Doesn´t look that much yet but I also get more and more ideas the more I work on it.
And considering that I never ever did anything with Python or APIs or setting up own DBs before I think it´s pretty good
Todays tasks will be to go through the campaigns to get info about LPs and offers and add them to the DB.
Then I will think about some logics for the automation.
At all of you, I also think about making a frontend to display some stats in tables but probably also with graphs.
I am not sure if Flask or Django would be better for it.
Flask is more lightweight and I can be more flexible there, Django seems to offer more options out of the box but is more strict in it´s structure and modules.
If you have any input about these 2 frameworks I am all ears.
Originally Posted by twinaxe
It was pretty frustrating at some points where I just didn´t know what went wrong and how to solve it but I am like a pitbull, when I get my teeth into such stuff I don´t release until the problem is solved.
So I stayed on my PC till 3am, did a lot of trial and error and finally I got it done.
Here´s what the script does so far:
- Get all tracker campaigns from a specific group
- Filter the campaigns for a specific keyword
- Loop through the campaigns to get info: ID, name, country, revenue, trafficsource campaign ID
- Add these info to the database. If the campaign is in list already it updates the values, if it´s not in the list already it inserts
- Connect to the trafficsource API to get cost for the campaigns
- Based on cost and revenue it calculates profit and ROI
- Add cost, profit and ROI to database
Doesn´t look that much yet but I also get more and more ideas the more I work on it.
And considering that I never ever did anything with Python or APIs or setting up own DBs before I think it´s pretty good
Todays tasks will be to go through the campaigns to get info about LPs and offers and add them to the DB.
Then I will think about some logics for the automation.
At all of you, I also think about making a frontend to display some stats in tables but probably also with graphs.
I am not sure if Flask or Django would be better for it.
Flask is more lightweight and I can be more flexible there, Django seems to offer more options out of the box but is more strict in it´s structure and modules.
If you have any input about these 2 frameworks I am all ears.
Also you're already staying up to 3am coding?! You're basically a full time programmer already!
Thats a HUGE update for one day, pulling everything together!
I also made my first webapp with Flask few days ago.
Update:
Yesterday I added code to get tracker stats for landing pages and offers in each campaign.
The script gets the LP/offer name, the path_com (will be needed later), nr. conversions, conversion rate, the path and if it´s active or paused.
Then it adds all these values to the database.
Even when I get stuck here and there it´s still amazing to see how "easy" all that stuff is compared to what I thought how it will be.
Today I added code to get the current date as well as the first day where a campaign was started, with these 2 values I calculate for how many days the campaign is running already.
This will be needed at a later point when I start working on scripts to sort and display stats.
For now I try to collect as many different metrics as possible because with automation it can easily generate reports that I never could do manually because it just would take too much time to go through everything myself.
The collection right now looks like this:
{
'_id': ObjectId('xxx'),
'Campaign ID': 'xxx',
'Campaign Name': 'xxx',
'Country': 'xxx',
'Days Running': 'xxx Days',
'Conversions': 'xxx',
'Active LPs': 'xxx',
'Active Offers': 'xxx',
'Trafficsource ID': 'xxx',
'Cost': 'xxx',
'Revenue': 'xxx',
'Profit': 'xxx',
'ROI': 'xxx%',
'Landing Pages': [{
'LP ID': 'xxx',
'Name': 'xxx',
'Conversions': 'xxx',
'Conversion Rate': 'xxx%',
'Status': 'Active',
'Path': 'xxx'
}, {
'LP ID': 'xxx',
'Name': 'xxx',
'Conversions': 'xxx',
'Conversion Rate': 'xxx%',
'Status': 'Active',
'Path': 'xxx'
}, {
'LP ID': 'xxx',
'Name': 'xxx',
'Conversions': 'xxx',
'Conversion Rate': 'xxx%',
'Status': 'Active',
'Path': 'xxx'
}, {
'LP ID': 'xxx',
'Name': 'xxx',
'Conversions': 'xxx',
'Conversion Rate': 'xxx%',
'Status': 'Paused',
'Path': 'xxx'
}],
'Offers': [{
'Offer ID': 'xxx',
'Name': 'xxx',
'Conversions': 'xxx',
'Conversion Rate': 'xxx%',
'Status': 'Active',
'Path': 'xxx'
}, {
'Offer ID': 'xxx',
'Name': 'xxx',
'Conversions': 'xxx',
'Conversion Rate': 'xxx%',
'Status': 'Active',
'Path': 'xxx'
}]
}
One thing that I realize is that coding such stuff yourself is amazing because I can collect, calculate and store exactly the stats that I want to have.
That´s definitely a huge advantage over public software where you just have to work with what´s given.
often have good ideas that I can´t use when the software doesn´t support it.
With this new freedom I feel like I could achieve so much more, even if it´s not profitable at the end of the day it´s still lots of fun to learn how to do these things.
Today I will start working on the rules to pause LPs and offers and hopefully get it running soon so that I can run first test campaigns.
Few more things that I want to implement later:
- A Telegram bot to message me in case of errors or other situations
- Script to regularly get performance stats from CPA network for geos and carriers.
I will store it all in a database then to get a better overview about geos and carriers that are top performers more often or to see which are showing god performance more longtime.
- Collect stats about bids to find the sweetspot
- Collect stats about user freshness for push campaigns
I also think about defining functions for the different code blocks so that I can use them as modules.
That way it´s probably easier when I want to re-use them in other projects (and this will probably happen).
Will also optimize my campaign name structure in
Originally Posted by twinaxe
Even playing around in postman you can see just how "easy" using the API to get and send values is.
I admire your, well, your everything really, but right now I admire how you're taking this head on.
Next comes the fun part
Time for an update.
Maybe you have seen it already but I stopped THIS project because there are few issues that are out of my control so I better work on a bigger project that can be used for many more things.
In this project I am also not much further in terms on progress because right now I think about the best way to build the scripts that they can be used for many different campaigns and not only for IVR or only for TC Top offers.
I did some brainstorming the last days and started a mindmap to get a better idea how I want to continue.
Right now my project structure is like this:
├── create_campaign.py ├── create_scripts │ ├──Binom .py │ ├── trafficsource.py │ ├── scale.py │ └── test_campaign.py ├── db_check ├── opti_scripts │ └── stop_campaigns.py ├── optimization.py ├── stats │ ├── first_entry.py │ ├── get_ts_campaign_ids.py │ └── stats_update.py ├── stats_update.py └── tc_api.py
The files inside the folders are imported as modules and the scripts from the top level basically only call functions from these modules then.
For example in create_campaign.py is only
from create_scripts import test_campaign from create_scripts import scale test_campaign.run_test() scale.scaling()
For specific tasks like using the scripts only for IVR offers it may work but when I want to use it more flexible for other campaigns as well I guess it´s a good idea to structure it as bit better.
I think it´s a good idea to bundle "similar" scripts as functions in *.py files that I can then just import as modules.
My idea would be something like this:
├── create_campaign.py ├── binom.py ├── trafficsource1.py ├── trafficsource2.py ├── optimization.py └── tc_api.py
create_campaign.py would be with functions to create either test campaigns or campaigns for scaling and also functions to prepare everything for the different trafficsources, ad formats, bid options and so on.
binom.py would be everything binom related like grabbing stats, adding offers/LPs/campaigns, update cost etc.
trafficsource1-xx.py would be one file per trafficsource with different functions inside for tasks related to that trafficsource.
These could be campaign creation, stopping campaigns, add placements to BL, get stats.
Also functions for the different ad formats and bid options the trafficsource has.
That way it should be easy to add new trafficsources later without changing much code in other files.
optimization.py is for *surprise surprise* eveything optimization related.
The metrics for the automated optimization of different campaigns can either be grabbed from the DB or run by rules.
In the end I think this could work pretty well when I build it in such modular structure, each platform as an own module to keep it separated.
This should work good when I then want to extend the scripts so that I can just write a new module for a new platform and import it in the already running scripts.
But then I searched a bit about Python project structure and got confused.
On THIS site the dude explains his project structure pretty nice and also shows examples but I still don´t really get it.
In his example the project structure is like this:
omission-git ├── LICENSE.md ├── omission │ ├── app.py │ ├── common │ │ ├── classproperty.py │ │ ├── constants.py │ │ ├── game_enums.py │ │ └── __init__.py │ ├── data │ │ ├── data_loader.py │ │ ├── game_round_settings.py │ │ ├── __init__.py │ │ ├── scoreboard.py │ │ └── settings.py │ ├── game │ │ ├── content_loader.py │ │ ├── game_item.py │ │ ├── game_round.py │ │ ├── __init__.py │ │ └── timer.py │ ├── __init__.py │ ├── __main__.py │ ├── resources │ └── tests │ ├── __init__.py │ ├── test_game_item.py │ ├── test_game_round_settings.py │ ├── test_scoreboard.py │ ├── test_settings.py │ ├── test_test.py │ └── test_timer.py ├── pylintrc ├── README.md └── .gitignore
Then there is the tests directory.
I already read about it on few other websites and don´t understand for what it´s good.
I write my scripts in PyCharm and test them in PyCharm.
When everything is working the way I want to have it I put it on my server.
I am also working alone on the stuff so in my opinion I also don´t need a test directory.
Then there are the app.py, __main__.py and __init__.py files.
Maybe my mind just isn´t made to understand these things but I don´t really get what they are used for.
In the __main__.py is only
from omission import app
if __name__ == '__main__':
app.run()
That way I can run my scripts by just calling the directory and not the file itself.
But where´s the advantage of typing
python -m omission
python app.py
When I have several *.py in a directory it´s just a directory with *.py files.
When I have several *.py files in a directory and add an empty __init__.py to the directory then the directory becomes a package.
What´s the advantage of doing so?
I can also just put my *.py files in a directory without a __init__.py file and still import them as modules so where´s the advantage of making a package from the directory?
My old structure is already working fine, when I would add __init__.py now to the directories they would become packages, would it change anything for my scripts?
Next thing is modules and functions vs classes and methods.
Seriously, I don´t get it as well.
Where´s the big advantage of classes and methods over modules and functions?
As far as I understand "methods" is exactly the same as "functions", it´s just a different term.
Right now I am pretty pissed off that I even started reading about these things.
In the end my goal was to have a better structure for my project to keep it overseeable and to makes it easier for me to implement new platforms/modules without having to rewrite everything.
Result is that I read about lots of stuff that just doesn´t make sense for me or that I simply don´t understand.
I absolutely get it that such structure as in the example above can make sense when you work in a team or when you want to publish your project but is it really needed when I work on it all alone and don´t plan to make it public anyway?
Do I have a disadvantage of just building it the way that I think it´s best for me instead of following the generally accepted Python standard?
Maybe our Python pros like @jeremy can tell me a thing or two about my own planned structure vs example structure above as well as modules and functions vs classes and methods.
In the meantime I will still work on the code because no matter which way I choose, the code can be used for it anyway
I am rather rookie at python. So take what I write with a big grain of salt.
Let's say you want to do different things with you
You could write several functions for each camp related task, like:
binom_camp_create, binom_camp_add_path, binom_camp_change_path.... etc.
Problem: Maybe you name one of your functions "binom_campaign_do_something" --> then you may have struggle to find that functions in a bigger project or when importing a py file.
With classes you would just create a binom_camp class as a container. In that container you put your "functions". When you do it like this it will be much "cleaner".
In a class it would look like this: binom_camp.path.add, binom_camp.path.change... etc.
When you type in PyCharm "binom_camp." --> then it would auto suggest you all the methods available.
Also you can store data in your classes and reach them from everywhere (once the class is instantiated). Without classes you would need to use global variables (which are a pain).
And then again you have the advantage of name structure in your class for your data. Because you would call it like binom_camp.ID, binom_camp.path.ID
Punchline:
You don't need classes. Especially for smaller scripts not using them probably is way faster. But once things get a bit more complex they could help a lot.
Classes do have many other advantages like polymorphism and stuff... which most likely are overkill if you dont need it.
I read your words, I know what you mean but I still don´t really see why it could be better for my project at this point.
With classes you would just create a binom_camp class as a container. In that container you put your "functions". When you do it like this it will be much "cleaner".
In a class it would look like this: binom_camp.path.add, binom_camp.path.change... etc.
Then I would just import "binom_campaign.py" as a module and call the function with "binom_campaign.do_something", where´s the difference or real advantage of classes and methods there?
I could imagine that I don´t really see a sense in it yet because of how my project is working.
In the end it´s just several scripts that run through crontabs every x minutes and every run follows basically exactly the same steps.
All these things make probably more sense when I run it as a web app with user input, navigating through pages and so on.
This is also when the __main__.py would make sense so that I can call my script with https://mydomain.com instead of https://mydomain.com/myscript.py
Long story short, as long as I run my project only on small scale and run the scripts only from commandline it probably won´t make a difference for me if I run it my way or with a "real" project structure and classes but when I scale it and run it with frontend it´s probably better to switch to more mature Python
I plan to continue working and improving it for long time so I better get used to these things already so that I don´t have to rewrite everything at a later point when the project is already grown bigger.
The work that I invest in the beginning is work that I don´t have to do later.
I guess that's for me, as @jeremy was last active in 2014
Your structure is fine if it fits your needs.
I already gave my 2 cents about classes here (as well as on post 36/37 of the same thread):
https://stmforum.com/forum/showthrea...l=1#post418905
Maybe you should re-read it now that you are a bit more advanced in python to see if it makes more sense.
Originally Posted by twinaxe
Look at the URL testing unit of Hugo for example
https://github.com/gohugoio/hugo/blo...rs/url_test.go
See more about that here:
https://www.atlassian.com/continuous...omated-testing
https://docs.python.org/3/library/unittest.html
I don't use it. And I don't think you need that.
Originally Posted by twinaxe
With crons and modules, you have a linear way of programming. It will loop through all campaigns, fetch data and make optimization. That is fine when everything is fixed, with the same schedule. Classes give more flexibility
I am happy that it found the way to the one I was looking for
https://stmforum.com/forum/showthrea...l=1#post418905
Maybe you should re-read it now that you are a bit more advanced in python to see if it makes more sense.
But for example
I found this on Reddit and I think that´s pretty good:
"Classes for the sake of classes is pointless, and doesn't help testability at all."
Yes, there are classes and methods and there is a reason why they exist but same goes for modules and functions.
I sat down today (again) to get a better idea what I want to achieve with my scripts and how I want to achieve it.
I think I found a good structure for my project and for now I will stick more or less with what I did before, modules and functions instead of classes and methods.
A difference is that I will structure my scripts better/smarter but when I really need to use classes I will probably know that I should do so.
Form me most important now is to don´t procrastinate too much because I can´t decide between modules vs classes, I can still check it out when I really need it.
I already ported the first scripts to my "new" project so I hope that I can showcase some new stuff here soon.
Stay tuned
@jeremie I re-read your posts about classes and I have the feeling it finally clicked a bit and I see few things where classes could be helpful for my project.
1) group functions (called methods in a class) that share the same purposes
Generally speaking in my case a module is basically exactly the same as a class.
I can import both and both will encapsulate data and values.
My modules consist of functions and code outside of functions that is executed each time I import the module, classes consist of methods = functions and can also have code outside of the methods that is executed each time I import the class.
When I start working with classes it would be like having several different modules combined in 1 module.
Let´s say my module is example.py:
def helloworld():
print('Hello World')
print('Outside Text')
class ExampleClass:
def helloworld(self):
print('Hello World')
print('Outside Text')
One question about the __init__() function, what´s the difference between
class MyClass: var = 123 x = MyClass() print(x.var)
class MyClass:
def __init__(self):
self.var = 123
x = MyClass()
print(x.var)
- I make a class PropellerAds where I declare all variables and general stuff that is used for all tasks on Propeller Ads
- From class PropellerAds I make subclasses GetStats, StartCampaign, BlockZone and so on.
These subclasses have automatically all functionality of the parent class and in these subclasses I then set variables and add methods for these specific tasks
- From subclass StartCampaign I then make subclasses for the different ad formats like Pop, Push, Interstitial
So when PropellerAds introduces a new ad format like banners I would only have to prepare a new subclass "class StartCampaign(Banners)" instead of writing the whole codeblock to start a campaign on Propeller Ads again.
At least this is how I understand it but I guess it should be pretty accurate
My biggest problem is to see a connection between what I read in the tutorials or explanations and how it could have a real practical use for my very own
Originally Posted by twinaxe
But let's assume you want have a class PropellerAds that 1) inherits methods from a class TrafficSource (that can itself inherit from another one), and 2) takes the PropellerAds API keyvalue as parameter (for example because you have two accounts, or because as a safety practice, you don't want to hardcode your API keys in the code).
In that case, you need to initialize the internal variables from the class TrafficSource first, which is what super().__init__() does (it calls the constructor of the parent class), then pass the keyvalue variable as a parameter when creating a class instance. For this, you need a class constructor.
class PropellerAds(TrafficSource):
def __init__(self, keyvalue):
super().__init__()
self.base_url = 'https://ssp-api.propellerads.com/v5'
self.req_headers = {'Authorization': 'Bearer ' + keyvalue}
p = PropellerAds('topsecretkey')
Originally Posted by twinaxe
Once/if it gets bigger, you can still hire a specialist in code refactoring to put things in order:
https://www.upwork.com/hire/code-ref...g-freelancers/
What I realize is that the coding itself isn´t even the most hardest task.
The planning of the code can be much more complicated.
"How do I group the functions", "How do I sort the modules", "What parameters do I have to pass to the functions" and all such stuff.
Now I see that it really makes a difference if you just have 1-2 files in your project or if it gets bigger and needs more planning so that you don´t have a huge clusterfuck
New update but again rather about structuring and not much practical progress.
Two days ago I implemented remote deployment in PyCharm so that the files on my server automatically get updated when I make changes to them on my PC.
I excluded a test directory from my PC that I use to write new code.
When the new code is ready I paste it in the appropriate files and it´s automatically updated on the server as well.
I ran into some issues when I imported functions from modules in other directories.
On my PC it all worked, on my server it didn´t.
Then I learned that you can easily create a setup.py from within PyCharm that basically installs the packages with the modules on my server and voila, everything is working.
Along the way I also learned how to create and how to use a requirements.txt, that´s also very helpful.
The structure for now is:
automation ├── __init__.py ├── automation.egg-info │ ├── PKG-INFO │ ├── SOURCES.txt │ ├── dependency_links.txt │ └── top_level.txt ├── build │ ├── bdist.linux-x86_64 │ └── lib │ ├── cronscripts │ │ ├── __init__.py │ │ ├── perf_recent.py │ │ ├── tc_ivr_api.py │ └── modules │ ├── __init__.py │ ├──Binom .py │ └── traffic_company.py ├── cronscripts │ ├── __init__.py │ ├── perf_recent.py │ └── tc_ivr_api.py ├── db_check ├── dist │ └── automation-1.0-py3.8.egg ├── index.html ├── modules │ ├── __init__.py │ ├──Binom .py │ └── traffic_company.py ├── requirements.txt └── setup.py
In the modules directory are (as the name says) only modules that get imported, in the cronscripts directory are only very small files that are executed from crontabs.
These files basically only import a specific function and call it.
The top directory will be for the frontend script once it´s done as well as for more complicated scripts that are executed directly and not just imported.
Right now I have two scripts running 24/7 on my server, one script to grab the TC IVR API and add it to the DB and another script to calculate the performance difference between the last 2 entries for each geo/carrier combination.
The scripts to create campaigns are already done and only need to be added and implemented to the new strcuture.
But I changed my priority a bit, before I continue wth automated campaign creation I will first work on automated optimization.
Reason is that this will have a much bigger direct benefit for me because I can use the optimization then for manually created campaigns as well and it will help me to get away from running only CPA campaigns so that I can run CPC/CPM campaigns then as well.
When this is done I will next add the campaign creation scripts, this could give the whole project a nice boost then.
I also prepared a list of few more CPA networks that provide a good API that I will implement later.
For trafficsources I have a list as well.
When the basic structure for getting stats, create and optimize campaigns is done it´s only about adding more platforms one by one
The more advanced the project gets the more challenging it gets but I really enjoy it and it motivates me to create something really good there.
Once the backend is running properly it will get interesting when I then have to think about how to code and implement a good frontend for it.
One more update without real progess.
The last days I spent to find a way to sort and separate my
I thought I foudn a solution to do it with Regey but didn´t work as expected.
Unluckily I need to figure it out somehow so that I can continue.
This really sucks when you get stuck with such things but I am optimistic that progress will pick up pace when this obstacle is done.
@twinaxe send me a pm with the details. I can have a look if you want.
Originally Posted by jeremie
No need for PM, maybe others can learn from it as well
Let me explain first.
For example I want to get placement stats for several campaigns at once so that I only need 1 API call instead of a separate API call per campaign.
In
When I grab these stats through the API I receive a string like this
[{"level":"1","path":null,"name_postfix":null,"path_com":null,"name":"Campaign 1"},{"level":"2","path":null,"name_postfix":null,"path_com":null,"name":"zone 1"},{"level":"2","path":null,"name_postfix":null,"path_com":null,"name":"zone 2"},{"level":"2","path":null,"name_postfix":null,"path_com":null,"name":"zone 3"},{"level":"2","path":null,"name_postfix":null,"path_com":null,"name":"zone 4"},{"level":"2","path":null,"name_postfix":null,"path_com":null,"name":"zone 5"},{"level":"2","path":null,"name_postfix":null,"path_com":null,"name":"zone 6"},{"level":"2","path":null,"name_postfix":null,"path_com":null,"name":"zone 7"},{"level":"2","path":null,"name_postfix":null,"path_com":null,"name":"zone 8"},{"level":"1","path":null,"name_postfix":null,"path_com":null,"name":"Campaign 2"},{"level":"2","path":null,"name_postfix":null,"path_com":null,"name":"placement 1"},{"level":"2","path":null,"name_postfix":null,"path_com":null,"name":"placement 2"},{"level":"1","path":null,"name_postfix":null,"path_com":null,"name":"Campaign 3"},{"level":"2","path":null,"name_postfix":null,"path_com":null,"name":"placement 1"},{"level":"2","path":null,"name_postfix":null,"path_com":null,"name":"placement 2"}]
I could convert it to JSON and then do something like
for x in str:
if x['level'] == 1:
name = campaign_name
else:
name = placement
What I try to do is a regex like this:
- Match starts with {"level":"1
- Then any character, any amount of occurrences
- Match ends with }
- After match is either ,{"level":"1 or ]
And shortest possible match.
Maybe it would also possible without the "Match ends with }"
I am confident that it would be a very easy task with the Zennoposter Regex creator but then I would have to install Windows to use it (what I probbaly even would do when I don´t find another solution).
Edit: This regex should work, will test it now with several different settings
\{"level":"1.*?(?=,\{"level":"1|])
I don't see any reason why you would use a regex while the data is already structured as JSON
[{
level1
}, {
level2
}, {
level2
}, {
level2
}, {
level1
}, {
level2
}, {
level2
}, {
level2
}, {
level1
}, {
level2
}, {
level2
}, {
level2
}]
[{
level1,
level2,
level2,
level2
}, {
level1,
level2,
level2,
level2
}, {
level1,
level2,
level2,
level2
}]
[{
level1: {
level2,
level2,
level2
}
}, {
level1: {
level2,
level2,
level2
}
}, {
level1: {
level2,
level2,
level2
}
}]
From the API you just get the stats for campaign name and placement as list items one after another.
I´d like to rather have the stats for each campaign name with its placements per list item.
Assuming you only have level 1 and 2, I would go with
res = []
for item in src:
if item['level'] == "1":
res.append(item)
else:
# in the last level 1 (res[-1]), create a 'plac' list if not exists and add the current level 2 item to it
res[-1].setdefault('plac', []).append(item)
Originally Posted by jeremie
if x['level'] == 1:
name = campaign_name
else:
name = placement
I know that I probably overcomplicate the stuff a bit but for me it seemed to be "safer" to first encapsulate the stats per campaign somehow before I process it forther.
New update, new issues.
It´s like fighting with a Python-Hydra, as soon as one issue is solved 2 new issue appear
This time it´s something trivial like grabbing the stats.
Although the stats in the tracker interface update in realtime the stats that you can pull through the API only show from the last full hour to now.
In other words, when we have 09:37 and I want to get stats for the last 10 minutes then it´s simply not possible to get stats from 09:27-09:37.
Instead I would receive stats from 09:00-09:37.
Most annoying is that I sat down befor, planned the project with its steps, modules, functions and stuff and now I had to throw everything over to come up with a new approach.
Then I had a new idea but it also won´t work because I also can´t grad realtime stats from the trafficsource, it´s similar to the tracker stats.
I still think I found a solution now but it also means quite some restructuring and recoding, good thing is that I slowly get used to it 
Although such obstacles are really annoying I also like it because it´s a great feeling when you suddenly face a new challenge and still you´re able to master it.
Additionally it forces me to learn many new things along the way and I am excited to see that for every problem I have there are good solutions available.
Time for an update, unluckily still not much progress in terms of new functionality or so.
The last days I spent some time to hunt for new trafficsources and I found a few that look promising.
Some have a good API where I can create and stop campaigns as ell as blacklist/whitelist placements, these will be first implemented.
Right now I am testing lots of things to find new platforms to work with and in the meantime I continue working on the scripts.
Originally Posted by twinaxe
I don´t know how regularly I update it because it´s more of a fun project and not mandatory for my work but I do my best to update as often as possible.
Before I start let me first tell you a bit about my motivation to run the FA.
Main reason is that I really would like to see more action in our programming and coding section.
Of course it´s no easy task for me to post new content there when I am no coder myself but for many years already I want to learn coding.
Not for big projects or so, just for myself because I enjoy it to program own stuff, expecially when it´s helpful then
I don´t have a coding background at all and even today I am not able to make a simple website from scratch but when you work with affiliate stuff for 15 years then of course you still learn a bit of this and a bit of that.
This basic knowlegde and basic understanding of coding logic was enough that I started working on my own cloaker in 2009 or 2010.
It was a very basic and 100% PHP based cloaker and I improved it many times in the following years.
I learned all I needed for it along the way, learning by doing.
That way I didn´t have to learn things that I don´t need and could learn exactly what was needed in my situation.
I also took some other scripts and changed them a good bit so that they work for my stuff.
Nonetheless I always wanted to learn more about coding but never got deeper in it.
My main goal is to work on automation scripts to support my lazyness and to make my daily affiliate life easier.
This is when I thought about different programming languages.
Usually my decision would be PHP because there I know a bit already.
But for the stuff I want to do I thought that Python probably is better for it because it has many good libraries for AI, machine learning and other scientific stuff already and from what I read it´s very good for automation anyway and also easy to learn.
Somehow I still didn´t find the motivation to finally start working on some stuff.
Recently I learned that @jeremie and @bennimen have some coding background and have scripts for their own affiliate stuff and this got me a bit more motivated.
It seems the good ol´ @jaybot also know a little bit about coding or even Python.
Right now my biggest obstacle is to find the best way to start at all.
I use Sublime to write the code and then run it in terminal.
But I would prefer to use an IDE where I can write, run and debug all in one interface.
I tested VS Code and Pycharm, on VS code I somehow can´t get the integrated terminal working so I can´t run and debug my code there.
Pycharm works ok so far, I tested the community version.
@bennimen also recommended Jupyter Notebooks and he uses it himself but before I get used to it I think I better try to get used to a real IDE
Will try to find the best way for me but until then I just use Sublime and terminal.
Next thing is about all the terms and stuff, what is needed and how I use it, interpreters, functions, modules, libraries.
What is included in Python already, what do I have to download/install?
Basically all these pain everyone has when we start something new and I am confident that the fun starts when the basics are clear
Alright, thanks for still reading after the long prologue.
Now let´s start with the project itself.
My plan is to get some automation for my campaigns running so that I have more time for other things.
For blacklisting placements I already use TheOptimizer so what I would need is to have some automation for my tracker stuff = landing pages and offers.
To keep it as simple as possible in the beginning I will start with direct linking campaigns where I test many offers at once.
The
I guess it gets a bit more tricky when I have to update the campaigns then later so that I have to post data back through the API.
Special thanks to @bennimen for sending me a script for reference so that I can check the code a bit.
What I did so far is to grab the all-time stats for a specific campaign group in
So this little code snippet is the beginning of my journey to tracker automation
import json
import requests
response = requests.get("http://mydomain.com/index.php?page=Campaigns&user_group=all&status=2&group=40&traffic_source=all&date=9&timezone=+2:00&api_key=##my_api_key##")
db = json.loads(response.text)
items = []
for item in db:
items.append(item['name'])
for x in items:
print(x)
Then I will make the script to loop through all these campaigns stats and pause offers based on simple rules.
Will update with some fresh code later
Quick tip:
instead of:
items.append(item['name'])
items.append(item.get('name', 'default value'))
instead of:
Code:
items.append(item['name'])
Code:
items.append(item.get('name', 'default value'))
But the post you quoted is pretty outdated already, much happened to the project in the meantime.
However, feel free to read the thread a bit and contribute with whatever knowledge pearls you can drop
I would absolutely enjoy to see more coding stuff, especially Python, in the forum.
Home > Technical & Creative Skills > Programming, Servers & Scripts