What maintenance does FF require? (13)
As a non-techie is FF a viable option?
What regular maintenance does it require, if I just set it up and started running without pouring over server logs etc can I expect trouble down the road?
Would love some sort of managed offering, ideally self-hosted but monitored/managed...
Hey mate,
Regular maintenance ? I would say close to none... I'm running 2 instances since 1 year without problem. Is your FF install using CentMinMod ? As far as I know CentMinMod is pretty well optimized right off the bat.
The only thing I had to do is upgrade my Digital Ocean VPS (cause I was running out of disk space) which is basically a 5-10 clicks process you do directly on Digital Ocean / Vultr.
Now I didn't really have to upgrade, but to keep my instance on the same VPS I would have had to "truncate" FF database to get rid of the old campaigns data... No clue if that's difficult to do or not, I'm sure @vitavee can tell you.
Cheers,
Myself and partners running FF long enough now without maintenance. Since it released the public version.
Apart from the installation itself, it's perfectly suitable to anyone, regardless of server management experience. The only time you will have to deal with server stuff is during the installation process, and still, we have this part covered with our server installation scripts:
http://www.funnelflux.com/vultr-setup/
However, if you're running millions of visitors per day, it's advised to have a sys admin monitor things for you, or have some kinds of alerts put in place, like when the disk is running out of space for example.
Agree that the installation is super simple.
After having gone through the process a few times now, it just takes me 10 minutes to perform a fresh install (that's on vultr, and without having any prior server experience).
But how are you other FF users running your reports?
I notice that when I run a more resource-intensive report, the CPU usage goes through the roof: regular CPU usage is around 25% due to traffic, but reporting makes the CPU usage go into the hundreds, with peaks up to 1000%.
If that goes on for too long, then my tracking clicks are no longer processed (504 Gateway Time-out).
How are you guys getting around this?
- do you report directly on the database (maximising use of indexes) instead of frontend UI?
- do you move the data to do your heavier reporting separate from your tracking instance?
- is there perhaps some backend setting where I could control how many server cores get allocated to reporting, while the remaining server cores would always remain available for traffic/click processing?
Is it by using the background reports or the realtime ones?
I reckon that reports can take long time, especially when you're grouping by data that isn't cached (we're actively working on that one).
But in any case, your redirects *should* never be affected because the reporting system is never accessing the same database tables as the ones used during the redirects.
That is in theory 
But in theory, theory and practice are the same. In practice, they are not
That being said, it's also tried and tested on installs with more than 10 million hits per day.
If it's not working well on your side, it may mean that you're getting more traffic than what this server can handle, or that the server settings could be tweaked better. Best thing to do would be to contact our sys admins at our support desk
Tell them:
1/ What are your server specs
2/ How much traffic you're getting on average
3/ Do you have spikes of traffic? If yes, what kind
4/ How many records you currently have in your DB
5/ What kind of reports are causing these CPU spikes
Thanks
It's the use of background reports that seems to be causing it, while the server configs are far above what they should be, based on the benchmarks Goshev shared with me (ref. LMN-QXLYV-720).
This 16 core server is processing 25k clicks/day which is nothing compared to the millions/day this config is able to handle.
Still when I launch a heavy background report, the CPU instantly goes over 100%: http://prntscr.com/cpng5p
The DB is virtually brand new (only 500MB in stats_root).
This 6-core instance is able to handle around 1.5 million clicks/day, and I'm only sending 10k clicks/day to it.
But a background report caused such an extreme overload that even the redirects stopped working here: http://prntscr.com/cpnhd7
The DB is close to empty (only 300MB in stats_root).
This 6-core instance is handling around 150k clicks/day which is well below what it can handle and the graph again shows that the traffic has virtually no impact at all on server resources, because the spikes are only when reports were running: http://prntscr.com/cpnpay
This DB has around 8GB in stats_root.
As for the type of reports:
I don't remember all of them, but the one I used to trigger the first example above is a report which returned +500k rows which I definitely appreciate is not a standard/regular report, but I needed to trigger an example.
Drill-down report (flat table)
Params: Hit ID, Hit Time, Conversion ID, Conversion Time
Time: past 7 days
Purpose of the report: reconcile FF data with affiliate network data.
I need all the conversions from yesterday (conversion time = yesterday) but reporting works on Hit Time so I take the past 7 days Hits to then filter out those Hits for which conversions got registered yesterday, which I can then reconcile with the affiliate network reports.
I'm currently talking to Anthony if there's a better way to do this report that I'm not aware of.
If not possible already, then I'll probably be raising a feature request to enable reporting on conversion date directly.
Meanwhile I just found out that I'll be able to get this same data very fast and easy from the DB directly.
Regarding the original issue (background report CPU usage): if you think there's anything else to investigate then I will reopen the ticket, but unless a hard limit can be placed on how much server resources the reports are able to consume, I guess there's not much that can be done to prevent those CPU spikes.
@mindfume: I have recurring problems with FF reporting on "non standard reports" as well... lately it seems like a report on a 1 month period took almost 2 days to generate. I also need an export in CSV and it doesn't seem to work either. [GGD-MGHRF-515]
In my case it's probably not the hit volume (non pop traffic) but the size of the database (about 37GB)
@vitavee: As users we REALLY need to have well defined rules as to what DB size can FF run without issues on a given server spec... after all if
e.g. in my case: 37GB is close to unmanageable on D.O. 4GB RAM 60GB Disk... I'm trying to export the valuable data I have stuck in the DB to make a Stats Reset and see if it works fine again.
Load balancing should probably be a feature request then. Ability to load click data from multiple places etc especially if you run huge pop campaigns. 20GB should be a limit I think.
@Vita - alerts in the panel about these stuff makes sense.
@wilfm, are you dumping from commandline?
Originally Posted by pain2k
Something else I forgot to mention:
If you have 16 CPUs, then you can go up to 1600% in the CPU activity chart (100% * 16 CPU).
With 16 CPUs, 200% only means that 2 of your CPUs are working at their max. You still have 14 CPUs free to do other tasks.
Thanks for digging into this VitaVee.
I've forwarded you my login details in BCF-VSHWN-248
Also very good to know that it's OK for the CPU graph to go over 100%.
That makes me a ton more comfortable now
I got it - checking now
Home > Tracking Campaigns > FunnelFlux