STM Mobile Tracker Was Working Fine... Now It's Broken? (27)
Just finished setting up the STM mobile tracker on my server, and everything seemed to be working fine.
Links worked, redirects worked, ect.
Now I go to log in and I get:
[PHP]INSERT INTO 202_users_log SET user_name='*****', user_pass='*********', ip_address='66.***.***.**', login_time='1374013851', login_success = '1', login_error='N;', login_server='a:34:{s:11:\"HTTP_ACCEPT\";s:63:\"te xt/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\";s:20:\"HTTP_ACCEPT_ENCODING\";s:17:\"gzi p,deflate,sdch\";s:20:\"HTTP_ACCEPT_LANGUAGE\";s:1 4:\"en-US,en;q=0.8\";s:15:\"HTTP_CONNECTION\";s:10:\"keep-alive\";s:12:\"CONTENT_TYPE\";s:33:\"application/x-www-form-urlencoded\";s:14:\"CONTENT_LENGTH\";s:2:\"89\";s: 11:\"HTTP_COOKIE\";s:1571:\"tracking202subid_a_28= 4991; tracking202subid_a_27=4991; tracking202subid_a_30=4991; tracking202subid_a_31=4991; tracking202subid_a_33=4991; tracking202subid_a_32=4991; tracking202subid_a_34=4991; tracking202subid_a_35=4991; tracking202subid_a_36=4991; tracking202subid_a_29=4991; tracking202subid_a_38=4992; tracking202subid_a_39=4992; tracking202subid_a_43=4992; tracking202subid_a_42=4992; tracking202subid_a_40=4992; tracking202subid_a_41=4992; tracking202subid_a_46=4992; tracking202subid_a_44=4992; tracking202subid_a_45=4992; tracking202subid_a_47=4992; tracking202subid_a_48=4992; tracking202subid_a_37=5188; tracking202subid_a_26=5189; tracking202subid_a_23=5339; tracking202subid_a_49=5467; tracking202subid_a_50=5859; tracking202subid_a_53=7072; tracking202subid_a_51=7263; tracking202subid_a_55=7290; tracking202subid_a_54=7824; tracking202subid_a_58=8382; tracking202subid_a_18=8547; __gads=ID=216d5e27510b0d53:T=1372431398:S=ALNI_MZS VyG0GugMO4MXpZlbwqfUomX1Aw; tracking202subid_a_20=8556; tracking202subid_a_52=8986; tracking202subid_a_60=9002; tracking202subid_a_=9145; tracking202subid_a_0=9372; tracking202subid_a_21=9372; tracking202subid_a_17=9674; tracking202subid_a_63=10012; tracking202subid_a_65=10118; tracking202subid_a_56=10292; tracking202subid_a_66=10582; tracking202subid_a_13=10631; tracking202subid_a_62=10914; tracking202subid_a_4=10925; PHPSESSID=9bc754526e72cb88130d57d62ec811cd; tracking202subid=11011; tracking202subid_a_70=11011; tracking202subid_a_1=7; tracking202subid_a_2=11; tracking202subid=12; tracking202subid_a_3=12; t202id=795\";s:9:\"HTTP_HOST\";s:20:\"*****.us\";s :12:\"HTTP_REFERER\";s:41:\"http:/*****.us/202-login.php\";s:15:\"HTTP_USER_AGENT\";s:141:\"Mozil la/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36 AlexaToolbar/alxg-3.1\";s:18:\"HTTP_CACHE_CONTROL\";s:9:\"max-age=0\";s:11:\"HTTP_ORIGIN\";s:27:\"http://*****.us\";s:13:\"DOCUMENT_ROOT\";s:28:\"/home/*****/public_html/m\";s:11:\"REMOTE_ADDR\";s:14:\"66.176.237.105\";s :11:\"REMOTE_PORT\";s:5:\"51604\";s:11:\"SERVER_AD DR\";s:11:\"8.36.41.220\";s:11:\"SERVER_NAME\";s:2 0:\"*******.us\";s:12:\"SERVER_ADMIN\";s:30:\"webm aster@*******.us\";s:11:\"SERVER_PORT\";s:2:\"80\" ;s:11:\"REQUEST_URI\";s:14:\"/202-login.php\";s:15:\"SCRIPT_FILENAME\";s:42:\"/home/*****public_html/m/202-login.php\";s:12:\"QUERY_STRING\";s:0:\"\";s:10:\" SCRIPT_URI\";s:41:\"http://************.us/202-login.php\";s:10:\"SCRIPT_URL\";s:14:\"/202-login.php\";s:11:\"SCRIPT_NAME\";s:14:\"/202-login.php\";s:15:\"SERVER_PROTOCOL\";s:8:\"HTTP/1.1\";s:15:\"SERVER_SOFTWARE\";s:9:\"LiteSpeed\";s :14:\"REQUEST_METHOD\";s:4:\"POST\";s:8:\"PHP_SELF \";s:14:\"/202-login.php\";s:18:\"REQUEST_TIME_FLOAT\";d:13740138 50.9269249439239501953125;s:12:\"REQUEST_TIME\";i: 1374013850;s:4:\"argv\";a:0:{}s:4:\"argc\";i:0;s:2 0:\"HTTP_X_FORWARDED_FOR\";s:14:\"66.176.237.105\" ;}', login_session='a:1:{s:5:\"token\";s:32:\"c002095af 1e25808a78382f13d87d632\";}'
Out of resources when opening file './*****_***/202_users_log.MYD' (Errcode: 24)[/PHP]
And when I visit a tracking link I get:
[PHP]Fatal error: Uncaught exception 'WURFL_Storage_Exception' with message 'Couldn't show tables from database <<mysqy database>> (Can't read dir of './******_STMTracker/' (errno: 24))' in /home/*****/public_html/m/3rd-parties/wurfl/WURFL/Storage/Mysql.php:72 Stack trace: #0 /home/*****/public_html/m/3rd-parties/wurfl/WURFL/Storage/Mysql.php(51): WURFL_Storage_Mysql->initialize() #1 /home/*****/public_html/m/3rd-parties/wurfl/WURFL/Storage/Factory.php(43): WURFL_Storage_Mysql->__construct(Array) #2 /home/*****/public_html/m/3rd-parties/wurfl/WURFL/WURFLManagerFactory.php(142): WURFL_Storage_Factory::create(Array) #3 /home/*****/public_html/m/3rd-parties/wurfl/WURFL/WURFLManagerFactory.php(57): WURFL_WURFLManagerFactory:
ersistenceStorage(Array) #4 /home/*****/public_html/m/3rd-parties/wurfl/includes/header_config_inc.php(15): WURFL_WURFLManagerFactory->__construct(Object(WURFL_Configuration_ArrayConfi g)) #5 /home/*****/public_html/m/tracking202/includes/devices_detect_inc.php(26): include_once('/home/*****/... in /home/*****/public_html/m/3rd-parties/wurfl/WURFL/Storage/Mysql.php on line 72[/PHP]
Any ideas what just happened?? 
First question - are you on a VPS or shared hosting? And did you install MySQL yourself or have a host install it?
It looks as if, for some reason, MySQL is having trouble opening its files. That could imply that MySQL's not set up properly, it could imply that there's some filesystem corruption, it could mean that there's a file system limitation that's too low for some reason - or it could mean, if you're on shared hosting, that someone else is using All Of The Files.
I'd recommend talking to your hosting provider about this and seeing if they have any ideas. Show them the last line of the first error - "Out of resources when opening file './*****_***/202_users_log.MYD' (Errcode: 24) "
Yep , looks like MySQL is broken.
Try running a repair if you know how to , otherwise just submit a ticket to host.
Thanks for the tips guys. And yeah, it was MySQL tanking after all. It came back after I notified my hosting provider, but my problems with the tracker haven't gone away.
And I'm on the VPS 1024 from Beyond Hosting. Upgraded from the Hybrid as it wasn't compatible with the tracker.
Anyways, onto the tracker issues..
After the tracker tanked and came back, I noticed some of the tracking links randomly took a really long time to load, or simply timed out. So I did a few Blitz.io stress tests and here's what I got:
That does not seem right to me. And my regular P202 install is solid without any ripple, so I know it's only the STM tracker. I've tried a few different tests and they all do the same thing; mass timeouts. The one pictured is 48 visits in 60 seconds.
Right now the guys at Beyond are trying to figure out what's wrong with it. I'm waiting for a lvl 2 tech to come take a look, as so far they haven't been able to fix it.
Any advice?
What p202 URL/file were you blitz testing?
Originally Posted by zeno
Ah, right - so I'm guessing you were testing a link with tracker.php in it? That's a decent test. To be absolutely sure, it might be worth re-testing with a new test campaign that direct links to www.google.com - that eliminates the possibility that there's something about your lander that's slowing the load down. However, frankly, that sounds pretty unlikely.
Blitz.io and anything Prosper-based can be a bit tricksy together, particularly with the default timeout. I'd also recommend you try re-testing, but increasing the timeout on Blitz - that may give you a more accurate picture of what's going on.
Also, can you post a similar benchmark from your Prosper install? You can use the methods in this article to get an accurate benchmark.
Sorry for the big pile of "test this, please" requests - speed issues are a bit of a benchmark-fest to narrow down. Alternatively, you might just want to wait for the BH support guys - I'm sure they'll be able to get to the bottom of it.
Okay, so the guys at Beyond told me the -p 1-250:60 test was way too much for the VPS 1024, and I should try running something smaller, as it would very likely continue to timeout otherwise.
So I went with -p 1-48:60, as they said 48 visits a minute is more or less what it can handle safely. The STM tracker still times out at that rate, and the graph is sporadic like the one I posted above.
As you requested, here's my regular Prosper install in the 1-48 test:
One of the lvl 2 techs just replied, and said that's not an issue they've experienced before with the STM tracker, and he wants to schedule a time to monitor the server while I run the test again. I think I'm gonna try installing the tracker again and giving this another shot.
Alright, just set up a new STM install and created a new test campaign to stress test. Same lander and settings as before. But as soon as I test the tracking link, I get this:
[PHP]Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 4194304 bytes) in /home/*****/public_html/stm/3rd-parties/wurfl/WURFL/CustomDeviceRepository.php on line 208[/PHP]
Any thoughts?
Okay, BH took care of that last issue. So I went ahead and re-ran the Blitz.io test with 48 views in 60 seconds, and still getting crazy timeouts.
I re-re ran the test again, except this time I raised the timeout to 5 seconds. Results:
No timeouts, but I'm still getting the lag spikes. Some visits took as long as 3 seconds. Is this normal?
The guys over at Beyond aren't sure what's causing it.
Another Update: Decided to run a bit of a longer test, so I went with 160 visits in 180 seconds with a 5 second timeout. Here are the results:
Needless to say, I'm not too confident about running real traffic just yet. Any ideas what could be the problem???
The guys at Beyond Hosting don't know what's causing it.
blitz.io for some reason isn't very reliable, try testing with ab ( from apache ).
Also on my dedicated I never had any problems , even when doing 1mill clicks a day.
Originally Posted by bbrock32
Or if you could kindly PM me your name, the BH tech I'm talking to now says he can look up your tickets and optimize my server accordingly.
And thanks, I'll look into the AB tool.
Well, the tweaks haven't been made by Beyond , hired an external sysadmin and not sure what he exactly did.
Not sure if they can copy everything he did. Anyway tell them to search by Besmir if they can find anything useful.
What's different about the STM modded P202 vs normal P202 that might lead to differences like this? Could it be the WURFL database lookups? Could the bottleneck be something to do with simultaneous lookups from it and lack of threading -> everything getting queued up? Otherwise could the STM modded P202 do some different MySQL related action that scales poorly under load?
We have a user pushing 1.5mm clicks a day on our Tracker right now with no issue in performance according to them.
Originally Posted by bbrock32
I have the guys over at BH looking into it. Hopefully they work out a fix.
Originally Posted by zeno
Originally Posted by The Angry Russian
I'm not pushing a lot of volume right now, and didn't want to be a bother emailing you guys about subids for every little test I do, so I decided to give the STM tracker a shot. That led to upgrading my VPS, and now I'm stuck with it. Regardless, this is a speed issue on my server and I need to have it resolved.
After some more testing, I've found out that my regular P202 install is problematic as well. In the original test I did on it, the P202 tracking link linked directly to my landing page. Which is loading fine, and quickly. The STM tracker link is different in that it links to the tracking domain, then redirects to the lander.
Well, I decided to make a regular P202 campaign direct linking to an offer (Google.com), so that Blitz would have to pass through my tracking domain and get redirected. This resulted in the same problem I was having with the STM tracker.
I guess this would explain why I've been having such shit conversions for the past 2 months?
Well, unless you were doing 10 hits a second then probably not. In any case blitz.io is just a simulated load test, doesn't necessarily equate to the performance you get from running real campaigns. 10/s is a lot though. That's 600 people a minute, 36,000 hits an hour or 864,000 hits a day. So the blitz test is suggesting the VPS is getting toward the equivalent of over half a million visits daily before timeouts are arising. Not bad? If you're running half a million clicks a day chances are a VPS1024 is way below what you're prepared to cash out for.
Originally Posted by zeno
I used to think the hits were visits as well until I found this: http://www.webopedia.com/TERM/H/hit.html
Anyways, I've been tinkering around with the server and am seeing some better results. Will post back some updates once I have some solid results.
Yep , I think the server specs are the bottleneck at this point.
I have stress-tested our tracker on my dedi and difference in response times was just a few milliseconds more ( and that's normal because of ISP / Model / Brand etc lookups ).
Cool - glad to hear things are starting to work out!
I had a couple of ideas, but I'll wait for your next post before jumping in with them.
So, last night I reinstalled MySQL and ran the tuner found in this thread then did the tweaks.
The results were a night and day difference. Here's the Blitz.io test from last night:
So much better than my last test, and that one even had a 1 second timeout.
BUT........
I ran the same test again this morning, without even touching any settings on the server, the STM tracker or Blitz, and here's what I got:
Disaster once again.
Beyond Hosting has no idea what's going on. Any advice?
That last one I have seen before with my VPS and it went away with a few tweaks. I think it's not that your server can't handle it but I think it bounces the connections when there are too many - e.g. for DoS protection or based on some concurrent users/connections etc. settings somewhere. Could also be MySQL related memory/thread/blah blah limits.
Try running that MySQL primer stuff again.
Yep, that's a good suggestion. I know it's a bit of a pain, but here's what I'd recommend:
1) Run a bunch of Blitz.io rushes - 3 or so - with a long timeout set. I tend to use 8 seconds.
2) Run the tuner script again. It'll now optimise based on the load you've been bashing it with.
3) Re-test.
I suspect that the tuner script may have optimised for a minimal load, and MySQL has subsequently gotten very upset 
Other things to check:
- Have you got Memcached running? If not, definitely get that bad boy on there. You may want to ask if you can get a PHP cache installed too.
- Check your webserver settings too. If you're running Apache, you might be experiencing the Keepalive Instakill or your MaxClients might be wrong.
- Whilst you're running the load test, run "top" and check your memory usage. If you're running completely out of memory, that might explain the timeouts. If that's happening, reduce your MaxClients, or otherwise let us know and we'll investigate further!
BTW, what traffic source are you intending to run, and at what volume? Currently you should be able to handle approximately 8,000 clicks a day even without further optimisation, which may be enough if you're using a banner-based traffic source. You're testing database load here, so one "hit" is roughly equivalent to 1 visit, because the STM tracker will only run the script you're benchmarking here once per visitor.
Thanks for the help guys, much appreciated.
That's exactly what I did last night. Surprisingly, what made the difference and got rid of those timeouts was disabling the Query Cache, which the tuning script actually told me to enable. There's still a ton more variable I'm testing out, but I'm happy to say that I'm no longer getting the ridiculous timeout issues I was getting before.
Although after 2 whole days of bugging the support team at Beyond, I would've expected them to have done some of this stuff for me, or in the least pointed me in the right direction. Not looking to bash them or anything, as they're super quick to respond and have solved a lot of my problems; but given the premium they charge for their stuff, I surely wasn't expecting this headache to get things to perform as they should.
Had it not been for you guys here on STM, I'd be paying for a slow server that lost clicks. 
And as far as traffic goes, I'm currently doing some mobile, search and social. Was doing some PPV, but that didn't turn out too well so I'd like to focus on what I know best for now.
Home > Other Systems (CPVLab, iMobiTrax, P202, Track Revenue, Click, Google Analytics, etc.) > The STM Mobile Tracker