Home > General > Affiliate Marketing Forum

Amazon S3 is down - pause anything that relies on it. (20)

02-28-2017 06:21 PM #1 caurmen (Administrator)
Amazon S3 is down - pause anything that relies on it.

UPDATE: everything's back up now. If you're seeing this on the 1st March 2017 or later, no need to worry.

There's a huge Amazon outage at present, and it's taking sites down left, right and center.

That might well include your landers if they're on S3, even if they're being hosted by Cloudfront subsequently. It's being reported that Cloudfront is throwing errors for S3-hosted content in some cases.

So check your landers just now, and if you can't get to them, pause your campaigns!

More info: probably the best place is the Hacker News thread, where the world's top coders and tech guys are currently Losing Their Shit over the issue:

https://news.ycombinator.com/item?id=13755673

02-28-2017 06:23 PM #2 manu_adefy (Veteran Member)

The issues appear to be affecting Buffer, Business Insider, Chef, Citrix, Codecademy, Coursera, Cracked, Docker, Expedia, Expensify, Giphy, Heroku, Home Chef, iFixit, IFTTT, Imgur, isitdownrightnow.com (*Manu laughs uncontrollably at the irony*), Lonely Planet, Mailchimp, Medium, Microsoft's HockeyApp, News Corp, Quora, Razer, Slack, Sprout Social, Travis CI, Trello, Twilio, Unbounce, the U.S. Securities and Exchange Commission (SEC), and Zendesk, among other services. Airbnb, Down Detector, Freshdesk, Pinterest, SendGrid, Snapchat's Bitmoji, and Time Inc. are currently working slowly.

http://venturebeat.com/2017/02/28/aw...-slack-trello/

02-28-2017 06:30 PM #3 foussama (Member)

Isn't that the perfect moment for those who aren't using AWS to increase their bids and run all their paused campaigns ? Since everyone is pausing the campaigns that are running in AWS... Some placements should be free. Explain How wrong I am please

02-28-2017 06:31 PM #4 manu_adefy (Veteran Member)

Originally Posted by foussama

Traffic sources still work, I didn't pause anything. Less people pause than needed to affect bids imo.

02-28-2017 06:41 PM #5 foussama (Member)

Well that sounded too easy effectively... Even the DLs are down since they're using aws

02-28-2017 06:59 PM #6 Mr Payne (Member)

Yea I noticed this issue earlier today and took appropriate actions.

Now its time to capitalize on those lazy affiliates who paused everything

02-28-2017 07:01 PM #7 voluum (Veteran Member)

Hi guys,

As is, we've only encountered global frontend issues as a result of the recent AWS outage.

With it's isolation to the US East region, traffic pushed to Voluum campaigns has been routed to other regions that have been scaled to accommodate for it.
We also foresee that reports caching will need to be refresehd once the fault is fixed. i.e. Data integrity is expected to be retained once we are back with all systems operational and refreshed.

Apologies for the fuss, we're patiently waiting for the green from AWS.
Updates will be posted in our incident page here.

Regards,
Adrian

02-28-2017 07:43 PM #8 foussama (Member)

It's good now !

Edit: Well not really, the frontend is there but data is missing...

02-28-2017 08:05 PM #9 voluum (Veteran Member)

Originally Posted by foussama

It's good now !

Edit: Well not really, the frontend is there but data is missing...

Hi foussama,

AWS have since had a status update -
'Update at 12:52 PM PST: We are seeing recovery for S3 object retrievals, listing and deletions. We continue to work on recovery for adding new objects to S3 and expect to start seeing improved error rates within the hour.'

Update from our end -
Auto-scaling traffic had incurred some faults, so we can't guarantee that all traffic requests were resolved correctly when routing to other regions was applied.

Also, with monitoring continued on our end, Frontend issues seem to be global, so you might be able to log-in, but not be able to view your Voluum dashboard.
It is expected that some report services may be affected by the outage also.

We will have a full scope of this once we have all services back online.

I'll push notice here and on our status page incident in due course. http://status.Voluum.com/incidents/3pg933wnd6cg

Regards,
Adrian

02-28-2017 08:23 PM #10 matuloo (Legendary Moderator)

Originally Posted by Mr Payne

Yea I noticed this issue earlier today and took appropriate actions.

Now its time to capitalize on those lazy affiliates who paused everything

The problem is, advertisers and affiliate networks are affected too - some of them ... I had to pause a lot of stuff.

02-28-2017 08:26 PM #11 manu_adefy (Veteran Member)

The biggest problems were frontend though, so the effects there are not as bad. Of course, Cake and HasOffers networks are likely to have been affected quite a bit though.

And also possibly some networks that have custom solutions. Overall, the whole Internet was affected to some extent but depending on who you ran with you might not have needed to pause anything.

02-28-2017 08:30 PM #12 platinum (Veteran Member)

What were the odds of having Amazon as a single point of failure? :P
I'm curious what the estimated damage due to this service disruption might be.

02-28-2017 08:59 PM #13 caurmen (Administrator)

Looks like S3's starting to come back up now but it's still flaky as of latest reports.

Will probably be fine in a couple of hours.

02-28-2017 09:21 PM #14 voluum (Veteran Member)

Hi guys,

S3 is back to green with the issue marked as resolved.
We'll still continue to monitor and also update reporting.

Once all is cleared with our devs, I'll publish a postmortem of the incident here and in our status page.

Again, apologies for any inconvenience caused and thank you for your patience and understanding.

Regards,
Adrian

02-28-2017 11:03 PM #15 matuloo (Legendary Moderator)

Yeah, the stuff I checked myself seems to be working fine again ... hopefully the problem is over

02-28-2017 11:51 PM #16 alex973 (Member)

Hi,

Thanks for posting updates here.

The data on Voluum seems very out of date, when do you think everything will be back?

Thanks!

Alex

03-01-2017 12:14 AM #17 chris_climbs (Member)

It seems that there are still issues with Voluum's data reporting?

I understand that the issue's root is not with Voluum, but with S3, so I realize that you guys are doing what you can given the situation. Regardless, it's been a little while since the last update, and I'd love to gain any more insight into the current situation.

If there have been any new metrics to base such suggestions on, can we get another update, if possible, on when the service could be useable/back to realtime again?? Thanks for all your hard work, in what I imagine to be a stressful day!!

UPDATE:
Looks like the impressions/clicks are coming in, but quite severely delayed. My above question still stands

03-01-2017 01:52 AM #18 priest (Member)

Voluum stats are all caught up for me now

03-01-2017 10:11 AM #19 davidep (Member)

It looks like now everything's fine with S3

03-02-2017 03:41 PM #20 voluum (Veteran Member)

Originally Posted by alex973

Hi,

Thanks for posting updates here.

The data on Voluum seems very out of date, when do you think everything will be back?

Thanks!

Alex

Originally Posted by chris_climbs

Hi guys,

Sorry for the delay in getting back to you on this, it's been a tad bit hectic this week.

We had our last phase of updates* for reporting closed at around 15:00 CET yesterday.
*For data that wasn't processed during the outage.

Are you still experiencing discrepancies in your data? If so, can you shoot me a message with details so I can ask our guys to check it over.

Also, for full disclosure, our postmortem of the outage is available here.

Regards,
Adrian

Home > General > Affiliate Marketing Forum