Home > Tracking Campaigns > Voluum

Is Voluum Down? (16)

02-28-2017 05:21 PM #1 mikemartel (Member)
Is Voluum Down?

Is Voluum down? I can't get into the back end.

02-28-2017 05:23 PM #2 manuovi (Member)

Is down for me

02-28-2017 05:24 PM #3 paujacks (Member)

Yep, sure is down

02-28-2017 05:28 PM #4 manu_adefy (Veteran Member)

Yep, seems down for everyone.

02-28-2017 05:30 PM #5 priest (Member)

Amazon S3 is down apparently. Might want to pause your campaigns. This could mean a good chunk of the internet is not functioning properly.

https://status.aws.amazon.com/

02-28-2017 05:30 PM #6 toadler ()

Logged in here to make sure it wasn't just me!

02-28-2017 05:32 PM #7 BeyondHosting-Tyler (Member)

Hahahahahaha.

Put all your eggs in one basket.

02-28-2017 05:34 PM #8 voluum (Veteran Member)

Hey guys,

We're experiencing an elevated amount of frontend errors due to an outage on our service provider - AWS - S3.
Our devs are aware of the issue and have set monitoring in place while we hold tight for AWS to fix the issue.

The incident has had implications on frontend resources. Traffic and reporting is unaffected.

Apologies for the inconvenience and updates will be posted in our incident page here.

Regards,
Adrian

02-28-2017 05:38 PM #9 mikemartel (Member)

I can't get into tracking.

02-28-2017 05:48 PM #10 voluum (Veteran Member)

Hi Mike,

Apologies for the fuss, our devs are monitoring the incident and we're awaiting a status update from AWS.
Once they're back to green, we'll be back up at normal operation shortly after. Notice will be posted here.

Regards,
Adrian

02-28-2017 08:14 PM #11 andyvon (AMC Alumnus)

Are you sure this only affected front end? We can log in to the dashboard now, but see 0 traffic for the last couple hours. Is this just a case of the stats being delayed or did this actually have an impact on tracking?

Edit: nevermind, just read your post in the other thread which mentions that report caching will need to be refreshed: https://stmforum.com/forum/showthrea...l=1#post305394

02-28-2017 08:33 PM #12 voluum (Veteran Member)

Hi andy,

Yeah, since the start of the outage data was registered, but not processed to our reporting.
So it will represent inaccurate data until we refresh our reporting. I'm awaiting the est' time for this to be done

The scope of this AWS outage was quite extensive, so if your campaign resources, networks and sources also used S3, you're likely to have been harder hit by this incident.

I'm still awaiting confirmation from our team if there was any indication of traffic loss during the routing procedures.
I'll update here once I receive feedback / when the dust settles.

Best,
Adrian

02-28-2017 09:25 PM #13 voluum (Veteran Member)

Hi all,

Phew! Just an update - The S3 incident has since been resolved. https://status.aws.amazon.com/
I will have a full postmortem update of the incident following consultation with our dev team early tomorrow morning.

Thanks for your patience and sincere apologies for any inconvenience caused.

Kind regards,
Adrian

03-01-2017 12:26 AM #14 erikgyepes (Moderator)

What a night guys!

BTW: Data is slowly coming back now.

03-01-2017 01:00 AM #15 thuglife (Member)

Quote Originally Posted by BeyondHosting-Tyler View Post
Hahahahahaha.

Put all your eggs in one basket.
This is why I left BeyondHosting.

03-02-2017 03:22 PM #16 voluum (Veteran Member)
Update - AWS outage postmortem

Hi guys,

Slightly delayed than planned, our postmortem of the AWS outage earlier this week is now published here.
Felt it best to link it instead of pasting in here - due to it's length.

TLDR; We came out relatively unscathed by the outage, but will still hold our focus on better resiliency to bounce back from such incidents in future.

In case you have any Qs or concerns, feel free to ping me.

Regards,
Adrian

Home > Tracking Campaigns > Voluum