Friday, April 24, 2009

Resolved - Sign up outage

If you tried signing up for our service over the last few hours the sign-ups were failing due to a network timeout error from Authorize.net we have contacted them to resolve this issue and we are now back up and running 100% we apologize for any inconvenience this may have caused.

Wednesday, April 15, 2009

Resolved - Minor Bug Found In API Functionality

** Update **



API hotfix was successfully applied during our emergency maintenance at 17:00 PST. The API behavior is now correct and all list items are returned for all API calls returning list data over 100 items in length.

Tuesday, April 14, 2009

Minor Bug Found In API Functionality

** Update **

There will be an emergency maintenance at 17:00 PDT (24:00 GMT) today, April 15, 2009, to support pagination within our API to work around this bug.  Please refer to updated documentation on GoGrid's Wiki site to get the updated syntax for the new pagination variable 'num_items' at http://wiki.gogrid.com.

Our most recent release introduced a minor bug in API functionality. Returned data lists greater than 100 entries may be truncated. This will usually only occur when listing private IP addresses via the API. You will not be impacted unless using the API and attempting to list private IP addresses. We expect a hotfix will be scheduled soon. This entry will be updated when we have more details.

Friday, April 10, 2009

Completed -- System Maintenance

Maintenance Window:
Tuesday, April 14, 01:00 - 05:00 PDT
(Tuesday, April 14, 08:00 - 12:00 GMT)

Customer Impact:
The customer service portal my.gogrid.com and order/billing systems will be unavailable.  The API will be available, but will be queuing requests until maintenance is completed.

Maintenance Details:
We are upgrading a core database layer to provide additional capacity to the service.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.

Thursday, April 9, 2009

GoGrid Update from the Founders

After the events of last week, we would like to publicly answer the common questions we are receiving from many of you, our customers who were impacted by last week's network attacks.

What is GoGrid  doing so that this issue does not happen again?
Network attacks are a fact of life on the Internet, though generally their impacts on customers can be prevented. In the past week we have seen large scale attacks on a number of companies besides GoGrid (UltraDNS, Register.com, and The Planet). GoGrid has taken the following actions to better prepare ourselves for future attacks:

Partnership with Cisco
We had attack prevention systems in place that generally deflect multiple attacks each day, but were unable to fully handle the scale and new type of these attacks. We are working closely with our vendor partner, Cisco, and others to identify any known vulnerabilities in their products or our architecture that could weaken our services and defenses.

Network Infrastructure Improvements
Our network team, working with Cisco and a network engineering consultant, has made a number of improvements to our network defense systems, router configurations, and monitoring and alerting systems.  For security reasons, we don't publish the specifics of these changes, but would be happy to communicate with any of you individually if you have concerns that you would like addressed.

What will happen differently if there is another outage?
While we work to prevent future outages and events that degrade service quality, we are simultaneously improving internal and external communications. Some of the steps we are taking are as follows:

Faster internal escalation and alarm procedure
We have refined our internal policies to improve response times so that any employee can ring a virtual fire alarm with immediate response from Networking, Systems and Engineering. The goal is to save precious seconds and minutes.

Improved external communication at http://www.gogridstatus.com
In the event of any outage, the status blog will be updated more frequently with as much information as possible. We recommend that all clients subscribe to the RSS feed from this blog. Please consider GoGridStatus.com as your first resource to learn of any issues affecting your GoGrid service.

Expanded phone and chat coverage
We are accelerating our plans to expand our in-house customer service organization. Once hiring is complete, this group will be available to answer all commonly asked questions (product, billing, sales, account) and augment our existing 24/7 technical support. This means that more people will be available in the near future to answer phone and chat and keep you up to date.

What is the status of GoGrid's 10,000% guarantee?
GoGrid's Service Level Agreement (SLA) documents our commitment to our customers' success.

SLA Credits Issued
Understanding that many of our customers depend upon our product to run their business and expect a high level of service, we issued SLA credits to all GoGrid customers who we knew were impacted by the outage.  The Company was not obligated to pay these credits, but we wanted to extend this quick credit to demonstrate GoGrid's strong commitment to our customers.

What else is going on with GoGrid?

New Service Teams
As of April 1, all GoGrid customers now have their own Service Team. This means there is a Sales Account Manager, a Support Systems Administrator, a Billing Associate, a Data Center Technician, and a Network Engineer all on a team dedicated to your account. The same people (during business hours PDT) will be helping you get the most out of GoGrid, and you can get to know them and they can get to know you and your infrastructure needs. (On call technical support remains available 24/7, our growing in-house customer service team is available to you as well, and both of these organizations report any issues into your assigned Service Team). To learn more about service teams, please visit http://www.gogrid.com/support/service-teams.php

Expanded GoGrid Features
We are listening and responding to your requests for key features as quickly as we can. The ability to save your own image to deploy additional prebuilt servers, or to park the data in cloud storage so you don't have to pay RAM fees when you are not using the server, will be here in the beginning of the summer. Other features and improvements to the portal are in the works as well.

Once again, I wanted to personally thank you for your business as we continue to execute on our goal of offering the most flexible and easiest to use cloud computing infrastructure.

Best Regards,
David Hecht
Co-Founder & Chief Marketing Officer

Resolved -- Verio Offline

** Update @ 4/10 17:00 PDT **
Verio service has been restored, and traffic is being balanced.  No customers were affected.

Customer Impact
At 05:34 PDT (GMT -7) our BGP session with Verio disconnected.  We're presently working with Verio to resolve the problem.  All traffic automatically failed over to our other transit providers.

Wednesday, April 8, 2009

Resolved -- Emergency Portal Maintenance

Maintenance Window:
Wednesday, April 8, 2009, 16:00 - 17:00 PDT
(Wednesday, April 8, 2009, 23:00 - 24:00 GMT)

Customer Impact:
The my.gogrid.com customer portal will be unavailable.  The API will not be affected by this maintenance.

Details:
We will be deploying some patches to the GoGrid customer portal to resolve issues that prevent new user creation in the user management section under My Account.

If there are any additional questions please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.

Resolved -- GoGrid Load Balancer Add/Delete

** Update @ 4/14 16:30 PDT **
A change in the F5 API resulted in updated error code handling in our system.  We've now pushed updates to the engine which deploys load balancers.  A side-effect is that the lose artifacts of prior create/delete failed jobs must be cleaned up first.  If you continue to experience problems, please open a case and refer to problem identified in case 00203213.

Customer Impact:
Load Balancer additions and deletions fail.

Details:
Presently we're investigating recent failed add and delete jobs for GoGrid Load Balancers.  We'll update you when we have more information.

Until a resolution is reached, we suggest the following.

If a delete load balancer job failed:
The Load Balancer shouldn't be answering.  If it is not answering, then use the same IP on your secondary public interface of a system that can answer (Local Area Connection 3 on Windows systems, and eth2 on Linux Systems).  Please submit a case to GoGrid Support to clean up the remaining Load Balancer artifacts as your Load Balancer is only partially deleted.  We can remove the bad database entries and clean up the hardware.

If you cannot deploy a load balancer:
Use the IP you would use on the Load Balancer as a secondary IP on your secondary public interface of your system (Local Area Connection 3 on Windows systems, and eth2 on Linux Systems).

Monday, April 6, 2009

Completed -- GoGrid Emergency Maintenance - Load Balancers

** Update @ 20:05 **
Maintenance was successful and the suggested vendor patch didn't resolve the defect identified.  We'll be consulting with the vendor to seek a resolution.

Maintenance Window:
Monday, April 6, 2009 18:00 - 20:00 PDT
(Tuesday, April 7, 2009 01:00 - 03:00 GMT)

Customer Impact:
There should be no customer impact as GoGrid's load balancers are a redundant pair.

Details:
The firmware on the load balancers are being upgraded to a version recommended by the vendor in order to fix a software defect that causes depletion of shared memory resulting in occasional loss of connectivity for some customers.

If there are any additional questions please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.  You may also open a support case at https://my.gogrid.com.

Friday, April 3, 2009

Resolved -- Scheduled UPS Maintenance

Maintenance Window:
Tuesday, April 7, 2009, 10:00 - 17:00 PDT
(Tuesday, April 7, 2009, 17:00 - 24:00 GMT)

And

Wednesday, April 8, 2009, 10:00 - 17:00 PST
(Wednesday, April 8, 2009, 17:00 - 24:00 GMT)

Customer Impact:
There should be no down time.  This notice is for advisory purposes only.

Maintenance Details:
Tuesday, April 7, 2009. We will be performing maintenance on the battery cabinets for UPS 1 and 2. No unit will be put into bypass, but we will have limited battery backup during the maintenance.

Wednesday, April 8, 2009. We will be performing battery maintenance and full diagnostic maintenance on UPS 3 and 4. Both units will be placed in bypass (1 at a time) during the maintenance.

If there are any additional questions please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.  You may also open a support case at https://my.gogrid.com.

Wednesday, April 1, 2009

Resolved -- GoGrid Portal

** Update @ 16:20 **
Problems with logging in to https://my.gogrid.com are now resolved.

We're aware that https://my.gogrid.com is not accepting login information for some customers.  Please do not attempt to log in more than twice while we are resolving the problem.

The GoGrid API is still operational and not affected.