Monday, December 21, 2009

blog.gogrid.com down for maintenance

Maintenance Window:
Monday, December 21st, 2009 8:00

Customer Impact:
The GoGrid "blog.gogrid.com" website is currently unavailable.

Maintenance Details:
GoGrid is making software improvements to the blogging website.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.

Resolved - - Network Issues

**Update @ 15:32 PST**

The network issues have been resolved by our Engineering team. If you have any questions please call GoGrid Support (877-9Go-Grid or 415-869-7444) and we will be glad to help you.

Thursday, November 5, 2009

Completed -- Gogrid Maintenance

Maintenance Window:
Thursday, November 5th, 2009 19:00 - 23:00 PST
(Friday, November 6th, 2009 03:00 - 07:00 GMT )

Customer Impact:
The GoGrid "my.gogrid.com" customer portal and API will be unavailable for up to 4 hours while we upgrade our infrastructure. There will be no impact to existing GoGrid VMs, load balancers or cloud storage during this time.

Maintenance Details:
GoGrid is making software improvements to the GoGrid customer portal and API.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.

Tuesday, September 29, 2009

Completed -- GoGrid Portal Maintenance

Maintenance Window:

Tuesday, September 29th, 19:00 - 23:00 PDT
(Wednesday, September 30th, 02:00 - 06:00 GMT )

Customer Impact:
The GoGrid "my.gogrid.com" customer portal and API will be unavailable for up to 4 hours while we upgrade our infrastructure. There will be no impact to existing GoGrid VMs, load balancers or cloud storage during this time.

Maintenance Details:
GoGrid is making software improvements to the GoGrid customer portal and API.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.

Thursday, September 24, 2009

UPS Maintenance and Generator Run

** Update @ 16:00 PDT **
Generator run for Oct 1st has been canceled. Scheduled maintenance is not affected for Oct 13th


Maintenance Window:

Thursday, October 1st, 10:00 to 17:00 PDT
(Thursday, October 1st, 17:00 to 02:00 GMT)


And

Tuesday, October 13th, 10:00 to 17:00 PDT
(Tuesday, October 13th, 17:00 to 02:00 GMT)

Customer Impact:
There should be no down time. This notice is for advisory purposes only.

Maintenance Details:
Tuesday, October 13th we will be performing maintenance on the UPS system. No downtime is expected as this is regularly scheduled preventative maintenance.

Thursday, October 1st we will be performing a generator run required to ensure contingent power in case of an outage. No downtime is expected as this is a regularly scheduled test.

If there are any additional questions please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2. You may also open a support case at https://my.gogrid.com.

Tuesday, September 15, 2009

Resolved -- Portal not displaying correctly (my.gogrid.com)

** Update @ 08:55 PDT **
Display problems are now resolved.

We are aware that some clients are not seeing the portal displayed correctly, and are instead seeing a mostly blank page.  Our systems engineers are attempting to resolve this problem currently.

Thursday, September 10, 2009

Resolved -- API Regression

** Update 09/15 @ 13:30 PDT **
Our software team has resolved our issues with the new API methods for our last portal release and our API has been updated.  You may now use the new API methods related to GSIs.

We have found an API regression that has forced us to revert to the previous API version, 1.2, today, September 10, 2009, at noon.  Users cannot edit, delete, or restore their MyGSIs via the API.  It is available through the portal.  We will issue a hot fix, as soon as it becomes available.

Resolved -- Billing Information in my.gogrid.com

** Update @ 15:46 PDT **
Problems are now resolved and the page now displays properly.

** Update @ 09:40 PDT **
Problem has resurfaced. Our engineers are looking into this problem again.

** Update @ 08:28 PDT **
Problems are now resolved and the page now displays properly.

Currently we're experiencing issues displaying the Billing tab under 'My Account' at my.gogrid.com. Engineers have identified the problem and expect resolution soon. Thank you for your patience.

Wednesday, September 9, 2009

Completed - GoGrid Portal and API Maintenance

Maintenance Window:

Wednesday, September 9, 19:00 - 22:00 PDT
(Thursday, September 10th, 02:00 - 05:00 GMT )

Customer Impact:

The GoGrid "my.gogrid.com" customer portal and API will be unavailable for up to 3 hours while we upgrade our infrastructure. There will be no impact to existing GoGrid VMs, load balancers or cloud storage during this time.


Maintenance Details:

GoGrid is making software improvements to the GoGrid customer portal and API.


Inquiries:
If there are any additional questions before, during or following the maintenance, please direct them to the GoGrid Support Center at 1-415-869-7444 or 1-877-9Go-Grid (877-946-4743).

For the latest details on GoGrid systems & Data Center status, please visit our blog:
http://www.GoGridStatus.com

Sincerely,
The GoGrid Team

Thursday, August 6, 2009

Completed - GoGrid Portal and API Maintenance

Maintenance Window:

Thursday, August 6, 19:00 - 23:59 PDT
(Friday, August 7, 02:00 - 06:59 GMT )


Customer Impact:

The GoGrid "my.gogrid.com" customer portal and API will be unavailable for up to 5 hours while we upgrade our infrastructure. There will be no impact to existing GoGrid VMs, load balancers or cloud storage during this time.


Maintenance Details:

GoGrid is making software improvements to the GoGrid customer portal and API.


Inquiries:

If there are any additional questions before, during or following the maintenance, please direct them to the GoGrid Support Center at 1-415-869-7444 or 1-877-9Go-Grid (877-946-4743).

For the latest details on GoGrid systems & Data Center status, please visit our blog:
http://www.GoGridStatus.com

Sincerely,
The GoGrid Team

Friday, July 31, 2009

Resolved -- Level3 Network Issue

At approximately 01:10 PDT we noticed a sharp decline in traffic on our Level3 connection.  Traffic returned at approximately 01:30 PDT and we are currently investigating the cause.

RFO
Level 3 was performing a scheduled maintenance in their internal network during which they inadvertently black-holed all our inbound and outbound traffic for around 20 minutes.  Our traffic didn’t get automatically re-routed through other carriers because our BGP session with our next-hop router at Level 3 stayed up.

Wednesday, July 29, 2009

GoGrid API Upgrade

We are doing an upgrade of our API infrastructure today at 6PM PST.  There should  be no interruption to your API service as a result

Tuesday, July 28, 2009

Completed -- GoGrid Emergency Border Router Maintenance

We've discovered a failed power supply on our second border router which will be replaced tonight at 21:00 PDT.

No downtime is expected.

Wednesday, July 22, 2009

Resolved -- Issues Starting Servers

The issue with starting servers has been resolved.  The handful of affected customers will receive further information in an official Reason for Outage.

Completed -- Emergency GoGrid System Maintenance

Maintenance Window:
Wednesday, July 22, 2009 10:00 - 10:30 PDT
(Wednesday, July 22, 2009 17:00 - 17:30 GMT)


Customer Impact:
Servers on the affected node will be rebooted during the emergency maintenance. Email notifications have been sent to the affected customers.


Maintenance Details:
GoGrid has discovered a problem with the hardware you are hosted on. In order to resolve the issue and to avoid further instability, we will need to reboot the node.


=== IMPORTANT ===
In order for GoGrid to perform this maintenance, please take a moment to update the "Password" section of the https://my.gogrid.com portal (in the "Support" tab) with the current username and password for each affected system listed. If no valid credentials are provided then GoGrid staff will be unable to gracefully shutdown your system.


If there are any additional questions before, during or following the maintenance, please direct them to the GoGrid Support Center at 877-9-GOGRID (877-946-4743). You may also reply to the email notification you received. We apologize for the short notice.

Tuesday, July 21, 2009

Completed -- UPS Maintenance

*** Update 7/29 @ 16:00 PDT ***
The UPS maintenance has been extended to 19:00 PDT.

Maintenance Windows:
Monday, July, 27 2009 10:00 - 17:00 PDT
(Monday, July, 27 2009 17:00  - 00:00 GMT)

Tuesday, July, 28 2009 10:00 - 17:00 PDT
(Tuesday, July, 28 2009 17:00  - 00:00 GMT)

Wednesday, July, 29 2009 10:00 - 17:00 PDT
(Wednesday, July, 29 2009 17:00  - 00:00 GMT)

Customer Impact:
There should be no impact to the power in the facility and no down time is expected.  This is an advisory notice

Maintenance Details:
UPS systems will be undergoing standard quarterly maintenance.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-866-946-4743 or 415-869-7444 option 2.  You may also open a support case at http://my.gogrid.com.

Thursday, July 9, 2009

Resolved -- GoGrid API

** Update @ 19:00 PDT **
API hotfix was successfully applied at 19:00 PDT. The API behavior is now correct and the expected delete response is returning.

Our most recent release introduced a minor bug in API functionality. API Delete operations are currently not returning the correct response. This is affecting all output formats: JSON, XML, and CSV. API delete calls will delete objects properly, but not send a complete API response back. This may affect API code that parses delete responses and expects the documented API response. We expect a hot fix will be scheduled shortly. This entry will be updated when we have more details.

Tuesday, July 7, 2009

Resolved -- GoGrid API

** Update @ 18:30 PDT **
An API hotfix was successfully applied at 18:30 PDT.  API behavior is now correct and all listed items are returned for all API calls returning data sets over 100 items in length.

Our most recent release introduced a minor bug in API functionality.  Returned data lists greater than 100 entries may be truncated.  This will usually occur when listing private IP addresses via the API.  You will not be impacted unless using the API to return lists larger than 100 entries (such as Private IPs or Jobs history).  We expect a hotfix to be scheduled shortly.  We will update this post again when we have more details.

Friday, July 3, 2009

Resolved -- Online orders on hold

Currently, orders made through www.gogrid.com are being put on hold due to an outage authorize.net, our credit card transaction provider, is having. The orders are in the system though, so we will process the orders manually as soon as possible.

Wednesday, July 1, 2009

GoGrid Portal and API Maintenance

Maintenance is complete, thank you for your co-operation.

**Update @ 1:30 AM PDT
Maintenance window has been extended till 01:45 am PDT.

**Update @ 12:58 AM PDT

Maintenance window has been extended till 01:30 am PDT.

**Update @ 11:58 PM PDT
Maintenance window has been extended till 01:00 am PDT.

Maintenance Window:


Wednesday, July 1st, 19:00 - 23:59 PDT
(Thursday, July 2nd, 02:00 - 06:59 GMT )


Customer Impact:

The GoGrid "my.gogrid.com" customer portal and API will be unavailable for up to 5 hours while we upgrade our infrastructure. There will be no impact to existing GoGrid VMs, load balancers or cloud storage during this time.


Maintenance Details:

GoGrid is making software improvements to the GoGrid customer portal and API.


Inquiries:


If there are any additional questions before, during or following the maintenance, please direct them to the GoGrid Support Center at 1-415-869-7444 or 1-877-9Go-Grid (877-946-4743).

For the latest details on GoGrid systems & Data Center status, please visit our blog:
http://www.GoGridStatus.com

Sincerely,
The GoGrid Team

Tuesday, June 30, 2009

Firewall Outage

Update @ 11:44pm
The fortinet outage did not affect any gogrid customers, only Servepath dedicated customers.


At approximately 7:45 PM PDT, we experienced a failure on one of our blades on the Fortinet Firewall. As of 8:45 PM PDT, the blade on the firewall has been restored to service.

Emergency-Network-Maintenance-CloudConnect

Update @ 9:21 PM PDT

The pertinent changes on both cloud connect environments have been made. Customer traffic wasn’t affected by this maintenance.



Maintenance Window:
Wednesday, June 30, 2009 19:00 -20:00 PDT (Wednesday, July 1, 2009 02:00 - 03:00 GMT)

Customer Impact:
Your Gogrid server(s) will lose network connectivity to your dedicated servers for up to 1 minute.

Maintenance Details:
GoGrid is performing an emergency upgrade to a core infrastruture component in order to add reliability to the cloud connect service.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.

Friday, June 26, 2009

Completed -- GoGrid Portal and API Maintenance

Maintenance Window:
Wednesday, July 1st, 19:00 -  23:59 PDT
(Thursday, July 2nd, 02:00 - 06:59 GMT )

Customer Impact:
The GoGrid customer portal and API will be unavailable for up to 5 hours while we upgrade our infrastructure. There will be no impact to existing GoGrid VMs, load balancers or cloud storage during this time.

Maintenance Details:
GoGrid is making software improvements to the GoGrid customer portal and API.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.

Saturday, June 13, 2009

Support case system maintenance

** Update @ 6/13 14:15 PDT **
Our support system is back online.  Customers can once again view existing support cases and submit new ones from the portal.

Customer Impact
Our support case system became unavailable due to maintenance by the vendor.  Customers were unable to submit new cases or view existing cases from the customer portal, and GoGrid support staff were unable to view or receive updates to existing cases, but phone support was still available

Wednesday, June 10, 2009

GoGrid Portal Maintenance

GoGrid network engineers will be performing network maintenance that will potentially reset your active session on my.gogrid.com. Service should be restored almost immediately and you will be able to re-login to my.gogrid.com.

The maintenance will occur at approximately 7:10 pm PDT.

Tuesday, June 2, 2009

Resolved -- GoGrid Portal Issues

** Update @ 1:00 PM PDT **
Connectivity has been restored. https://my.gogrid.com is back online. We have identified the source of the issue, and has resolved it.

Customer Impact:
You may experience connectivity issues or slow performance while browsing https://my.gogrid.com.

Impact Details:
We're aware that there are performance issues with https://my.gogrid.com. We are working quickly to ascertain the root cause of the performance issues and resolve the issues.

Monday, June 1, 2009

Completed -- Generator Run

Maintenance Window:
Thursday, June 4, 2009 19:00 - 23:59 PDT
(Friday, June 5, 2009 02:00 - 07:00 GMT)

Maintenance Details:
Generators are undergoing standard monthly testing.

Customer Impact:
There should be no impact to the power in the facility and no down time is expected.  This notice is for advisory purposes only.

If there are any additional questions before, during, or following the maintenance, please direct them to GoGrid Support Center via phone or open a case online at http://my.GoGrid.com.

For the latest details on GoGrid's systems, datacenter status, and status of this maintenance item, please visit  http://www.gogridstatus.com.  Thank You for your continuing support of GoGrid.

Wednesday, May 27, 2009

Completed -- GoGrid Portal Maintenance

GoGrid network engineers will be performing network maintenance that will potentially reset your active session on my.gogrid.com.  Service should be restored almost immediately and you will be able to re-login to my.gogrid.com.

Maintenance will occur at approximately 17:30 PDT.

Saturday, May 23, 2009

Resolved - my.gogrid.com offline

** Update @ 5:09 AM PDT **
Connectivity has been restored. https://my.gogrid.com is back online.

Public RFO
The load balancers in front of my.gogrid.com were in an unstable state. To resolve the issue, we disabled one and failed over, resulting in service being restored. We apologize for the service interruption.

Customer Impact
We're aware of a problem with my.gogrid.com and are presently working to resolve the issues. The GoGrid API has not reported having the same issue and should be available.

Friday, May 22, 2009

Resolved -- my.GoGrid.com Offline

** Update @ 10:29 PDT **
Connectivity has been restored.  https://my.gogrid.com is back online.

Public RFO
The load balancers in front of my.gogrid.com were in an unstable state.  To resolve the issue, we disabled one and failed over, resulting in service being restored.  We apologize for the service interruption.

Customer Impact
We're aware of a problem with my.gogrid.com and are presently working to resolve the issues.  The GoGrid API has not reported having the same issue and should be available.

Tuesday, May 19, 2009

Completed -- GoGrid Portal Maintenance

** Update @ 22:22 PDT **
This maintenance has been completed and access to http://my.gogrid.com has been restored. If you have reached this page while attempting to navigate to http://my.gogrid.com, please clear your browser’s cache and try again.

Maintenance Window:
Thursday, May 21th, 19:00 – 23:00 PDT
(Friday, May 22th, 02:00 – 06:00 GMT )

Customer Impact:
The GoGrid customer portal will be unavailable for up to 4 hours while we upgrade infrastructure. However, there will be no impact to existing customer servers and load balancers during this time. Please note that the API will not be impacted.

Maintenance Details:
GoGrid is making software improvements to "my.gogrid.com" to support increased demand.


Note: This maintenance has been completed and access to http://my.gogrid.com has been restored. If you have reached this page while attempting to navigate to http://my.gogrid.com, please clear your browser’s cache and try again.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.

Resolved -- GoGrid Packet Loss

RFO
On May 19th, 2009, from approximately 10:18 to 10:50 a.m. PDT, one of the GoGrid routers had an issue that caused packet loss. The issue has now been resolved. We are investigating the root cause and a detailed RFO will be forthcoming shortly.

** Update @ 10:58 PDT **
Service has been restored, all problems have been resolved.  If you do continue to have problems please open a case at https://my.gogrid.com.

** Update @ 10:53 PDT **
To resolve packet loss we've rebooted the affected router.  Service should be restored in approximately 5 minutes

Customer Impact
We're aware of packet loss to one of our GoGrid routers.  We're working to restore connectivity.

Friday, May 15, 2009

Completed -- GoGrid Portal Maintenance

** Update **
Maintenance was unsuccessful.  This will be re-scheduled.

Maintenance Window:
Monday, May 18th, 19:00 – 23:00 PDT
(Tuesday, May 19th, 02:00 – 06:00 GMT )

Customer Impact:
The GoGrid customer portal will be unavailable for up to 3 hours while we upgrade infrastructure. However, there will be no impact to existing customer servers and load balancers during this time. Please note that the API will not be impacted.

Maintenance Details:
GoGrid is making software improvements to "my.gogrid.com" to support increased demand.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.

Monday, May 11, 2009

Canceled -- GoGrid Portal Maintenance

** Update 5/14/2009 @ 19:45 PDT **
Maintenance has been canceled.  We will re-schedule this in the future and notify you again.

Maintenance Window:
Thursday, May 14th, 19:00 – 23:00 PDT
(Friday, May 15th, 02:00 – 05:00 GMT )

Customer Impact:
The GoGrid customer portal will be unavailable for up to 3 hours while we upgrade infrastructure. However, there will be no impact to existing customer servers and load balancers during this time. Please note that the API will not be impacted.

Maintenance Details:
GoGrid is making software improvements to "my.gogrid.com" to support increased demand.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.

Friday, April 24, 2009

Resolved - Sign up outage

If you tried signing up for our service over the last few hours the sign-ups were failing due to a network timeout error from Authorize.net we have contacted them to resolve this issue and we are now back up and running 100% we apologize for any inconvenience this may have caused.

Wednesday, April 15, 2009

Resolved - Minor Bug Found In API Functionality

** Update **



API hotfix was successfully applied during our emergency maintenance at 17:00 PST. The API behavior is now correct and all list items are returned for all API calls returning list data over 100 items in length.

Tuesday, April 14, 2009

Minor Bug Found In API Functionality

** Update **

There will be an emergency maintenance at 17:00 PDT (24:00 GMT) today, April 15, 2009, to support pagination within our API to work around this bug.  Please refer to updated documentation on GoGrid's Wiki site to get the updated syntax for the new pagination variable 'num_items' at http://wiki.gogrid.com.

Our most recent release introduced a minor bug in API functionality. Returned data lists greater than 100 entries may be truncated. This will usually only occur when listing private IP addresses via the API. You will not be impacted unless using the API and attempting to list private IP addresses. We expect a hotfix will be scheduled soon. This entry will be updated when we have more details.

Friday, April 10, 2009

Completed -- System Maintenance

Maintenance Window:
Tuesday, April 14, 01:00 - 05:00 PDT
(Tuesday, April 14, 08:00 - 12:00 GMT)

Customer Impact:
The customer service portal my.gogrid.com and order/billing systems will be unavailable.  The API will be available, but will be queuing requests until maintenance is completed.

Maintenance Details:
We are upgrading a core database layer to provide additional capacity to the service.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.

Thursday, April 9, 2009

GoGrid Update from the Founders

After the events of last week, we would like to publicly answer the common questions we are receiving from many of you, our customers who were impacted by last week's network attacks.

What is GoGrid  doing so that this issue does not happen again?
Network attacks are a fact of life on the Internet, though generally their impacts on customers can be prevented. In the past week we have seen large scale attacks on a number of companies besides GoGrid (UltraDNS, Register.com, and The Planet). GoGrid has taken the following actions to better prepare ourselves for future attacks:

Partnership with Cisco
We had attack prevention systems in place that generally deflect multiple attacks each day, but were unable to fully handle the scale and new type of these attacks. We are working closely with our vendor partner, Cisco, and others to identify any known vulnerabilities in their products or our architecture that could weaken our services and defenses.

Network Infrastructure Improvements
Our network team, working with Cisco and a network engineering consultant, has made a number of improvements to our network defense systems, router configurations, and monitoring and alerting systems.  For security reasons, we don't publish the specifics of these changes, but would be happy to communicate with any of you individually if you have concerns that you would like addressed.

What will happen differently if there is another outage?
While we work to prevent future outages and events that degrade service quality, we are simultaneously improving internal and external communications. Some of the steps we are taking are as follows:

Faster internal escalation and alarm procedure
We have refined our internal policies to improve response times so that any employee can ring a virtual fire alarm with immediate response from Networking, Systems and Engineering. The goal is to save precious seconds and minutes.

Improved external communication at http://www.gogridstatus.com
In the event of any outage, the status blog will be updated more frequently with as much information as possible. We recommend that all clients subscribe to the RSS feed from this blog. Please consider GoGridStatus.com as your first resource to learn of any issues affecting your GoGrid service.

Expanded phone and chat coverage
We are accelerating our plans to expand our in-house customer service organization. Once hiring is complete, this group will be available to answer all commonly asked questions (product, billing, sales, account) and augment our existing 24/7 technical support. This means that more people will be available in the near future to answer phone and chat and keep you up to date.

What is the status of GoGrid's 10,000% guarantee?
GoGrid's Service Level Agreement (SLA) documents our commitment to our customers' success.

SLA Credits Issued
Understanding that many of our customers depend upon our product to run their business and expect a high level of service, we issued SLA credits to all GoGrid customers who we knew were impacted by the outage.  The Company was not obligated to pay these credits, but we wanted to extend this quick credit to demonstrate GoGrid's strong commitment to our customers.

What else is going on with GoGrid?

New Service Teams
As of April 1, all GoGrid customers now have their own Service Team. This means there is a Sales Account Manager, a Support Systems Administrator, a Billing Associate, a Data Center Technician, and a Network Engineer all on a team dedicated to your account. The same people (during business hours PDT) will be helping you get the most out of GoGrid, and you can get to know them and they can get to know you and your infrastructure needs. (On call technical support remains available 24/7, our growing in-house customer service team is available to you as well, and both of these organizations report any issues into your assigned Service Team). To learn more about service teams, please visit http://www.gogrid.com/support/service-teams.php

Expanded GoGrid Features
We are listening and responding to your requests for key features as quickly as we can. The ability to save your own image to deploy additional prebuilt servers, or to park the data in cloud storage so you don't have to pay RAM fees when you are not using the server, will be here in the beginning of the summer. Other features and improvements to the portal are in the works as well.

Once again, I wanted to personally thank you for your business as we continue to execute on our goal of offering the most flexible and easiest to use cloud computing infrastructure.

Best Regards,
David Hecht
Co-Founder & Chief Marketing Officer

Resolved -- Verio Offline

** Update @ 4/10 17:00 PDT **
Verio service has been restored, and traffic is being balanced.  No customers were affected.

Customer Impact
At 05:34 PDT (GMT -7) our BGP session with Verio disconnected.  We're presently working with Verio to resolve the problem.  All traffic automatically failed over to our other transit providers.

Wednesday, April 8, 2009

Resolved -- Emergency Portal Maintenance

Maintenance Window:
Wednesday, April 8, 2009, 16:00 - 17:00 PDT
(Wednesday, April 8, 2009, 23:00 - 24:00 GMT)

Customer Impact:
The my.gogrid.com customer portal will be unavailable.  The API will not be affected by this maintenance.

Details:
We will be deploying some patches to the GoGrid customer portal to resolve issues that prevent new user creation in the user management section under My Account.

If there are any additional questions please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.

Resolved -- GoGrid Load Balancer Add/Delete

** Update @ 4/14 16:30 PDT **
A change in the F5 API resulted in updated error code handling in our system.  We've now pushed updates to the engine which deploys load balancers.  A side-effect is that the lose artifacts of prior create/delete failed jobs must be cleaned up first.  If you continue to experience problems, please open a case and refer to problem identified in case 00203213.

Customer Impact:
Load Balancer additions and deletions fail.

Details:
Presently we're investigating recent failed add and delete jobs for GoGrid Load Balancers.  We'll update you when we have more information.

Until a resolution is reached, we suggest the following.

If a delete load balancer job failed:
The Load Balancer shouldn't be answering.  If it is not answering, then use the same IP on your secondary public interface of a system that can answer (Local Area Connection 3 on Windows systems, and eth2 on Linux Systems).  Please submit a case to GoGrid Support to clean up the remaining Load Balancer artifacts as your Load Balancer is only partially deleted.  We can remove the bad database entries and clean up the hardware.

If you cannot deploy a load balancer:
Use the IP you would use on the Load Balancer as a secondary IP on your secondary public interface of your system (Local Area Connection 3 on Windows systems, and eth2 on Linux Systems).

Monday, April 6, 2009

Completed -- GoGrid Emergency Maintenance - Load Balancers

** Update @ 20:05 **
Maintenance was successful and the suggested vendor patch didn't resolve the defect identified.  We'll be consulting with the vendor to seek a resolution.

Maintenance Window:
Monday, April 6, 2009 18:00 - 20:00 PDT
(Tuesday, April 7, 2009 01:00 - 03:00 GMT)

Customer Impact:
There should be no customer impact as GoGrid's load balancers are a redundant pair.

Details:
The firmware on the load balancers are being upgraded to a version recommended by the vendor in order to fix a software defect that causes depletion of shared memory resulting in occasional loss of connectivity for some customers.

If there are any additional questions please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.  You may also open a support case at https://my.gogrid.com.

Friday, April 3, 2009

Resolved -- Scheduled UPS Maintenance

Maintenance Window:
Tuesday, April 7, 2009, 10:00 - 17:00 PDT
(Tuesday, April 7, 2009, 17:00 - 24:00 GMT)

And

Wednesday, April 8, 2009, 10:00 - 17:00 PST
(Wednesday, April 8, 2009, 17:00 - 24:00 GMT)

Customer Impact:
There should be no down time.  This notice is for advisory purposes only.

Maintenance Details:
Tuesday, April 7, 2009. We will be performing maintenance on the battery cabinets for UPS 1 and 2. No unit will be put into bypass, but we will have limited battery backup during the maintenance.

Wednesday, April 8, 2009. We will be performing battery maintenance and full diagnostic maintenance on UPS 3 and 4. Both units will be placed in bypass (1 at a time) during the maintenance.

If there are any additional questions please direct them to GoGrid Support at 1-877-9Go-Grid or 415-869-7444 option 2.  You may also open a support case at https://my.gogrid.com.

Wednesday, April 1, 2009

Resolved -- GoGrid Portal

** Update @ 16:20 **
Problems with logging in to https://my.gogrid.com are now resolved.

We're aware that https://my.gogrid.com is not accepting login information for some customers.  Please do not attempt to log in more than twice while we are resolving the problem.

The GoGrid API is still operational and not affected.

Tuesday, March 31, 2009

Update from GoGrid Founders

It has been a long couple of days here at GoGrid. We are hearing from many of you that you want more information, more often. Engineers are often working so hard to fix an issue that you they don't give you enough visibility into what is going on.

In that spirit of improved communication and transparency, I want to continue sending updates via email.

While some of you have been unscathed by the network attacks and portal issues over the past few days, we know that many of you are frustrated at the downtime and impacts to your Internet infrastructure. We are frustrated as well. We've been in the hosting business for over 8 years now, and have generally been able to prevent most incidents from impacting customers as heavily as this attack did.

ONGOING DDoS ATTACK
Our network is currently the target of a large, distributed DDoS attack that began on Monday afternoon.   We took action all day yesterday to mitigate the impact of the attack, and its targets, so that we could restore service to GoGrid customers.  Things were stabilized by Monday, March 30, 2009 16:00 PDT and most customer servers were back online, although some of you continued to experience intermittent loss in network connectivity.

We had a maintenance window scheduled for Monday, March 30, 2009 21:00 PDT to do a major expansion of GoGrid's capacity and roll out some minor feature improvements and bug fixes. Because this maintenance window required the portal being down and support cases would have to be opened by phone, we considered postponing the maintenance to a time when things were calmer.

In the end, the decision was made to proceed with the maintenance because this capacity expansion had been planned for several months and would give us more flexibility in ensuring low utilization across our infrastructure. In hindsight this may have been a poor decision because the maintenance took longer to complete and the maintenance window had to be expanded by several hours.

ROUTING ISSUES THIS MORNING
We spent the night cleaning up servers that were still down, reboots that did not happen properly, and other issues, and continued to develop plans to establish a long term solution to this ongoing issue.

Beginning early Tuesday, March 31, 2009 PDT, our support team began to get more and more reports of servers that were unreachable from certain parts of the Internet. All of these servers were pingable and accessible from our testing connections outside the GoGrid network, but not to all locations worldwide. There appeared to be a routing issue with some networks not properly announcing GoGrid routes. Some of your web sites appeared offline to most or all of your own customers, while many were unaffected.

CURRENT STATUS
The routing issue was resolved around Tuesday, March 31, 2009 11:00 PDT.  Our network engineers localized the problem to an issue with our border routers improperly announcing some routes.  The issue was resolved by clearing our BGP cache on our border routers.  We are not certain at this time the root cause of the issue, and are continuing to investigate and will provide an RFO soon to customers who opened Cases.  We suspect the issue had something to do with the changes we implemented in an emergency maintenance window, yesterday, as part of our efforts to mitigate the DDoS attack.

If you are continuing to see any connectivity issues with your GoGrid servers, we ask that you run a traceroute to your servers' IP address so you can provide it to our support staff when logging an issue at http://my.GoGrid.com.

We appreciate your patience during this difficult time, and thank you for being a GoGrid customer.

Regards,
David Hecht
Co-Founder & Chief Marketing Officer

Resolved -- GoGrid Network Connectivity Issues

** UPDATE @ 13:16 PDT **
At this time we are no longer experiencing network connectivity issues with the public GoGrid Network Infrastructure. While the problem does seem to be resolved, we will not label it as such until a full Root Cause Analysis is complete. Please stand by for further updates.

** UPDATE @ 11:01 PDT **
We are making progress on the network connectivity issue which is related to specific ISP's and Routes coming into the GoGrid Network. You may notice that your systems are becoming accessible, though we are not noting this as "fixed" yet. If you are still experiencing a failure, please take the time to run a traceroute to your systems IP and send that to the GoGrid Support Staff

Customer Impact
We are currently experiencing an issue where our network is inaccessible from certain parts of the world and other networks. We are treating this as a top priority and expect resolution shortly. If you are opening a Case for this issue please try and include a traceroute so that we can use that information to help us isolate the issue.

Details
On Friday, March 27,2009 11:10 PDT, and again yesterday, Monday, March 30,2009 12:25 PDT, GoGrid suffered a series of large scale distributed denial of service (DDoS) attacks that affected the network connectivity of many GoGrid servers.

These network attacks were of a type that we had not seen before, and which our automated network attack prevention hardware was unfortunately unable to prevent.

We are still experiencing network connectivity issues resulting from the attacks, and a number of GoGrid servers remain unreachable from certain source networks.  Our network engineers are working continuously to resolve the problem.

Monday, March 30, 2009

Completed -- GoGrid Maintenance

** Update **
Maintenance was extended and completed at 06:45 PDT.  API and GoGrid Portal are now available.

Maintenance Window:
Monday, March 30, 2009 21:00 - 04:45 PDT
(Tuesday, March 31, 2009 04:00 - 11:45 GMT)

Customer Impact:
The GoGrid customer portal and API access will be unavailable for up to 4 hours while we upgrade the infrastructure. However, there will be no impact to existing customer servers and load balancers during this time.

Maintenance Details:
GoGrid is making software improvements to https://my.gogrid.com to support increased demand.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid (946-4743) or 415-869-7444 option 2.

Resolved -- Add/Delete/Restart/Start Slowness

** Update @ 18:00 PDT **
Queue cleared and service is back to normal

Window
Monday, March 30, 2009 16:48 - current PDT
(Monday, March 30, 2009 23:48 - current GMT)

Customer Impact
Many customers are affected

Details
Due to our recent DDoS attack we have noticed a lengthy queue build up with start, restart, and delete jobs.  This is an unexpected side-effect of mitigating the DDoS attack.  We are working to resolve this issue and will update this post again.

Resolved -- GoGrid Network Connectivity

** Update @ 17:35 PDT **
Thank you for your patience while GoGrid network engineers troubleshoot a DDoS attack on the GoGrid network.  A recent QoS policy put in place is affecting legitimate traffic at this time and we're working to resolve the problem.  An RFO is still pending and will be following when available.

Window
Monday, March 30, 2009 12:22 - current PDT
(Monday, March 30, 2009 19:22 - current GMT)

Customer Impact
This event is affecting some customers

Details
We are currently experiencing a network issue that is affecting part of our GoGrid infrastructure. Our network team is working hard to get your servers back online and more information will be forthcoming as soon as it is available.  GoGrid virtual machines are still on and still running.

If there are any additional questions before, during, or following this event please direct them to GoGrid Support by opening a support case at https://my.gogrid.com.

Friday, March 27, 2009

RFO - Network Service Disruption 3/27 @ 19:05 PDT (GMT -7)

Window
Friday, March 27, 2009 19:05 - 19:13 PDT
(Saturday, March 28, 2009 02:05 - 02:13 GMT)

Customer Impact
This event affected many customers

Details
While performing a proactive maintenance on the infrastructure to increase security and improve resilience to malicious attacks against our network, an incorrect configuration command was entered on the border routers which started to gradually block legitimate traffic. As soon as the maintenance engineer was notified by our monitoring tool, the erroneous configuration was reverted, and connectivity was restored.

RESOLVED - Temporary Network Access Issue

***UPDATE: The issue listed below has been resolved. A detailed Reason for Outage (RFO) will be sent to customers affected.***

Some of our clients are experiencing a lack of network connectivity to their Virtual Machines. This is due to network difficulties only and does not impact the integrity or uptime status of the Virtual Machine itself.

We apologize for this inconvenience and have made this issue the highest priority.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid (946-4743) or 415-869-7444 option 2.

Wednesday, March 25, 2009

Resolved -- Customer Portal (my.gogrid.com)

Some customers are experiencing intermittent behavior on the my.gogrid.com portal, where the website is inaccessible for brief periods of time. Our engineers are actively looking at this issue and expect to resolve problems shortly. Please note this does not impact your virtual machines or load balancers in any way. Thank you.

Wednesday, March 18, 2009

Completed -- System Maintenance

Maintenance Window (Re-scheduled):

Friday, March 20th, 01:00 - 06:00 PDT
(Friday, March 20th, 08:00 - 13:00 GMT)

Customer Impact:
The customer service portal my.gogrid.com and order/billing systems will be unavailable.

Maintenance Details:
We will be performing internal system upgrades to improve the customer experience.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid (946-4743) or 415-869-7444 option 2.

Friday, March 6, 2009

Completed -- Cloud Storage Maintenance

Maintenance Window:
Tuesday, March 10, 2009 21:00 - 22:00 PDT
(Wednesday, March 11, 2009 04:00 - 05:00 GMT)

Customer Impact:
GoGrid Cloud Storage will be unavailable for up to 20 minutes.

Maintenance Details:
GoGrid is doing some maintenance work on the Cloud Storage network to improve performance and scalability.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid (946-4743) or 415-869-7444 option 2. You may also open a support case at https://my.gogrid.com.

Tuesday, March 3, 2009

Completed -- Generator Run

Maintenance Window:
Thursday, March 5, 2009 20:00 - 00:00 PDT
(Friday, March 6, 2009 04:00 - 08:00 UTC)

Customer Impact:
There will be no customer impact.

Maintenance Details:
GoGrid will be performing a generator run and all power will be swapped over from city utility to our generators for up to four hours.

If there are any additional questions before, during, or following the maintenance please direct them to GoGrid Support at 1-877-9Go-Grid (946-4743) or 415-869-7444 option 2.  You may also open a support case at http://my.gogrid.com.