Google Apps Incident Report Gmail Delivery Delays - March 16, 2010 Prepared for Google Apps Premier Customers

The following is the incident report for email delivery delays that some Google Apps Gmail customers experienced on March 16. We understand that this service issue has affected our valued customers and their users, and we apologize for the disruption. Issue Summary Beginning at 5:34 AM PDT | 12:34 GMT, Tuesday, March 16, the affected customers experienced delays with incoming messages and received errors when sending outgoing messages. As a workaround, users could try to resend outgoing messages. At 3:23 PM PDT | 22:23 GMT, Tuesday, March 16, both inbound and outbound message delivery was restored to normal for all customers. During this time, some incoming messages may have been temporarily deferred; at no time were messages lost or deleted. Actions and Root Causes To help immediately mitigate slow delivery for Gmail, the Google Engineering team increased processing resources for Gmail routing and greatly increased the number of active Gmail routers. Following an internal investigation and analysis, the Google Engineering team determined that the root cause of the Gmail issue was the incorrect allocation of network resources. This misallocation was the result of a combination of the following: • A recent update to expand network capacity generated new routing information. The mechanism that allocates bandwidth resources interpreted this routing data incorrectly and subsequently reduced the network resources available to Gmail. • In this particular case, the network resources were reduced in a way that other internal systems, which normally detect and help manage network demand for Gmail, were not triggered. To resolve the mail delay, Google Engineering corrected the mechanism that allocates bandwidth, and Gmail delivery functions returned to normal. Corrective and Preventative Measures The Google Engineering team conducted an internal review and analysis, and is performing the following actions to help address the underlying causes of the problem and to help prevent recurrence: • Changing the behavior of the resource allocation system to help prevent this class of issue. This action has been completed. • Enhancing monitoring tools to help detect and provide alerts for this misallocation condition. We appreciate your patience and again apologize for the impact to your organization. We thank you for your business and continued support.

Sincerely, The Google Apps Team

Google Apps Incident Report: Gmail Delay, March 16, 2010

Mar 16, 2010 - resources for Gmail routing and greatly increased the number of active Gmail routers. Following an internal investigation and analysis, the ...

53KB Sizes 0 Downloads 296 Views

Recommend Documents

Google Apps Incident Report Gmail Delivery Delays - June 22, 2010
Engineering was made of aware of the problem and promptly began to work to manage excessive traffic ... your business and continued support during this time.

Google Apps Incident Report
This misconfiguration prevented changes to existing customer data for upgraded users. ... Eliminate the need for server restarts to recover from this type of error.

Google Apps Incident Report
At 7:50 AM PT | 16:00 UTC November 15, Google Calendar Engineering brought a system of servers ... your business and continued support during this time.

Google Apps Incident Report
Nov 15, 2010 - Prepared for Google Apps Customers ... Apps customers on November 15, 2010. ... your business and continued support during this time.

Google Apps Incident Report
Feb 27, 2011 - Google Engineering reverted the storage software update, and halted ... better identify this class of bug during the software development cycle.

Google Apps Incident Report
We understand this service issue has impacted our valued customers and users, and we apologize to everyone ... At 6:12 AM PDT, a bug in a thirdparty software update caused a partial failure of a Google network router in ... escalated the software iss

Google Apps Incident Report
Mar 18, 2013 - service disruption was an issue in the network control software. Actions and Root Cause Analysis. At 6:09 AM PT, a portion of Google's network ...

Google Apps Incident Report
Mar 17, 2014 - Issue Summary. From 8:35 AM to 12:10 PM PT, Google Talk, Google Hangouts (including Chat and Video), Google. Voice, and the Google App ...

Google Apps Incident Report
Apr 17, 2013 - The following is the incident report for the Google services access ... Talk, Google Sync, the Admin panel, and the Cloud Console, and to a ...

Google Apps Incident Report
Mar 19, 2013 - Applications using the Google Drive and Docs APIs also returned errors. ... We thank you for your business and continued support. Sincerely,.

Google Apps Incident Report
Feb 27, 2011 - Google Engineering reverted the storage software update, and halted further deployment. Restoration Process. While analyzing the issue and its root cause, Google Engineering also worked on the process to restore users' accounts. At 6:0

Google Apps Incident Report
Apr 17, 2012 - Prepared for Google Apps for Business customers. The following is the ... Enhance internal documentation for configuration management.

Google Apps Incident Report
Google Drive list. Applications using ... The Google Engineering team conducted an internal review and analysis of the March 21 event. They ... Modify the Drive software to more reliably serve user requests during short periods where overall.

Google Apps Incident Report
Dec 10, 2012 - Actions and Root Cause Analysis. Background: The load balancing software routes the millions of users' requests to Google data centers.

Google Apps Incident Report
Google Apps Incident Report. Google Docs Outage - September 7, 2011. Prepared for Google Apps for Business customers. The following is the incident report ...

Google Apps Incident Report
Google Apps Incident Report. Gmail Outage - September 23, 2011. Prepared for Google Apps for Business customers. The following is the incident report for the ...

Google Apps Incident Report - PDFKUL.COM
Apr 17, 2013 - The following is the incident report for the Google services access disruption that occurred on. April 17 ... Talk, Google Sync, the Admin panel, and the Cloud Console, and to a lesser extent Groups,. Sites, and ... misconfiguration oc

Google Apps Incident Report - PDFKUL.COM
Google Docs Outage - September 7, 2011. Prepared for Google Apps for Business customers. The following is the incident report for the Google Docs access ...

Google Apps Incident Report: August 25, 2010 - Duplicate Mail
from the Google Engineering team traced this problem to new code introduced at 3:00 PM PDT |. 22:00 UTC August 19. The Google Engineering team repaired ...

Google Apps Incident: May 11 to 13 - Gmail
May 11, 2010 - Google Apps Gmail - May 11 through May 13, 2010. Prepared for Google ... for your business and continued support during this time. Sincerely,.

Gmail Outage Incident Report - May 15 2009
A routing configuration file was released to production which incorrectly directed large volumes of global web traffic through ... traffic, resulting in timeouts and access delays to some Google services. The service ... continued support. Sincerely,

Incident Report: Google Apps Mail - September 01 2009
Sep 1, 2009 - On Tuesday, September 1, a small portion of Gmail's web capacity was taken ... request routing automatically directs users' requests to available servers. ... Over the next few weeks, we are dedicated to implementing these ...

Google Apps Incident Report 1 - Sept 24 - Service Disruption
Sep 25, 2009 - Between 7:00 AM - 9:50 PDT | 14:00 - 16:50 GMT, Thursday September 24, Google Apps users were unable to access the Contacts feature through the Gmail interface. However, they could view their contacts at an alternate URL. During this p

Google Apps Incident Report 1 - Sept 24 - Service Disruption
Sep 25, 2009 - Prepared for Google Apps Premier Edition Customers. Incident ... add users to their Google Apps accounts. ... business and continued support.