News Ticker

Recent Salesforce Outage and Next


Recently Salesforce.com database in North America NA14, suffered a nasty outage which lasted over two days and a 4 hours’ worth of data loss due to system failure. Salesforce CEO Marc Benioff tried to play the data loss down saying the data loss happened during the US night time and hence data loss is not significant. However, Salesforce also did not reveal the number of customers affected by this outage.

Salesforce Outage

Root Cause

As per Salesforce.com the root cause of the issue is power failure to the Washington, D.C. (WAS) datacenter which is caused due to a main critical board that tripped open a circuit breaker pair. The breakers are used to segment power from the datacenter ‘s universal power supply ring to direct into different rooms. This fault created an uncertain power condition, which led to a redundant breaker not closing to activate the backup feed because that electronic circuit breaker could not confirm the state of the problem board.

The data loss was caused by the file discrepancies in the NA14 database caused by database cluster failure. Additionally, it was found that the file discrepancies had been copied to the database in the WAS data center and that the local backup had not yet completed in the CHICAGO (CHI)data center. Because the file discrepancies had been copied to the standby copy of the database in the WAS data center, a site switch back to this copy of the NA14 instance in WAS was not a viable option to restore service to NA14 customers. Additionally, because the local backup of the instance in the CHI data center had not completed, this copy could not have been used to correct the file discrepancies and restore service. The root cause of the database failure was determined to be a latent firmware bug that was exposed due to increased traffic volume resulting from a combination of factors.

Impact

This was a nasty blow to Salesforce. Gartner VP Yefim Natis told InformationWeek in an interview, noting that Microsoft Azure and Amazon AWS have not been immune to outages either. “Outages happen to everyone. They vary in impact. This one was bad. People lost data. They didn’t have access to systems for a long period of time.”

This outage has affected customers on the US west coast and while service was restored on Wednesday after nearly a full day of down time, the instance continued to experience a degradation of service for quite some time. Salesforce Trust page showed the below message during this service degradation –

“The NA14 instance continues to operate in a degraded state. Customers can access the Salesforce service, but we have temporarily suspended some functionality such as weekly exports and sandbox copy functionality. In addition, search indexing may be delayed.”

What’s Next

While Salesforce stated that they have taken corrective measures to ensure such incidents do not occur in future; however, this incident has raised questions on Salesforce’s cloud capability with growing demand and aging storage.

Gartner VP Yefim Natis told “This happening once will be forgiven”. However, if this Salesforce outage is an indication of things to come for Salesforce’s systems, customers may start looking for other solutions.

Microsoft Dynamics CRM: Zero Data loss so far

Microsoft Dynamics CRM uses highly reliable Microsoft Azure datacenters and provides 99.9% financially backed SLA. Dynamics CRM Online has so far ZERO DATA LOSS and is compliant to EU Model clauses and many other regulatory certifications.

Microsoft Dynamics CRM Online stores FIVE (5) copies of data. TWO Synchronous for high availability in the same 
datacenter and TWO Asynchronous for backup in different datacenters within same continent and ONE nightly backup. 
Microsoft offers geographical redundancy to its customers by replicating data between two geographically distant sites 
in same continent.

Microsoft Data Centers

 

The following World map shows Microsoft Dynamics CRM Online data centers across the globe.

This ensures higher degree of data integrity and data loss prevention in case of any outage or database failure. Microsoft also provides 5 layers of security protection, including EU and ISO27001 standards compliance. Microsoft Dynamics CRM Online is also IRAP certified for Australian Customers.

Microsoft Data Centers Specialities

About Dipankar Bhattacharya (59 Articles)
A multi-skilled Dynamics 365 Professional with strong experience in delivering IT projects especially across multiple industries. A Microsoft technology evangelist, a regular speaker at tech events, blogger and avid reader. Certified IT Architect and well versed in Solution Architecture of Business Applications using Microsoft platforms like Dynamics 365, Azure and Office 365.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: