Update: As of 6:15 pm PST We have put in a fix & the services are now up. We are monitoring the logs closely, to see if the race condition below recurs. We will keep you updated of the status. Our development team is also doing an exhaustive review of the impacted parts of the code. If you encounter any errors accessing services, please post a comment here or in the forums. We sincerely regret the inconvenience caused, and we assure you of our vigilant effort to maintain a high level of service.
(Original post 5:35 pm PST) We encountered an unexpected error (a potential race condition) this evening in a part of the common software framework underlying our services, and as a matter of precaution, we have taken down Zoho services (except Zoho CRM & Zoho Projects, which are on a different framework), as we work to isolate the problem. This race condition seems to occur under heavy load (as they typically do), and the past couple of days, our load has been quite heavy, several times the normal load. We have sufficient hardware infrastructure to handle the load, but we detected some potential software race conditions from our logs. Multiple teams are working right now to isolate the issue. We will post an update here as soon as additional information is available.
We sincerely apologize for the inconvenience. We realize so many of you depend on Zoho services, and we take the responsibility you entrust with us seriously. We will bring up the service as soon as possible, likely within 3 hours (8:30 PM Nov 27, Pacific Standard Time i.e GMT - 8 hours).
Comments