MySQL Failure Following Windows Server 2003 Re-boot

I've just experienced two completely independent, unrelated servers with totally different web-applications reboot and unable to start the MySQL service, one on 1st April, second on 2nd April. Coincidence? Probably. Is there a Microsoft Windows 2003 server patch killing MySQL? Not found any other incidences being Googled... let's see if anyone finds this post also experiencing the same thing...

UPDATE: This appears to have been triggered by a fire at BT on March 31st.

Timeline:

  • 31 March 2010 07:30 hrs fire crews attended the fire at BT Exchange in Paddington, caused by an electrical fault. According to Gradwell, a business ISP, 437 local exchanges and up to 37,500 Datastream circuits have been affected. It said the fire was having nationwide repercussions on communications - http://www.theregister.co.uk/2010/03/31/burne_house_burns/
    • It was speculated that the systems would not be back to normal until Tuesday the 6th of April 2010.
    • The BBC was alerted to the fault by IT consultant Jerry Sanders, who said customers as far afield as Potters Bar, Hertfordshire and Nottingham were reporting problems with Pipex UK broadband coverage.
  • The exchange was flooded for some time and power was not returned until the following morning when the traffic was re-routed.
  • 01 April 2010 09:30 the traffic from the Paddington exchange finally makes it out along the re-routed options and hits our servers.
    • BT said in an updated statement issued at 8:30am this morning. "Customers in other parts of the country may also be affected."
    • Looks like this led to a flux of requests all at the SAME TIME when service resumed, causing MySQL to process lots of requests at once, delaying other requests, which they added to the delay and the servers snowballed into a frozen mess of MySQL connections

Comments

Post a Comment

Popular posts from this blog

Installing LAMP Apache/MySQL/PHP on Chromebook/Chromebox

Google Chromebook / Chromeboxes make excellent mini web-servers for development/testing/office use, or for using in kiosks / taking to tradeshows etc. Amazingly there is enough space on them for a reasonable sized web-site and database, and the oomph is comparable to a medium-size virtual server from one of the popular web-hosts. Firstly, get your Chromebox / Chromebook running Linux by installing Chrubuntu: Follow this guide here ->  http://webonaut.blogspot.co.uk/2012/11/installing-ubuntu-linux-on-your.html Bring up a Terminal. (Remember, your user is called "user" with password "user" by default). Type: sudo apt-get install tasksel sudo  tasksel Use the cursors to select "LAMP" (dom't deselect the default options or you may take away your desktop!) then hit return. (BTW: Now is a good time to try Edubuntu as it is cool, and openssh-server is useful too ;) Give the mysql root user a password when prompted. Test it out! Try this
Read more

Installing Ubuntu Linux on your ChromeBox/ChromeBook

NOTE: THIS WILL WIPE YOUR BOX! (but you only store stuff in the cloud, right?) Switch the developer switch on your machine (it's in a little hole on the back of the Chromebox. Use a paper clip and BE CAREFUL - it's not a strong switch) Switch the machine on. The unhappy face means you are in developer mode (sweet!). You can wait (and put up with the loud double-beep) or press Ctrl+D to continue Log into ChromeOS as normal Open a browser window and press Ctrl+Alt+T to bring up the mildly rubbish in-browser crosh terminal type "shell" to get a decent bash shell type "sudo bash" to get an even better root shell! type " chromeos-firmwareupdate --mode=todev " to update and properly enter developer mode Note: switch back later by typing the reverse into an Ubuntu shell, e.g. " chromeos-firmwareupdate --mode=normal " type "reboot" and follow steps 4,5 and 6 again The next steps are a bit convoluted, but here's a handy
Read more

Changing your Mac OSX Terminal's default text editor

For just this session, simply type: export EDITOR=emacs (or replace emacs with vim , nano or whatever) Personally, I like to use emacs on servers, but on my Mac TextWrangler is way more helpful. One of the great things about TextWrangler is that you can get it to magically pop edit requests out of the Terminal simply by installing it and typing "edit". To make all other programs use this method try: export EDITOR="edit -w" For a more permanent fix, add the same line to your .bashrc file. If you've not already got one of those, type: edit ~/.bashrc (The one exception is git, which you'll need to configure thus:) git config core.editor "edit -w"
Read more