Analysis of our recent downtime:

Post Reply
robotman
Admin Emeritus
Admin Emeritus
Posts: 1447
Joined: June 29th, 2008, 7:37 pm
Division: Grad
State: TX
Has thanked: 0
Been thanked: 3 times
Contact:

Analysis of our recent downtime:

Post by robotman »

Since the nature of Scioly.org is sharing information, I thought that it would be the correct response to explain the cuases of our recent downtime, as well as the step taken to fix the problems.
March 6th:
4 PM:
It started with myself starting a backup of the wiki and forum databases. The attempt to backup the databases ended with an error, regarding MYSQL. Due to this (and the fact it seemed I had a MYSQL database sitting in the update queue) I rand a full server update. This however was not my smartest move, the update caused MYSQL to crash, and not be revivable, thus the board and the wiki were both down.
Once I realized that MYSQL had crashed, I backed up the databases, and the necessary files (necessary being tests, and images). With backups complete (they took the most time of this whole process) I then started work on the mysql database, attempts at installation failed due to a package error in Ubuntu-Server Edition. Attempts at installation via separate packages failed due to running out of disk space errors. At this point I decided that way to return scioly.org as quickly as possible was to wipe our server and start from scratch.
At 9PM CST I started that process (yes backups took that long, halfway through I realized there was a more efficient method, however I was already committed). I started with restoration of the board. I installed phpBB from scratch, and then dealt with uploading content. By 12:03AM I had restored the board to working order, then focused on getting the wiki back in shape. I updated our Mediawiki install to the latest version (That will come in later) and had it back up with most of its content by 2:15 AM.

March 7th:
At 2:15 AM I had fixed the wiki, minus theme, and tests and hit the sack in order to make it to school. At 7am I started uploading the backed up tests to the wiki, which should have been complete midday.

March 8th to March 15th:
The server was mostly up, however due to a slight misconfiguration of apache, with enough load the server would crash, Due to being on vacation, I had limited access to internet, and manage to fix the server crashing I believe on Sunday the 10th. or monday the 11th.
However I had to leave emails down, as well as wiki login.

March 16th:
I restored email functionally, and resetup the administrator emails.
I fixed (and improved) wiki login, it is now a Single Sign In for both wiki and forum.
I have been working on formatting issues with the wiki templates.

If you have any questions (or complaints) feel free to ask(or inform me).
Edit the Wiki.
Upload to the Image Gallery
[medals]Get Medals[/medals]
[chat][/chat]
mnstrviola
Wiki/Gallery Moderator Emeritus
Wiki/Gallery Moderator Emeritus
Posts: 433
Joined: February 15th, 2011, 5:45 pm
Division: Grad
State: CA
Has thanked: 0
Been thanked: 0

Re: Analysis of our recent downtime:

Post by mnstrviola »

robotman wrote: I fixed (and improved) wiki login, it is now a Single Sign In for both wiki and forum.
Yay! This is much more convenient :)
twototwenty
Exalted Member
Exalted Member
Posts: 292
Joined: March 24th, 2011, 10:28 am
Division: Grad
State: NY
Has thanked: 0
Been thanked: 0

Re: Analysis of our recent downtime:

Post by twototwenty »

The wiki is still giving me issues: the single sign-in is not working at all for me, and when I try to log in on the wiki, I get this:
No such special page
You have requested an invalid special page.

A list of valid special pages can be found at Special pages.

Return to Main Page.
Luo
Staff Emeritus
Staff Emeritus
Posts: 507
Joined: March 21st, 2011, 1:31 pm
Division: Grad
State: MN
Has thanked: 0
Been thanked: 13 times

Re: Analysis of our recent downtime:

Post by Luo »

twototwenty wrote:The wiki is still giving me issues: the single sign-in is not working at all for me, and when I try to log in on the wiki, I get this:
No such special page
You have requested an invalid special page.

A list of valid special pages can be found at Special pages.

Return to Main Page.
This was confusing to me at first too (I encountered the same thing), but simply don't try to log in on the wiki at all. If you're logged in on the forums and you try to edit a wiki page, it should work. Let me know if it doesn't.
Proud alumnus of Mounds View High School Science Olympiad, Arden Hills, MN
Co-founder of the MIT Science Olympiad Invitational Tournament: http://scioly.mit.edu/
astro124
Member
Member
Posts: 44
Joined: November 23rd, 2012, 1:44 pm
Division: C
State: AZ
Has thanked: 0
Been thanked: 0

Re: Analysis of our recent downtime:

Post by astro124 »

I'm sorry but I don't really understand a lot of computer stuff (especially programming).

What is 'MYSQL'?
2012 Season: Reach for the Stars-3rd (State) / Keep the Heat-19th (State)
2013 Season: Astronomy-2nd (State) / Disease Detectives-15th (State)
2014 Season: Astronomy-1st (State) / Experimental Design-20th (State)/ Anatomy and Physiology-16th (State)
2015 Season: Astronomy

Thunder
iwonder
Admin Emeritus
Admin Emeritus
Posts: 1115
Joined: May 10th, 2011, 8:25 pm
Division: Grad
State: TX
Has thanked: 0
Been thanked: 1 time

Re: Analysis of our recent downtime:

Post by iwonder »

It's a database engine, basically it's the program that stores all the posts and wiki pages and things of that nature.
'If you're the smartest person in the room, you're in the wrong room' - Unknown
User avatar
Infinity Flat
Member
Member
Posts: 274
Joined: March 12th, 2010, 4:00 pm
Division: Grad
State: WA
Has thanked: 0
Been thanked: 0

Re: Analysis of our recent downtime:

Post by Infinity Flat »

Just FYI, some of the tests on the test exchange are still unavailable.
(State, Nationals)
2013: Astro (2, 6) / Chem (2, 5) / Circuits (8, 36) / Diseases (1,1) / Fermi (N/A, 24) / Materials (1, N/A)
2012 : Astro (1, 11) / Chem (N/A, 13) / Diseases (3, 1) / Optics (2, 3) / Sounds (2, 1)
2011: Astro(2,11) / Diseases (1,27) / Optics (1,13) / Proteins (2,15)
User avatar
foreverphysics
Moderator
Moderator
Posts: 497
Joined: May 20th, 2011, 8:41 pm
Division: Grad
State: AL
Has thanked: 0
Been thanked: 69 times
Contact:

Re: Analysis of our recent downtime:

Post by foreverphysics »

Which ones?
Image
Physics is difficult for 99% of the world's population because they don't understand it. The other 1% know too much.

"Physics is a psychiatrist?"
User avatar
Infinity Flat
Member
Member
Posts: 274
Joined: March 12th, 2010, 4:00 pm
Division: Grad
State: WA
Has thanked: 0
Been thanked: 0

Re: Analysis of our recent downtime:

Post by Infinity Flat »

foreverphysics wrote:Which ones?
This was just someone pointed out to me last night - the only ones I'm aware of at the moment are for experimental design: the Conestoga 2012 and Wright State Invitational
(State, Nationals)
2013: Astro (2, 6) / Chem (2, 5) / Circuits (8, 36) / Diseases (1,1) / Fermi (N/A, 24) / Materials (1, N/A)
2012 : Astro (1, 11) / Chem (N/A, 13) / Diseases (3, 1) / Optics (2, 3) / Sounds (2, 1)
2011: Astro(2,11) / Diseases (1,27) / Optics (1,13) / Proteins (2,15)
Post Reply

Return to “Announcements”

Who is online

Users browsing this forum: No registered users and 9 guests