Analysis of our recent downtime:
-
- Admin Emeritus
- Posts: 1447
- Joined: Sun Jun 29, 2008 7:37 pm
- Division: Grad
- State: TX
- Has thanked: 0
- Been thanked: 3 times
Analysis of our recent downtime:
Since the nature of Scioly.org is sharing information, I thought that it would be the correct response to explain the cuases of our recent downtime, as well as the step taken to fix the problems.
March 6th:
4 PM:
It started with myself starting a backup of the wiki and forum databases. The attempt to backup the databases ended with an error, regarding MYSQL. Due to this (and the fact it seemed I had a MYSQL database sitting in the update queue) I rand a full server update. This however was not my smartest move, the update caused MYSQL to crash, and not be revivable, thus the board and the wiki were both down.
Once I realized that MYSQL had crashed, I backed up the databases, and the necessary files (necessary being tests, and images). With backups complete (they took the most time of this whole process) I then started work on the mysql database, attempts at installation failed due to a package error in Ubuntu-Server Edition. Attempts at installation via separate packages failed due to running out of disk space errors. At this point I decided that way to return scioly.org as quickly as possible was to wipe our server and start from scratch.
At 9PM CST I started that process (yes backups took that long, halfway through I realized there was a more efficient method, however I was already committed). I started with restoration of the board. I installed phpBB from scratch, and then dealt with uploading content. By 12:03AM I had restored the board to working order, then focused on getting the wiki back in shape. I updated our Mediawiki install to the latest version (That will come in later) and had it back up with most of its content by 2:15 AM.
March 7th:
At 2:15 AM I had fixed the wiki, minus theme, and tests and hit the sack in order to make it to school. At 7am I started uploading the backed up tests to the wiki, which should have been complete midday.
March 8th to March 15th:
The server was mostly up, however due to a slight misconfiguration of apache, with enough load the server would crash, Due to being on vacation, I had limited access to internet, and manage to fix the server crashing I believe on Sunday the 10th. or monday the 11th.
However I had to leave emails down, as well as wiki login.
March 16th:
I restored email functionally, and resetup the administrator emails.
I fixed (and improved) wiki login, it is now a Single Sign In for both wiki and forum.
I have been working on formatting issues with the wiki templates.
If you have any questions (or complaints) feel free to ask(or inform me).
March 6th:
4 PM:
It started with myself starting a backup of the wiki and forum databases. The attempt to backup the databases ended with an error, regarding MYSQL. Due to this (and the fact it seemed I had a MYSQL database sitting in the update queue) I rand a full server update. This however was not my smartest move, the update caused MYSQL to crash, and not be revivable, thus the board and the wiki were both down.
Once I realized that MYSQL had crashed, I backed up the databases, and the necessary files (necessary being tests, and images). With backups complete (they took the most time of this whole process) I then started work on the mysql database, attempts at installation failed due to a package error in Ubuntu-Server Edition. Attempts at installation via separate packages failed due to running out of disk space errors. At this point I decided that way to return scioly.org as quickly as possible was to wipe our server and start from scratch.
At 9PM CST I started that process (yes backups took that long, halfway through I realized there was a more efficient method, however I was already committed). I started with restoration of the board. I installed phpBB from scratch, and then dealt with uploading content. By 12:03AM I had restored the board to working order, then focused on getting the wiki back in shape. I updated our Mediawiki install to the latest version (That will come in later) and had it back up with most of its content by 2:15 AM.
March 7th:
At 2:15 AM I had fixed the wiki, minus theme, and tests and hit the sack in order to make it to school. At 7am I started uploading the backed up tests to the wiki, which should have been complete midday.
March 8th to March 15th:
The server was mostly up, however due to a slight misconfiguration of apache, with enough load the server would crash, Due to being on vacation, I had limited access to internet, and manage to fix the server crashing I believe on Sunday the 10th. or monday the 11th.
However I had to leave emails down, as well as wiki login.
March 16th:
I restored email functionally, and resetup the administrator emails.
I fixed (and improved) wiki login, it is now a Single Sign In for both wiki and forum.
I have been working on formatting issues with the wiki templates.
If you have any questions (or complaints) feel free to ask(or inform me).
-
- Wiki/Gallery Moderator Emeritus
- Posts: 433
- Joined: Tue Feb 15, 2011 5:45 pm
- Division: Grad
- State: CA
- Has thanked: 0
- Been thanked: 0
Re: Analysis of our recent downtime:
Yay! This is much more convenientrobotman wrote: I fixed (and improved) wiki login, it is now a Single Sign In for both wiki and forum.

-
- Exalted Member
- Posts: 292
- Joined: Thu Mar 24, 2011 10:28 am
- Division: Grad
- State: NY
- Has thanked: 0
- Been thanked: 0
Re: Analysis of our recent downtime:
The wiki is still giving me issues: the single sign-in is not working at all for me, and when I try to log in on the wiki, I get this:
No such special page
You have requested an invalid special page.
A list of valid special pages can be found at Special pages.
Return to Main Page.
-
- Staff Emeritus
- Posts: 507
- Joined: Mon Mar 21, 2011 1:31 pm
- Division: Grad
- State: MN
- Has thanked: 0
- Been thanked: 13 times
Re: Analysis of our recent downtime:
This was confusing to me at first too (I encountered the same thing), but simply don't try to log in on the wiki at all. If you're logged in on the forums and you try to edit a wiki page, it should work. Let me know if it doesn't.twototwenty wrote:The wiki is still giving me issues: the single sign-in is not working at all for me, and when I try to log in on the wiki, I get this:
No such special page
You have requested an invalid special page.
A list of valid special pages can be found at Special pages.
Return to Main Page.
Proud alumnus of Mounds View High School Science Olympiad, Arden Hills, MN
Co-founder of the MIT Science Olympiad Invitational Tournament: http://scioly.mit.edu/
Co-founder of the MIT Science Olympiad Invitational Tournament: http://scioly.mit.edu/
-
- Member
- Posts: 44
- Joined: Fri Nov 23, 2012 1:44 pm
- Division: C
- State: AZ
- Has thanked: 0
- Been thanked: 0
Re: Analysis of our recent downtime:
I'm sorry but I don't really understand a lot of computer stuff (especially programming).
What is 'MYSQL'?
What is 'MYSQL'?
2012 Season: Reach for the Stars-3rd (State) / Keep the Heat-19th (State)
2013 Season: Astronomy-2nd (State) / Disease Detectives-15th (State)
2014 Season: Astronomy-1st (State) / Experimental Design-20th (State)/ Anatomy and Physiology-16th (State)
2015 Season: Astronomy
Thunder
2013 Season: Astronomy-2nd (State) / Disease Detectives-15th (State)
2014 Season: Astronomy-1st (State) / Experimental Design-20th (State)/ Anatomy and Physiology-16th (State)
2015 Season: Astronomy
Thunder
-
- Admin Emeritus
- Posts: 1115
- Joined: Tue May 10, 2011 8:25 pm
- Division: Grad
- State: TX
- Has thanked: 0
- Been thanked: 1 time
Re: Analysis of our recent downtime:
It's a database engine, basically it's the program that stores all the posts and wiki pages and things of that nature.
'If you're the smartest person in the room, you're in the wrong room' - Unknown
-
- Member
- Posts: 274
- Joined: Fri Mar 12, 2010 4:00 pm
- Division: Grad
- State: WA
- Has thanked: 0
- Been thanked: 0
Re: Analysis of our recent downtime:
Just FYI, some of the tests on the test exchange are still unavailable.
(State, Nationals)
2013: Astro (2, 6) / Chem (2, 5) / Circuits (8, 36) / Diseases (1,1) / Fermi (N/A, 24) / Materials (1, N/A)
2012 : Astro (1, 11) / Chem (N/A, 13) / Diseases (3, 1) / Optics (2, 3) / Sounds (2, 1)
2011: Astro(2,11) / Diseases (1,27) / Optics (1,13) / Proteins (2,15)
2013: Astro (2, 6) / Chem (2, 5) / Circuits (8, 36) / Diseases (1,1) / Fermi (N/A, 24) / Materials (1, N/A)
2012 : Astro (1, 11) / Chem (N/A, 13) / Diseases (3, 1) / Optics (2, 3) / Sounds (2, 1)
2011: Astro(2,11) / Diseases (1,27) / Optics (1,13) / Proteins (2,15)
-
- Moderator
- Posts: 497
- Joined: Fri May 20, 2011 8:41 pm
- Division: Grad
- State: AL
- Has thanked: 0
- Been thanked: 68 times
-
- Member
- Posts: 274
- Joined: Fri Mar 12, 2010 4:00 pm
- Division: Grad
- State: WA
- Has thanked: 0
- Been thanked: 0
Re: Analysis of our recent downtime:
This was just someone pointed out to me last night - the only ones I'm aware of at the moment are for experimental design: the Conestoga 2012 and Wright State Invitationalforeverphysics wrote:Which ones?
(State, Nationals)
2013: Astro (2, 6) / Chem (2, 5) / Circuits (8, 36) / Diseases (1,1) / Fermi (N/A, 24) / Materials (1, N/A)
2012 : Astro (1, 11) / Chem (N/A, 13) / Diseases (3, 1) / Optics (2, 3) / Sounds (2, 1)
2011: Astro(2,11) / Diseases (1,27) / Optics (1,13) / Proteins (2,15)
2013: Astro (2, 6) / Chem (2, 5) / Circuits (8, 36) / Diseases (1,1) / Fermi (N/A, 24) / Materials (1, N/A)
2012 : Astro (1, 11) / Chem (N/A, 13) / Diseases (3, 1) / Optics (2, 3) / Sounds (2, 1)
2011: Astro(2,11) / Diseases (1,27) / Optics (1,13) / Proteins (2,15)