News

The database is still purging, at a rate of about 400,000 workunits per day

April 23, 2009

The database is still purging, at a rate of about 400,000 workunits per day. We expect that this process will be completed by about 06:00 UTC tomorrow. At that time, we will do some additional database maintenance (table optimization) and then restart the project. So in total about another 24 hours will still be needed to bring everything back to normal. Thanks again for your patience during this unfortunate and long outage.

We think we have found a simple fix for the database problems

April 22, 2009

We think we have found a simple fix for the database problems. The database has grown to 45 GB in size and has gotten too large for the physical memory of the machine that hosts it. However it turns out that due to some mistakes made in the project operations during the past weeks, about 80% of the work in the database is already completed. So we are running a db_purge task tonight that should remove this already-completed work from the database, leaving only the work-in-progress still in place in the database.

The daemons worked through the backlog last night, but turning on the scheduler this morning for a few hours brought the DB down again. Some of the problems have been identified and fixed or worked around, but we are still working on some issues

April 21, 2009

The daemons worked through the backlog last night, but turning on the scheduler this morning for a few hours brought the DB down again. Some of the problems have been identified and fixed or worked around, but we are still working on some issues. The project has been shut down again for some DB analyis runs, it will probably be up and down again a number of times today as we fix the remaining issues one by one.