|
2009-11-09
§
|
| 23:15 |
<andrew> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling GlobalUsage due to major scaling issues, discussion on private-l' |
[production] |
| 22:56 |
<tstarling> |
synchronized docroot/secure/keys.html |
[production] |
| 22:55 |
<tstarling> |
synchronized docroot/secure/keys.txt |
[production] |
| 20:57 |
<Rob> |
wordpress upgraded for techblog. by upgraded we mean new install and manual copy of existing uplaoded files and customizations. |
[production] |
| 20:51 |
<Rob> |
techblog accessibility still spotty. Wordpress upgrades are sometimes painful. |
[production] |
| 20:12 |
<Rob> |
messing with updating wordpress for techblog. It failed before so techblog will be up and down during the next few minutes |
[production] |
| 20:11 |
<Fred> |
raised the number of apache threads on Singer to accommodate secure traffic. |
[production] |
| 19:29 |
<mark> |
Disconnected drbd resource nfshome on nfs2: RAID1 resync was still running, waiting for it to finish before attempting DRBD resync |
[production] |
| 19:15 |
<tfinc> |
synchronized php-1.5/extensions/ContributionReporting/ContributionHistory_body.php |
[production] |
| 18:59 |
<mark> |
Stopped NFS kernel server on db20 (yes, that's /home) |
[production] |
| 18:34 |
<mark> |
Turned off swap on db20: block device needed for DRBD external metadata |
[production] |
| 17:04 |
<mark> |
OS-installed nfs2 |
[production] |
| 16:40 |
<mark> |
Running restart of puppetd across the cluster |
[production] |
| 16:35 |
<mark> |
Running backup of db20 /home |
[production] |
| 16:28 |
<mark> |
Running dist-upgrade on locke |
[production] |
| 16:00 |
<mark> |
fsck repaired /var/backup on mchenry, filesystem remounted |
[production] |
| 15:55 |
<mark> |
Added tridge to the ALL node group |
[production] |
| 15:49 |
<mark> |
dist-upgrade and reboot of tridge |
[production] |
| 15:42 |
<mark> |
Running new fsck on mchenry while the system is up |
[production] |
| 15:41 |
<mark> |
/var/backups filesystem on mchenry is corrupt |
[production] |
| 15:22 |
<mark> |
Reset DRAC password of sanger |
[production] |
| 15:18 |
<mark> |
Rebooting mchenry |
[production] |
| 15:16 |
<mark> |
Running apt-get dist-upgrade on mchenry |
[production] |
| 15:06 |
<mark> |
Rebooting sanger |
[production] |
| 15:04 |
<mark> |
Running apt-get dist-upgrade on sanger |
[production] |
| 14:41 |
<mark> |
Shutdown bart for decommissioning |
[production] |
| 14:36 |
<mark> |
Deployed cron job to restart puppet daily |
[production] |
| 14:35 |
<mark> |
Deployed automatic updating of GeoIP database across the cluster |
[production] |
| 08:14 |
<brion> |
killed duplicate wikibugs irc bot connection, stopping double bug reporting in #mediawiki |
[production] |
| 03:17 |
<Tim> |
updating SecurePoll to r58802 |
[production] |
|
2009-11-08
§
|
| 22:10 |
<mark> |
Added the esams ganglia clusters to the Florida grid (why the hierarchy?) |
[production] |
| 21:52 |
<domas> |
dropped one snapshot on ixia, other still up. |
[production] |
| 21:52 |
<Andrew> |
domas says he did some magic with i/o on ixia, replag back to zero. |
[production] |
| 21:51 |
<andrew> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning GlobalUsage back on' |
[production] |
| 21:41 |
<mark> |
Experimentally installed unattended-upgrades on server sockpuppet |
[production] |
| 21:41 |
<mark> |
Reboot srv156 |
[production] |
| 21:38 |
<andrew> |
synchronized php-1.5/wmf-config/InitialiseSettings.php 'Deactivating GlobalUsage to check impact on ixia replag.' |
[production] |
| 21:24 |
<Andrew> |
ixia still has huge replication lag, refreshGlobalImageLinks definitely not still running. |
[production] |
| 18:31 |
<mark> |
Unmounted /home on thistle, ixia, lomaria, db1 and db11 |
[production] |
| 17:26 |
<aaron> |
synchronized php-1.5/extensions/FlaggedRevs/specialpages/Stabilization_body.php 'deployed r58786' |
[production] |
| 16:51 |
<brion> |
taking load off ixia to let it catch up; it's still lagged on GlobalUsage inserts |
[production] |
| 16:51 |
<brion> |
synchronized php-1.5/wmf-config/db.php |
[production] |
| 16:09 |
<Bryan> |
Correction: --start-image=Symbol_neutral_vote.svg --start-page=15088846 --maxlag=5 |
[production] |
| 16:04 |
<Bryan> |
Parameters to restart refreshGlobalimagelinks.php with on enwiki: --start-image Symbol_neutral_vote.svg --start-page=15088846 --maxlag 5 |
[production] |
| 16:02 |
<brion> |
stopped andrew's refreshGlobalimagelinks.php job on hume pending fixes and someone to babysit it |
[production] |
| 15:58 |
<brion> |
ixia lag from GlobalUsage population script; Bryan poking at it to make it behave nicer |
[production] |
| 15:55 |
<brion> |
commonswiki | Connect | 510 | NULL | INSERT /* GlobalUsage::setUsage */ INTO `globalimagelinks` (gil_wiki,gil_page,gil_page_namespace,g | |
[production] |
| 15:50 |
<brion> |
circa 8 minutes replag on ixia :( |
[production] |