|
2009-02-25
§
|
| 18:58 |
<mark> |
Installed ms3 on 2 of the 4 bootable disk drives (sda and sdi), on a 80 GB RAID-1 root partition, leaving the rest of those 250 GB drives free, as well as the other 46 drives |
[production] |
| 18:24 |
<Rob> |
srv215 deployed into service |
[production] |
| 18:16 |
<Rob> |
installed and setting up srv215 |
[production] |
| 18:08 |
<brion> |
merging r47809 live -- XSS fix for Collection extension |
[production] |
| 18:01 |
<brion> |
temporarily halting apache on srv78, which seems to have some borkage with nfs mounts and nis |
[production] |
| 17:55 |
<brion> |
srv78: \tls: cannot access /mnt/upload5/wikipedia: No such file or directory |
[production] |
| 17:53 |
<Rob> |
rename auth2 to sockpuppet in dns, pushed changes |
[production] |
| 16:55 |
<Rob> |
fixed IP allocation for ms4 and ms3 |
[production] |
| 16:55 |
<Rob> |
reset ms4 by accident due to IP misassignment on service processors for ms4 and ms3 |
[production] |
| 05:29 |
<Tim> |
updated ApiQueryLogEvents.php to r47781 |
[production] |
| 04:21 |
<Tim> |
killed long-running ApiQueryLogEvents queries on db4 (12000 seconds) |
[production] |
| 01:11 |
<Andrew> |
Deployed AbuseFilter on dewiki per request of DaB |
[production] |
| 01:08 |
<Andrew> |
adding AbuseFilter tables for dewiki |
[production] |
| 00:03 |
<brion> |
running mw-serve cache cleaning on pdf1; cron job was borked by missing log file |
[production] |
|
2009-02-24
§
|
| 22:46 |
<Andrew> |
Deployed AbuseFilter on metawiki. |
[production] |
| 22:43 |
<Andrew> |
Adding abuse filter tables to metawiki. |
[production] |
| 22:30 |
<brion> |
syncing merge of r47769 to fix regression in page history |
[production] |
| 22:22 |
<brion> |
fix to disable slow tag search has had a side effect breaking page history. Should be tidied up in a few minutes. |
[production] |
| 22:15 |
<brion> |
merging r47767 live to disable $wgUseTagFilter |
[production] |
| 20:37 |
<Rob> |
updating dns for wikimedia.cz |
[production] |
| 20:32 |
<brion> |
turning Collection/PDF on for all wikisources |
[production] |
| 19:47 |
<mark> |
Shutting down Apache and unmounting /home on srv32, for puppet testing |
[production] |
| 19:03 |
<Rob> |
updated wikibooks logo stuff per bug # 17034 |
[production] |
| 17:13 |
<brion> |
just noting that TeliaSonera has a planned maintenance which may affect KNAMS connectivity for a few minutes around 2009-02-25 05:00 UTC. |
[production] |
| 17:12 |
<brion> |
just noting that TeliaSonera has a planned maintenance which may affect KNAMS connectivity for a few minutes around 2009-02-25 05:00 UTC. |
[production] |
| 16:51 |
<brion> |
restarting some dump threads on srv31 |
[production] |
| 01:55 |
<river> |
ms1 is now replicating to ms4 instead of ms2 |
[production] |
| 00:06 |
<Rob> |
reverted wikimedia.cz and .cs dns redirects for danny |
[production] |
|
2009-02-23
§
|
| 22:56 |
<RobH_> |
srv136 reinstalled and redeployed as apache |
[production] |
| 22:33 |
<mark> |
Installed puppet (test install) and thereby automatically gmond as aggregator on srv33 |
[production] |
| 21:16 |
<domas> |
adler has disk with media errors (ID:5, 6th disk in array): http://p.defau.lt/?3_7_6aIatj3DeNBw_jjtBg - needs cannibalized samuel, disk replacement, and ubuntu install on raid10 |
[production] |
| 19:04 |
<Rob> |
srv136 back from repairs, reinstalling as apache server |
[production] |
| 18:44 |
<Rob> |
srv217 not running apache, synced and restarted |
[production] |
| 18:29 |
<Rob> |
srv33 reinstalled to ubuntu and deployed as apache server |
[production] |
| 18:24 |
<Rob> |
srv32 reinstalled to ubuntu and deployed as apache server |
[production] |
| 17:55 |
<Rob> |
reinstalling srv32 to ubuntu |
[production] |
| 17:38 |
<Rob> |
resynced and restarted apache on srv32, srv33, srv34 |
[production] |
| 17:32 |
<Rob> |
srv31 powered back up |
[production] |
| 17:25 |
<Rob> |
found a breaker flip in the DC, affects srv31-srv34 |
[production] |
| 13:40 |
<domas> |
oh, btw folks, kudos on perfect web2.0 engineering, now morebots complains when message is longer than 140 bytes, and we end up without our microblogging syndication |
[production] |
| 13:39 |
<domas> |
added "su -m 'www-data' -c 'find /opt/mwlib/var/cache/ -mindepth 3 -mtime +1 -delete'" to pdf1 crontab, does anyone actually look after this service? |
[production] |
| 12:57 |
<Tim> |
deployed r47704, now command line scripts don't access /home anymore |
[production] |
| 11:37 |
<Tim> |
switched archive directory over to /mnt/upload5, starting another rsync. Some files will be missing until the rsync is done |
[production] |
| 10:07 |
<Tim> |
moved all job runners from the previous ad hoc script to the new wikimedia-job-runner package |
[production] |
| 06:25 |
<Tim> |
moved the nagios plugins for fedora from /home/nagios to /h/w/common/nagios-fedora-plugins |
[production] |
| 05:21 |
<Tim> |
started udp2log on db20, MW UDP logs were dead |
[production] |
| 05:19 |
<Tim> |
killed errant jobs loop scripts still running on fedora servers |
[production] |
| 04:36 |
<Tim> |
fixed the log directory for /etc/cron.d/mw-central-notice, killed the process that was in a tight loop trying to write to a stale NFS file handle |
[production] |
| 04:28 |
<Tim> |
finished moving ExtensionDistributor working copy |
[production] |
| 04:14 |
<Tim> |
moving ExtensionDistributor working directory from /home to /mnt/upload5 |
[production] |