|
2011-10-24
§
|
| 14:30 |
<Jeff_Green> |
installed nagios/nsca on spence |
[production] |
| 14:27 |
<mutante> |
snapshot4 - dist-upgrade/kernel, reboot |
[production] |
| 14:19 |
<mark> |
Manually fixed package situation on formey |
[production] |
| 14:10 |
<mutante> |
ekrem - gzipped access.log.1, can move/delete access.logs older than X months? |
[production] |
| 14:08 |
<mutante> |
ekrem at 99% disk space ..again |
[production] |
| 13:14 |
<mutante> |
something fails btw with etherpad using openoffice ([soffice] <defunct>), (was before and after reboot) |
[production] |
| 13:10 |
<mutante> |
hooper - fsck was forced and OK, back up, blog and etherpad reachable again |
[production] |
| 13:06 |
<mutante> |
hooper - dist-upgrade/kernel, reboot |
[production] |
| 13:03 |
<mutante> |
hooper - temp. shutting down blog and etherpad for kernel update/reboot |
[production] |
| 11:34 |
<mutante> |
added TERM=dumb to sudo line in /home/w/bin/scap to prevent "unknown terminal type" errors (RT 1767) |
[production] |
| 10:52 |
<mutante> |
gurvin - dist-upgrade/kernel,reboot (is 'False' in pybal https conf) |
[production] |
| 10:52 |
<mutante> |
fixing date on yvon |
[production] |
| 10:43 |
<mutante> |
yvon - dist-upgrade/kernel,reboot (is 'False' in pybal https conf) |
[production] |
| 04:22 |
<tstarling> |
synchronizing Wikimedia installation... : [[rev:100578|r100578]] |
[production] |
| 03:48 |
<Tim> |
gave the whole Engineering group write access to the ops-requests queue |
[production] |
| 03:44 |
<Tim> |
added myself to the operations group in RT so that I could resolve RT #1789 |
[production] |
| 02:53 |
<Tim> |
re-enabled cron job |
[production] |
| 02:26 |
<tstarling> |
synchronized php-1.18/extensions/CheckUser/maintenance/purgeOldData.php '[[rev:100574|r100574]]' |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.18) at Mon Oct 24 02:23:42 UTC 2011 |
[production] |
| 02:09 |
<aaron> |
synchronized /php-1.18/resources/jquery.ui/themes/vector/images 'deployed [[rev:100571|r100571]]' |
[production] |
| 02:00 |
<Tim> |
disabled CheckUser purge cron job due to excessive replication lag |
[production] |
| 01:40 |
<Tim> |
on hume: installed CheckUser purging cron job |
[production] |
| 01:36 |
<Tim> |
made a shell script to run extensions/CheckUser/maintenance/purgeOldData.php on all wikis, for use in a cron job. Running it on hume to test it. |
[production] |
|
2011-10-22
§
|
| 17:45 |
<mark> |
storage1 rebooted, but didn't come back (probably iptables issue) |
[production] |
| 17:21 |
<mark> |
Moved storage1 to private subnet in SVN, and then reverted that change again |
[production] |
| 17:10 |
<mark> |
Running dist-upgrade on storage1 |
[production] |
| 17:03 |
<ben> |
synchronized wmf-config/db.php 'putting ms2 back in rotation after its reboot and RAID recovery' |
[production] |
| 16:41 |
<RobH> |
removed myself from nagios contacts and restarted nagios service, feels awesome. |
[production] |
| 16:34 |
<maplebed> |
copied mysql-at-facebook to ms2; it's now running that instead of the packaged version |
[production] |
| 16:19 |
<mark> |
ms2 back up |
[production] |
| 16:18 |
<mark> |
disk ata28 broken on ms2 |
[production] |
| 14:56 |
<maplebed> |
physically powercycling ms2; it's stuck at grub and the mgmt interface is unavailable. |
[production] |
| 04:02 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 31862 - New logo for Tamil Wikibooks' |
[production] |
| 03:55 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 31857 - Narayam for Odia Wiktionary' |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.18) at Sat Oct 22 02:24:14 UTC 2011 |
[production] |
| 02:15 |
<aaron> |
synchronized php-1.18/extensions/CheckUser 'pushed [[rev:100488|r100488]] code' |
[production] |
| 01:22 |
<ben> |
synchronized wmf-config/db.php 'taking ms2 out of rotation for a reboot - filesystem errors and an xfs kernel oops' |
[production] |
|
2011-10-21
§
|
| 22:11 |
<reedy> |
synchronized php-1.18/extensions/WikiLove/WikiLove.hooks.php '[[rev:100464|r100464]]' |
[production] |
| 20:14 |
<demon> |
synchronized php-1.18/extensions/Contest/includes/ContestContestant.php 'Push [[rev:100446|r100446]]. Shame on Reedy for deploying then going afk' |
[production] |
| 20:08 |
<Jeff_Green> |
taking payments4 out of pybal pool |
[production] |
| 19:36 |
<reedy> |
synchronizing Wikimedia installation... : Pushing [[rev:100440|r100440]] for contest |
[production] |
| 18:30 |
<mutante> |
powercycling snapshot3 - mgmt console just showed numbers !? |
[production] |
| 18:20 |
<mutante> |
snapshot3 - dist-upgrade/kernel,about to reboot... |
[production] |
| 18:06 |
<mutante> |
rebooting hume after dist-upgrade |
[production] |
| 15:58 |
<mutante> |
dataset2 - installed security upgrades/kernel, scheduled for reboot RT 1780 |
[production] |
| 15:44 |
<mutante> |
stafford - dist-upgrade/kernel,reboot |
[production] |
| 14:27 |
<mutante> |
hume - installed security updates but ask aaron for reboot (long running fixBug28348.php related process?) |
[production] |
| 14:06 |
<mutante> |
< Jeff_Green> alert: fundraising test is beginning |
[production] |