|
2012-03-13
§
|
| 16:23 |
<apergos> |
stole some free space from the phys volume on ms1002 to give us more time for the rsync to keep going til after the move to swift etc |
[production] |
| 15:28 |
<mark> |
Sending traffic from the USA to upload-lb.eqiad |
[production] |
| 15:27 |
<mark> |
Rebooting lvs1005 with upgraded kernel/packages |
[production] |
| 15:12 |
<LeslieCarr> |
manually deleted cp1025 info from nagios config file - nagios restored for now |
[production] |
| 14:51 |
<mark> |
Sending traffic from Canada to upload-lb.eqiad |
[production] |
| 14:33 |
<mark> |
Sending traffic from Brazil to upload-lb.eqiad |
[production] |
| 13:58 |
<mark> |
Sending traffic from Argentina to upload-lb.eqiad |
[production] |
| 12:58 |
<mark> |
Seeding the eqiad upload caches from live upload requests |
[production] |
| 11:59 |
<mark> |
Setup squid logging to oxygen, with oxygen relaying to multicast 233.58.59.1 |
[production] |
| 11:02 |
<mark> |
Rebooting lvs1002 with kernel updates |
[production] |
| 10:17 |
<mark> |
Rebooting manutius with newer 2.6.36 kernel to attempt avoiding i/o kernel bug with torrus |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.19) at Tue Mar 13 02:18:03 UTC 2012 |
[production] |
|
2012-03-12
§
|
| 22:55 |
<K4-713> |
synchronized payments cluster to [[rev:113679|r113679]], and tweaked the anti-fraud rules |
[production] |
| 21:51 |
<catrope> |
synchronized php-1.19/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js '[[rev:113671|r113671]]' |
[production] |
| 21:51 |
<catrope> |
synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php '[[rev:113671|r113671]]' |
[production] |
| 21:44 |
<Reedy> |
Running foreachwiki extensions/WikimediaMaintenance/cleanupBug31576.php in screen as me on hume |
[production] |
| 21:39 |
<RobH> |
search1014 repaired per rt 2483 |
[production] |
| 20:26 |
<RobH> |
cp1040 coming down for hardware stuffs |
[production] |
| 18:19 |
<Nikerabbit> |
Assuming scap has finished |
[production] |
| 17:48 |
<nikerabbit> |
synchronizing Wikimedia installation... : Deploying updated Translate |
[production] |
| 17:46 |
<notpeter> |
restarting indexer on searchidx2 |
[production] |
| 17:24 |
<nikerabbit> |
synchronized php-1.19/includes/Title.php '[[rev:113635|r113635]]' |
[production] |
| 17:22 |
<nikerabbit> |
synchronized php-1.19/languages/ '[[rev:113635|r113635]]' |
[production] |
| 17:14 |
<nikerabbit> |
synchronized php-1.19/extensions/Narayam/ 'Updating Narayam' |
[production] |
| 17:13 |
<mark> |
PXE booting cp1025-cp1028 |
[production] |
| 17:11 |
<nikerabbit> |
synchronized php-1.19/extensions/WebFonts/ 'Updating WebFonts' |
[production] |
| 15:16 |
<mark> |
Rebooted manutius, stuck in a similar state as streber always did |
[production] |
| 06:10 |
<mutante> |
turning off debug mode in nagios-nrpe, again had to kill it , restart fails |
[production] |
| 05:53 |
<mutante> |
dunno, copper was stuck (no mgmt output after reboot) but powercycling it and back |
[production] |
| 05:43 |
<mutante> |
rebooting copper to make sure grub update didnt break it and asked for restart anyways |
[production] |
| 05:37 |
<mutante> |
copper - installing (security) updates (apt,grub,openssl,ruby,libc6..) |
[production] |
| 03:40 |
<mutante> |
same (and nscd) on fenari |
[production] |
| 03:35 |
<mutante> |
upgrading libc6 and related packages on spence |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.19) at Mon Mar 12 02:17:28 UTC 2012 |
[production] |
|
2012-03-10
§
|
| 22:09 |
<Reedy> |
Make that wikimania2012, not wikimediawiki |
[production] |
| 22:08 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Disable anon page creation for wikimediawiki' |
[production] |
| 19:28 |
<binasher> |
set sync_binlog = 1 on all current masters and eqiad dbs |
[production] |
| 19:22 |
<binasher> |
reslaved db1033 |
[production] |
| 07:03 |
<mutante> |
ran puppet on db1022, another one that works fine manually but somehow did not by itself |
[production] |
| 05:11 |
<mutante> |
doing more (cp*, db*, msbe-* ,mw*) by hand / for loop |
[production] |
| 05:01 |
<mutante> |
starting nagios-nrpe-server on all via dsh (fail to restart on config change issue) |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.19) at Sat Mar 10 02:16:57 UTC 2012 |
[production] |
| 01:07 |
<maplebed> |
started swiftcleaner on owa1 looking for (and purging) bad objects |
[production] |
| 01:06 |
<maplebed> |
rebalanced the swift rings to finish decreasing traffic sent to ms1 and ms2 |
[production] |
| 00:18 |
<Ryan_Lane> |
powercycling ssl1003 |
[production] |