|
2012-03-13
§
|
| 15:12 |
<LeslieCarr> |
manually deleted cp1025 info from nagios config file - nagios restored for now |
[production] |
| 14:51 |
<mark> |
Sending traffic from Canada to upload-lb.eqiad |
[production] |
| 14:33 |
<mark> |
Sending traffic from Brazil to upload-lb.eqiad |
[production] |
| 13:58 |
<mark> |
Sending traffic from Argentina to upload-lb.eqiad |
[production] |
| 12:58 |
<mark> |
Seeding the eqiad upload caches from live upload requests |
[production] |
| 11:59 |
<mark> |
Setup squid logging to oxygen, with oxygen relaying to multicast 233.58.59.1 |
[production] |
| 11:02 |
<mark> |
Rebooting lvs1002 with kernel updates |
[production] |
| 10:17 |
<mark> |
Rebooting manutius with newer 2.6.36 kernel to attempt avoiding i/o kernel bug with torrus |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.19) at Tue Mar 13 02:18:03 UTC 2012 |
[production] |
|
2012-03-12
§
|
| 22:55 |
<K4-713> |
synchronized payments cluster to [[rev:113679|r113679]], and tweaked the anti-fraud rules |
[production] |
| 21:51 |
<catrope> |
synchronized php-1.19/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js '[[rev:113671|r113671]]' |
[production] |
| 21:51 |
<catrope> |
synchronized php-1.19/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php '[[rev:113671|r113671]]' |
[production] |
| 21:44 |
<Reedy> |
Running foreachwiki extensions/WikimediaMaintenance/cleanupBug31576.php in screen as me on hume |
[production] |
| 21:39 |
<RobH> |
search1014 repaired per rt 2483 |
[production] |
| 20:26 |
<RobH> |
cp1040 coming down for hardware stuffs |
[production] |
| 18:19 |
<Nikerabbit> |
Assuming scap has finished |
[production] |
| 17:48 |
<nikerabbit> |
synchronizing Wikimedia installation... : Deploying updated Translate |
[production] |
| 17:46 |
<notpeter> |
restarting indexer on searchidx2 |
[production] |
| 17:24 |
<nikerabbit> |
synchronized php-1.19/includes/Title.php '[[rev:113635|r113635]]' |
[production] |
| 17:22 |
<nikerabbit> |
synchronized php-1.19/languages/ '[[rev:113635|r113635]]' |
[production] |
| 17:14 |
<nikerabbit> |
synchronized php-1.19/extensions/Narayam/ 'Updating Narayam' |
[production] |
| 17:13 |
<mark> |
PXE booting cp1025-cp1028 |
[production] |
| 17:11 |
<nikerabbit> |
synchronized php-1.19/extensions/WebFonts/ 'Updating WebFonts' |
[production] |
| 15:16 |
<mark> |
Rebooted manutius, stuck in a similar state as streber always did |
[production] |
| 06:10 |
<mutante> |
turning off debug mode in nagios-nrpe, again had to kill it , restart fails |
[production] |
| 05:53 |
<mutante> |
dunno, copper was stuck (no mgmt output after reboot) but powercycling it and back |
[production] |
| 05:43 |
<mutante> |
rebooting copper to make sure grub update didnt break it and asked for restart anyways |
[production] |
| 05:37 |
<mutante> |
copper - installing (security) updates (apt,grub,openssl,ruby,libc6..) |
[production] |
| 03:40 |
<mutante> |
same (and nscd) on fenari |
[production] |
| 03:35 |
<mutante> |
upgrading libc6 and related packages on spence |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.19) at Mon Mar 12 02:17:28 UTC 2012 |
[production] |
|
2012-03-11
§
|
| 22:14 |
<Damianz> |
Increased nofile on deployment-squid and added max_filedesc option to squid config. Also installed squidclient. |
[releng] |
| 08:14 |
<apergos> |
restarted lighttp on dataset2 |
[production] |
| 07:49 |
<apergos> |
removed current htcp log file, restarted purger, it seems to be logging normallynow |
[production] |
| 07:35 |
<apergos> |
current ls shows 17416851456 2012-03-11 07:34 HTCPpurger.log while current du -sh shows 175M for /var/log. Sparse file that gets rotated badly? lots of leading nulls (many gb worth), why? |
[production] |
| 07:33 |
<apergos> |
on ms1004 the HTCPpurger.log file after rotation was 17 gb, filling the disk. Removed it. |
[production] |
| 04:36 |
<Ryan_Lane> |
also deployment-web3 |
[releng] |
| 04:35 |
<Ryan_Lane> |
also deployment-web |
[releng] |
| 04:34 |
<Ryan_Lane> |
make that deployment-web5 |
[releng] |
| 04:34 |
<Ryan_Lane> |
rebooting deployment-web, it OOM'd |
[releng] |
| 02:17 |
<LocalisationUpdate> |
completed (1.19) at Sun Mar 11 02:17:35 UTC 2012 |
[production] |
|
2012-03-10
§
|
| 22:09 |
<Reedy> |
Make that wikimania2012, not wikimediawiki |
[production] |
| 22:08 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Disable anon page creation for wikimediawiki' |
[production] |
| 19:28 |
<binasher> |
set sync_binlog = 1 on all current masters and eqiad dbs |
[production] |
| 19:22 |
<binasher> |
reslaved db1033 |
[production] |
| 07:03 |
<mutante> |
ran puppet on db1022, another one that works fine manually but somehow did not by itself |
[production] |
| 05:11 |
<mutante> |
doing more (cp*, db*, msbe-* ,mw*) by hand / for loop |
[production] |
| 05:01 |
<mutante> |
starting nagios-nrpe-server on all via dsh (fail to restart on config change issue) |
[production] |
| 02:16 |
<LocalisationUpdate> |
completed (1.19) at Sat Mar 10 02:16:57 UTC 2012 |
[production] |
| 01:07 |
<maplebed> |
started swiftcleaner on owa1 looking for (and purging) bad objects |
[production] |