|
2015-09-25
§
|
| 12:54 |
<andrewbogott> |
restarted rabbitmq-server on labcontrol1001 |
[production] |
| 12:52 |
<godog> |
stop puppet on restbase in production -- config deployment |
[production] |
| 12:46 |
<hashar> |
stopping nodepool to clear out left over mysql connections |
[production] |
| 12:21 |
<hashar> |
Nodepool is dead, or at least not adding new slaves anymore |
[production] |
| 12:16 |
<hashar> |
Seems Zuul/Jenkins is in trouble somehow :-/ |
[production] |
| 11:42 |
<andrewbogott> |
upgrading linux-image-generic on labnet1002 to get us away from the recently-crashed 3.13.0-59 |
[production] |
| 04:58 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: I252970886: Set "async" for SQL parser cache everywhere else (duration: 00m 18s) |
[production] |
| 02:57 |
<Krinkle> |
sync-common failed on mw1010.eqiad.wmnet |
[production] |
| 02:53 |
<krinkle@tin> |
Synchronized php-1.26wmf24/extensions/DonationInterface: 381faf5 (duration: 00m 21s) |
[production] |
| 02:33 |
<l10nupdate@tin> |
Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 11m 45s) |
[production] |
| 01:44 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki roa_tarawiki |
[production] |
| 01:42 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki itwikiquote |
[production] |
| 01:42 |
<Krinkle> |
mwscript deleteEqualMessages.php --wiki alswiki |
[production] |
| 01:28 |
<krinkle@tin> |
Synchronized php-1.26wmf24/extensions/WikimediaEvents: I5608f8ffd1c - Fix trailing comma (duration: 00m 17s) |
[production] |
| 00:10 |
<krinkle@tin> |
Synchronized php-1.26wmf24/extensions/EventLogging/modules/ext.eventLogging.core.js: Increase maxUrlSize to 2000 (duration: 00m 17s) |
[production] |
|
2015-09-24
§
|
| 23:02 |
<krenair@tin> |
Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/240880/ (duration: 00m 17s) |
[production] |
| 19:57 |
<ori@tin> |
Synchronized php-1.26wmf24/extensions/ContentTranslation: d079d5dd71: Updated mediawiki/core Project: mediawiki/extensions/ContentTranslation 8559ee614975f25b71a732ca0fb1bb6d489c9d33 (duration: 00m 18s) |
[production] |
| 19:35 |
<bblack> |
depooled cp1046 from confd, committed pybal depool for LVS as well |
[production] |
| 19:34 |
<chasemp> |
changing labs route on cr1 and cr2 from 10.68.16.0/22 to 10.68.16.0/21 which matches references, fw setting and manifests/network.pp |
[production] |
| 18:54 |
<catrope@tin> |
Synchronized php-1.26wmf24/extensions/Flow/: Debugging for FlowFixLinks.php (duration: 00m 20s) |
[production] |
| 18:21 |
<twentyafterfour@tin> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf24 |
[production] |
| 18:20 |
<legoktm> |
moved oauthadmin group from User:Yuvipanda@metawiki to User:YuviPanda@metawiki |
[production] |
| 18:19 |
<godog> |
restart restbase on restbase1005 |
[production] |
| 18:18 |
<godog> |
restart restbase on restbase1004 |
[production] |
| 18:16 |
<godog> |
restart restbase on restbase1003 |
[production] |
| 18:06 |
<paravoid> |
depooling cp1046, stability issues |
[production] |
| 18:00 |
<demon@tin> |
Synchronized multiversion/MWRealm.php: (no message) (duration: 00m 17s) |
[production] |
| 17:59 |
<ori> |
Merged Apache config change Ia095457fb. It will refresh the Apache service as it rolls out, causing elevated 503s for the next 20 minutes. |
[production] |
| 17:53 |
<godog> |
rolling restart restbase in eqiad |
[production] |
| 17:35 |
<chasemp> |
powercycling cp1046 at mgmt as I can't ssh in and it seems like it should be up |
[production] |
| 17:26 |
<godog> |
bounce restbase on restbase1002, apply new datacenter config |
[production] |
| 17:10 |
<_joe_> |
cleaning up /tmp on mw1152 |
[production] |
| 17:09 |
<cmjohnson1> |
powering down for the last time es1001 - es1010 |
[production] |
| 16:17 |
<thcipriani@tin> |
Synchronized php-1.26wmf23/extensions/Wikidata: SWAT: Do not filter affected pages by namespace [[gerrit:240727]] (duration: 00m 26s) |
[production] |
| 16:01 |
<robh> |
nothing on puppet swat window, easiest swat ever. |
[production] |
| 15:46 |
<thcipriani@tin> |
Synchronized php-1.26wmf24/extensions/Wikidata: SWAT: Do not filter affected pages by namespace [[gerrit:240711]] (duration: 00m 26s) |
[production] |
| 15:23 |
<thcipriani@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable suggestions in ca, en, es, fr, it, ja, tr, ru, zh [[gerrit:240638]] (duration: 00m 17s) |
[production] |
| 14:37 |
<paravoid> |
repooling codfw |
[production] |
| 12:54 |
<bblack> |
restarting varnish daemons on second half of maps, parsoid, misc clusters (package upgrade, shm_reclen change) |
[production] |
| 12:50 |
<bblack> |
restarting varnishd instances on text, mobile, upload clusters for package upgrade (slow salt, no parallelism, ~5m spacing - FE cache loss, BE cache stays, should take ~9h) |
[production] |
| 12:05 |
<moritzm> |
installed rpcbind security updates on eeden, baham, radon, maerlant, rhenium |
[production] |
| 11:56 |
<bblack> |
restarting varnish daemons on half of maps, parsoid, misc clusters (package upgrade, shm_reclen change) |
[production] |
| 11:36 |
<bblack> |
reinstall lvs300[12] to jessie - T96375 |
[production] |
| 11:21 |
<akosiaris> |
killed tail -f varnishncsa.log on cp1065 and ran apt-get clean to reclaim some disk space |
[production] |
| 11:14 |
<bblack> |
stopping pybal on lvs300[12]; lvs300[34] taking over |
[production] |
| 11:07 |
<bblack> |
upgrading varnishes to 3.0.6plus-wm8 (non-restarting, just pkg update on-disk) |
[production] |
| 09:40 |
<jynus> |
performing latest (software) steps to decom es1001-es1010 (puppet disabling, etc.) |
[production] |
| 08:39 |
<jynus> |
restarted HHVM @ mw1056, 1104, 1122 |
[production] |
| 05:33 |
<yuvipanda> |
deleted logstash indexes for 08/27 and 28 too |
[production] |
| 05:31 |
<yuvipanda> |
deleted indexes for 08/14, 15, 25, 26 on logstash |
[production] |