|
2015-12-21
§
|
| 15:44 |
<godog> |
adding new 1TB disk to restbase1007 |
[production] |
| 14:22 |
<andrewbogott> |
disabling puppet on labnet1002 for dnsmasq tests |
[production] |
| 14:07 |
<MaxSem> |
me and yurik are nuking old maps data and reimporting planet |
[production] |
| 13:46 |
<jynus> |
extending online s2-master data disk by +100GB |
[production] |
| 13:15 |
<akosiaris> |
disabled puppet on maps-test2001 and commented out osmupdater crontab entry until we fix the sync process |
[production] |
| 11:02 |
<jynus> |
emergency restart of db1047's mysql |
[production] |
| 09:54 |
<jynus> |
reenabling semisync replication on s3 |
[production] |
| 09:07 |
<godog> |
stop cassandra on restbase1004, decomissioned |
[production] |
| 02:29 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Dec 21 02:29:51 UTC 2015 (duration 6m 47s) |
[production] |
| 02:23 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.9) (duration: 09m 45s) |
[production] |
| 02:20 |
<andrewbogott> |
disabling puppet on labnet1002 to mess with dnsmasq |
[production] |
| 01:44 |
<andrewbogott> |
disabled puppet on holmium and labservices1001 to control roll-out of https://gerrit.wikimedia.org/r/#/c/260037/ |
[production] |
|
2015-12-20
§
|
| 23:24 |
<Reedy> |
Katie and Jeff paged about bellatrix |
[production] |
| 18:46 |
<andrewbogott> |
graceful restart of zuul as per https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart |
[production] |
| 18:31 |
<andrewbogott> |
restarting stuck Jenkins |
[production] |
| 17:47 |
<reedy@tin> |
Purged l10n cache for 1.27.0-wmf.6 |
[production] |
| 17:11 |
<godog> |
depool mw1228, reported ro fs |
[production] |
| 15:53 |
<reedy@tin> |
Synchronized README: noop (duration: 00m 32s) |
[production] |
| 15:50 |
<Reedy> |
reedy@tin Purged l10n cache for 1.27.0-wmf.6 (hanging due to mw1228 issue) |
[production] |
| 15:42 |
<Reedy> |
mw1228 reporting readonly fs |
[production] |
| 15:41 |
<reedy@tin> |
Purged l10n cache for 1.27.0-wmf.7 |
[production] |
| 09:00 |
<godog> |
powercycle ms-be2019, xfs lockup |
[production] |
| 02:28 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Dec 20 02:28:49 UTC 2015 (duration 6m 54s) |
[production] |
| 02:21 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.9) (duration: 08m 59s) |
[production] |
|
2015-12-19
§
|
| 21:55 |
<_joe_> |
restarted zotero on sca1001, various OOM messages |
[production] |
| 20:48 |
<gwicke> |
restbase1004: `systemctl mask cassandra` in preparation for the decommission finishing |
[production] |
| 19:49 |
<akosiaris> |
killed gmond on db2036. it was clearly misbehaving and running since Jan 02. db2036 was not listed on the ganglia web interface. killing the orphaned process and restarting seems to have fixed it |
[production] |
| 18:54 |
<akosiaris> |
scheduled maintenance of s3 slave lag on db2036, db2043, db2050, db2057 (all of db2018's family that pages) to effectively silence pages while debugging. Check is flapping since 15:00 UTC today |
[production] |
| 15:14 |
<krenair@tin> |
Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/259611/ - noop for prod, other than making icinga stop complaining (duration: 00m 31s) |
[production] |
| 10:07 |
<hashar> |
CI jobs for MediaWiki were broken because of cssjanus dependency. Should be fixed once mw/core https://gerrit.wikimedia.org/r/#/c/260169/ lands |
[production] |
| 02:28 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Dec 19 02:28:56 UTC 2015 (duration 6m 53s) |
[production] |
| 02:22 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.9) (duration: 08m 53s) |
[production] |
| 01:01 |
<gwicke> |
entire restbase cluster: removed 5% root reserve from data partition with tune2fs -m 0 /dev/mapper/restbase$NODE--vg-{srv,var} |
[production] |
| 00:49 |
<gwicke> |
restbase1008: removed 5% root reserve from data partition with tune2fs -m 0 /dev/mapper/restbase1008--vg-srv |
[production] |
|
2015-12-18
§
|
| 22:57 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.9/resources/src/mediawiki/mediawiki.searchSuggest.js: allow override of suggestion type reported in event loggin (duration: 00m 29s) |
[production] |
| 22:56 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.suggest.js: override suggestion type reported in event logging (duration: 00m 30s) |
[production] |
| 22:50 |
<aaron@tin> |
Synchronized php-1.27.0-wmf.9/includes/jobqueue/aggregator/JobQueueAggregatorRedis.php: 2c942ba1782c42ee68622278a5e0a77e9027945d (duration: 00m 30s) |
[production] |
| 22:30 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.suggest.js: override suggestion type reported in event logging (duration: 00m 30s) |
[production] |
| 22:20 |
<aaron@tin> |
Synchronized php-1.27.0-wmf.9/includes/jobqueue/aggregator/JobQueueAggregator.php: 2c942ba1782c42ee68622278a5e0a77e9027945d (duration: 00m 31s) |
[production] |
| 19:26 |
<aaron@tin> |
Synchronized wmf-config/jobqueue-eqiad.php: Adjust queue "maxPartitionsTry" and timeouts (duration: 00m 30s) |
[production] |
| 18:49 |
<mutante> |
disregard that, apache config only is enough |
[production] |
| 18:47 |
<mutante> |
gerrit will restart in a moment and be right back |
[production] |
| 18:44 |
<ori> |
ditto |
[production] |
| 18:43 |
<Krinkle> |
Created account "Krinkle" on collabwiki |
[production] |
| 16:28 |
<twentyafterfour> |
restarted apache on iridium to deploy redirect script changes |
[production] |
| 16:20 |
<jynus> |
restarting and reconfiguring mysql on db1047 |
[production] |
| 14:57 |
<godog> |
stop compactions on restbase1008 |
[production] |
| 14:55 |
<jynus> |
SET GLOBAL query_cache_type = 0; on db1025 |
[production] |
| 14:54 |
<hashar> |
gallium: restarted apache2 , was deadlocked/unresponsive somehow |
[production] |
| 14:44 |
<godog> |
update privatesettings with swift codfw configuration |
[production] |