|
2015-12-20
§
|
| 23:24 |
<Reedy> |
Katie and Jeff paged about bellatrix |
[production] |
| 18:46 |
<andrewbogott> |
graceful restart of zuul as per https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart |
[production] |
| 18:31 |
<andrewbogott> |
restarting stuck Jenkins |
[production] |
| 17:47 |
<reedy@tin> |
Purged l10n cache for 1.27.0-wmf.6 |
[production] |
| 17:11 |
<godog> |
depool mw1228, reported ro fs |
[production] |
| 15:53 |
<reedy@tin> |
Synchronized README: noop (duration: 00m 32s) |
[production] |
| 15:50 |
<Reedy> |
reedy@tin Purged l10n cache for 1.27.0-wmf.6 (hanging due to mw1228 issue) |
[production] |
| 15:42 |
<Reedy> |
mw1228 reporting readonly fs |
[production] |
| 15:41 |
<reedy@tin> |
Purged l10n cache for 1.27.0-wmf.7 |
[production] |
| 09:00 |
<godog> |
powercycle ms-be2019, xfs lockup |
[production] |
| 02:28 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Dec 20 02:28:49 UTC 2015 (duration 6m 54s) |
[production] |
| 02:21 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.9) (duration: 08m 59s) |
[production] |
|
2015-12-19
§
|
| 21:55 |
<_joe_> |
restarted zotero on sca1001, various OOM messages |
[production] |
| 20:48 |
<gwicke> |
restbase1004: `systemctl mask cassandra` in preparation for the decommission finishing |
[production] |
| 19:49 |
<akosiaris> |
killed gmond on db2036. it was clearly misbehaving and running since Jan 02. db2036 was not listed on the ganglia web interface. killing the orphaned process and restarting seems to have fixed it |
[production] |
| 18:54 |
<akosiaris> |
scheduled maintenance of s3 slave lag on db2036, db2043, db2050, db2057 (all of db2018's family that pages) to effectively silence pages while debugging. Check is flapping since 15:00 UTC today |
[production] |
| 15:14 |
<krenair@tin> |
Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/259611/ - noop for prod, other than making icinga stop complaining (duration: 00m 31s) |
[production] |
| 10:07 |
<hashar> |
CI jobs for MediaWiki were broken because of cssjanus dependency. Should be fixed once mw/core https://gerrit.wikimedia.org/r/#/c/260169/ lands |
[production] |
| 02:28 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Dec 19 02:28:56 UTC 2015 (duration 6m 53s) |
[production] |
| 02:22 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.9) (duration: 08m 53s) |
[production] |
| 01:01 |
<gwicke> |
entire restbase cluster: removed 5% root reserve from data partition with tune2fs -m 0 /dev/mapper/restbase$NODE--vg-{srv,var} |
[production] |
| 00:49 |
<gwicke> |
restbase1008: removed 5% root reserve from data partition with tune2fs -m 0 /dev/mapper/restbase1008--vg-srv |
[production] |
|
2015-12-18
§
|
| 22:57 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.9/resources/src/mediawiki/mediawiki.searchSuggest.js: allow override of suggestion type reported in event loggin (duration: 00m 29s) |
[production] |
| 22:56 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.suggest.js: override suggestion type reported in event logging (duration: 00m 30s) |
[production] |
| 22:50 |
<aaron@tin> |
Synchronized php-1.27.0-wmf.9/includes/jobqueue/aggregator/JobQueueAggregatorRedis.php: 2c942ba1782c42ee68622278a5e0a77e9027945d (duration: 00m 30s) |
[production] |
| 22:30 |
<ebernhardson@tin> |
Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.suggest.js: override suggestion type reported in event logging (duration: 00m 30s) |
[production] |
| 22:20 |
<aaron@tin> |
Synchronized php-1.27.0-wmf.9/includes/jobqueue/aggregator/JobQueueAggregator.php: 2c942ba1782c42ee68622278a5e0a77e9027945d (duration: 00m 31s) |
[production] |
| 19:26 |
<aaron@tin> |
Synchronized wmf-config/jobqueue-eqiad.php: Adjust queue "maxPartitionsTry" and timeouts (duration: 00m 30s) |
[production] |
| 18:49 |
<mutante> |
disregard that, apache config only is enough |
[production] |
| 18:47 |
<mutante> |
gerrit will restart in a moment and be right back |
[production] |
| 18:44 |
<ori> |
ditto |
[production] |
| 18:43 |
<Krinkle> |
Created account "Krinkle" on collabwiki |
[production] |
| 16:28 |
<twentyafterfour> |
restarted apache on iridium to deploy redirect script changes |
[production] |
| 16:20 |
<jynus> |
restarting and reconfiguring mysql on db1047 |
[production] |
| 14:57 |
<godog> |
stop compactions on restbase1008 |
[production] |
| 14:55 |
<jynus> |
SET GLOBAL query_cache_type = 0; on db1025 |
[production] |
| 14:54 |
<hashar> |
gallium: restarted apache2 , was deadlocked/unresponsive somehow |
[production] |
| 14:44 |
<godog> |
update privatesettings with swift codfw configuration |
[production] |
| 14:43 |
<godog> |
set temp-url-key for mw:media account in swift codfw |
[production] |
| 12:19 |
<paravoid> |
upgrading tor on radium, rebooting for kernel upgrade |
[production] |
| 12:18 |
<_joe_> |
disabled puppet on all lvs hosts for a potentially harmful change (should be a noop) |
[production] |
| 11:47 |
<_joe_> |
restarted hhvm on mw1107, stuck at startup |
[production] |
| 11:40 |
<hashar> |
logstash: reorganized list of dashboards per sections https://logstash.wikimedia.org/#/dashboard/elasticsearch/default |
[production] |
| 09:43 |
<akosiaris> |
rebooting planet1001, memory exhaustion, OOM showed up |
[production] |
| 09:20 |
<hashar> |
Killed Zuul entirely, the queues were full / deadlocked. Patches need to be retriggered |
[production] |
| 06:47 |
<gwicke> |
restbase1004: nodetool stop -- COMPACTION to avoid running out of disk space |
[production] |
| 03:07 |
<ori@tin> |
Synchronized php-1.27.0-wmf.9/includes/api/ApiStashEdit.php: ab32f4e740: Make ApiStashEdit use statsd metrics (duration: 00m 49s) |
[production] |
| 02:29 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Dec 18 02:29:10 UTC 2015 (duration 6m 55s) |
[production] |
| 02:22 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.9) (duration: 08m 45s) |
[production] |
| 01:52 |
<ori> |
re-enabled puppet on rdb* / mc* |
[production] |