1901-1950 of 10000 results (77ms)
2019-09-04 §
07:21 <mutante> ununpentium - a2dismod ssl - systemctl restart apache2 [production]
05:53 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
02:46 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.21/resources/src/startup/mediawiki.js: 8a1b13026 (duration: 00m 55s) [production]
02:42 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.21/resources/src/mediawiki.base/mediawiki.base.js: 8a1b13026 (duration: 00m 56s) [production]
02:21 <chaomodus> extending downtime on netmon1002 and netmon2001, netbox1001, netbox2001, netboxdb1001 and netbox2001 should be stable but are still being debugged [production]
01:02 <krinkle@deploy1001> Synchronized wmf-config/CommonSettings.php: ed5297c10 / T217830 (duration: 00m 59s) [production]
00:02 <chaomodus> installing and setting up netbox instances T223291 [production]
2019-09-03 §
23:57 <niharika29@deploy1001> Synchronized wmf-config/CommonSettings.php: Revert - [bugfix]Growth experiments not loading conf properly T231935 (duration: 00m 55s) [production]
23:56 <niharika29@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Revert - [bugfix]Growth experiments not loading conf properly T231935 (duration: 00m 55s) [production]
23:53 <niharika29@deploy1001> Synchronized php-1.34.0-wmf.21/extensions/GrowthExperiments/: Set correct merge strategy for help panel links T231935 (duration: 00m 55s) [production]
23:52 <niharika29@deploy1001> Synchronized php-1.34.0-wmf.20/extensions/GrowthExperiments/: Set correct merge strategy for help panel links T231935 (duration: 00m 56s) [production]
23:42 <niharika29@deploy1001> Synchronized php-1.34.0-wmf.20/tests/phpunit/: Allow CompositeBlock::appliesToRight to return null when unsure T229417, T231145 (duration: 00m 57s) [production]
23:41 <niharika29@deploy1001> Synchronized php-1.34.0-wmf.20/includes/block: Allow CompositeBlock::appliesToRight to return null when unsure T229417, T231145 (duration: 00m 55s) [production]
23:28 <niharika29@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable and configure ORES damaging and goodfaith on zhwiki T225562 (duration: 00m 58s) [production]
23:10 <ebernhardson> production-search-eqiad all indices index.merge.policy.deletes_pct_allowed=20 [production]
22:54 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: T208694 Set CentralNotice's wgNoticeProjects for wikimedia (duration: 00m 59s) [production]
22:45 <eileen> process-control config revision is 100334de4a adjust silverpop schedule [production]
19:42 <XioNoX> rollback OSPF metric change on eqiad-codfw Zayo link (1320->320) [production]
19:20 <fdans@deploy1001> Started restart [analytics/aqs/deploy@fc1d232]: (no justification provided) [production]
19:18 <fdans@deploy1001> Started restart [analytics/aqs/deploy@fc1d232]: (no justification provided) [production]
19:14 <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch high-traffic jobs to eventgate. Take 2 - T228705 (duration: 00m 56s) [production]
19:12 <ottomata> switching jobqueue events to eventgate-main - T228705 [production]
18:41 <urbanecm@deploy1001> Synchronized wmf-config/: Emergency fix: GE not loading configuration properly: newbie facing feature (duration: 00m 57s) [production]
18:35 <Urbanecm> Livetesting on mwdebug1002 [production]
17:45 <James_F> Pulled I9b64a2bb770 into wmf.21 production on the deploy server; no need to deploy to app-servers, CI-only fix. [production]
17:40 <hashar@deploy1001> rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.21 [production]
16:35 <catrope@deploy1001> Synchronized php-1.34.0-wmf.21/extensions/Graph/includes/ApiGraph.php: T231894 (duration: 00m 55s) [production]
16:01 <joal@deploy1001> Finished deploy [analytics/refinery@8b17711]: Fixes for regualr analytics deploy (duration: 136m 59s) [production]
15:55 <reedy@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T227260 (duration: 00m 54s) [production]
15:32 <ebernhardson> unban elastic1027 from production-search-eqiad [production]
15:07 <hashar@deploy1001> rebuilt and synchronized wikiversions files: testwiki 1.34.0-wmf.21 for T231894 - T220746 [production]
14:57 <hashar@deploy1001> rebuilt and synchronized wikiversions files: Rollback group0 to 1.34.0-wmf.21 - T220746 [production]
14:45 <hashar@deploy1001> rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.21 - T220746 [production]
14:39 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Promote db1133 as wikitech master T229657 (duration: 00m 54s) [production]
14:28 <hashar@deploy1001> Finished scap: testwiki to 1.34.0-wmf.21 and rebuild l10n cache - T220746 (duration: 50m 09s) [production]
14:21 <moritzm> upgrading app server canaries to PHP 7.2.22 T230024 [production]
13:44 <joal@deploy1001> Started deploy [analytics/refinery@8b17711]: Fixes for regualr analytics deploy [production]
13:38 <hashar@deploy1001> Started scap: testwiki to 1.34.0-wmf.21 and rebuild l10n cache - T220746 [production]
13:26 <hashar> Gerrit should be fine again, apparently was due to the wmf branch cut taking too much resources (sic) - T231872 filled to investigate [production]
13:25 <hashar> 1.34.0-wmf.21 cut [production]
13:16 <hashar> Gerrit has some random times out from time to time (no reason) [production]
13:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1073 from wikitech T229657', diff saved to https://phabricator.wikimedia.org/P9038 and previous config saved to /var/cache/conftool/dbconfig/20190903-131456-marostegui.json [production]
13:13 <marostegui> Re-enable puppet on db1073 and db1133 T229657 [production]
13:11 <marostegui> Reload haproxy on dbproxy1005 T229657 [production]
13:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Set wikitech back to RW after maintenance T229657', diff saved to https://phabricator.wikimedia.org/P9037 and previous config saved to /var/cache/conftool/dbconfig/20190903-131000-marostegui.json [production]
13:09 <moritzm> upgrading remaining mwdebug servers to PHP 7.2.22 T230024 [production]
13:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1133 to wikitech master T229657', diff saved to https://phabricator.wikimedia.org/P9036 and previous config saved to /var/cache/conftool/dbconfig/20190903-130937-marostegui.json [production]
13:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Set wikitech as read-only for maintenance T229657', diff saved to https://phabricator.wikimedia.org/P9035 and previous config saved to /var/cache/conftool/dbconfig/20190903-130839-marostegui.json [production]
13:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Set wikitech as read-only for maintenance T229657', diff saved to https://phabricator.wikimedia.org/P9033 and previous config saved to /var/cache/conftool/dbconfig/20190903-130113-marostegui.json [production]