1651-1700 of 10000 results (29ms)
2020-12-17 §
07:08 <elukey> update analytics-in4 filter on cr1/cr2-eqiad for https://gerrit.wikimedia.org/r/c/operations/homer/public/+/649706 [production]
07:08 <ryankemper> [wdqs] depooled `wdqs1013` while it catches up on lag [production]
07:06 <ryankemper> [wdqs deploy] Restarting `wdqs-categories` across all wdqs instances, one host at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'` [production]
07:05 <ryankemper> [wdqs deploy] Restarting `wdqs-categories` across all test instances: `sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories'` [production]
07:05 <ryankemper> [wdqs-deploy] Restarting `wdqs-updater` across all instances, 4 instances at a time: `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` [production]
07:04 <ryankemper@deploy1001> Finished deploy [wdqs/wdqs@90f9bdd]: 0.3.56 (duration: 10m 39s) [production]
06:54 <ryankemper> [wdqs deploy] Tests passing on canary instance `wdqs1003` following canary deploy, proceeding to rest of fleet [production]
06:53 <ryankemper@deploy1001> Started deploy [wdqs/wdqs@90f9bdd]: 0.3.56 [production]
06:53 <ryankemper> [wdqs deploy] All tests passing on canary instance `wdqs1003` prior to deploy [production]
06:52 <kart_> Updated cxserver to 2020-12-16-164911-production (T234220, T269437) [production]
06:52 <kart_> Updated cxserver to 2020-12-16-164911-production (T234220, T234220) [production]
06:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1013 for decommissioning T268436', diff saved to https://phabricator.wikimedia.org/P13562 and previous config saved to /var/cache/conftool/dbconfig/20201217-062249-marostegui.json [production]
06:22 <kartik@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
06:19 <kartik@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
06:17 <kartik@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
06:13 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
06:05 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
05:56 <marostegui> Stop mysql on db1106 to clone db1154 [production]
05:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1106 for cloning db1154:3311 T268742 ', diff saved to https://phabricator.wikimedia.org/P13560 and previous config saved to /var/cache/conftool/dbconfig/20201217-055556-marostegui.json [production]
01:35 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1019.eqiad.wmnet with reason: REIMAGE [production]
01:33 <andrew@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1019.eqiad.wmnet with reason: REIMAGE [production]
01:01 <twentyafterfour> preparing to update phabricator translations [production]
00:22 <mutante> running puppet on mw2266, mw2370, mw2354 [production]
2020-12-16 §
23:56 <bstorm> bootstrapped meta_p database for the new s7 replicas T269427 [production]
20:12 <marxarelli> group1 to 1.36.0-wmf.22 complete. no new errors or concerning rates (refs T267415) [production]
20:06 <dduvall@deploy1001> Synchronized php: group1 wikis to 1.36.0-wmf.22 (duration: 01m 01s) [production]
20:05 <legoktm> added myself to the ops LDAP group [production]
20:05 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.22 [production]
19:23 <dcausse> Morning backport window deploy done [production]
19:21 <dcausse@deploy1001> Synchronized php-1.36.0-wmf.22/extensions/WikimediaEvents/: T266027: Revert [cirrus] setup perfield builder A/B test on spaceless languages (duration: 01m 03s) [production]
19:18 <dcausse@deploy1001> Synchronized php-1.36.0-wmf.21/extensions/WikimediaEvents/: T266027: Revert [cirrus] setup perfield builder A/B test on spaceless languages (duration: 01m 03s) [production]
19:09 <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T266359: wgMinervaCountErrors config was removed (duration: 01m 03s) [production]
17:52 <effie> uploading python-thumbor-wikimedia_2.9-1 to stretch-wikimedia/component/thumbor [production]
16:40 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2019.codfw.wmnet with reason: REIMAGE [production]
16:38 <jiji@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mc2019.codfw.wmnet with reason: REIMAGE [production]
16:38 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1019.eqiad.wmnet with reason: REIMAGE [production]
16:36 <jiji@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on mc1019.eqiad.wmnet with reason: REIMAGE [production]
16:32 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'echostore' for release 'staging' . [production]
16:32 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'echostore' for release 'production' . [production]
16:32 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'sessionstore' for release 'staging' . [production]
16:32 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'sessionstore' for release 'production' . [production]
16:23 <gehel@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
16:03 <jiji@deploy1001> Synchronized wmf-config/ProductionServices.php: Swap Redis lock managers with upgraded ones - T265643 (duration: 01m 03s) [production]
15:55 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'mathoid' for release 'staging' . [production]
15:55 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'mathoid' for release 'production' . [production]
15:53 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
15:43 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
15:43 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
15:42 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
15:42 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]