301-350 of 10000 results (27ms)
2020-07-22 §
06:26 <marostegui> Stop MySQL on db1107 [production]
06:11 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:09 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1107 to clone db1084', diff saved to https://phabricator.wikimedia.org/P12003 and previous config saved to /var/cache/conftool/dbconfig/20200722-060432-marostegui.json [production]
05:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1126', diff saved to https://phabricator.wikimedia.org/P12002 and previous config saved to /var/cache/conftool/dbconfig/20200722-051607-marostegui.json [production]
2020-07-21 §
23:37 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Bump cirrus MLR models to latest (duration: 01m 06s) [production]
23:13 <Urbanecm> Evening backport window done [production]
23:12 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: 7a50168d54b5e86834606fb8d7880eb3a923ffd5: Updating UploadWizard template: PD-old-70-1923->PD-old-70-expired (T258523) (duration: 01m 06s) [production]
23:06 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 7acc9d966a07d589bb6aed5f801c9e1defc75fe1: Enable $wgWatchlistExpiry on testwiki (T257506) (duration: 01m 08s) [production]
19:10 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.1 [production]
19:02 <catrope@deploy1001> Synchronized php-1.36.0-wmf.1/includes/Storage/PageUpdater.php: Fix handling of null edits (T257766) (duration: 01m 06s) [production]
19:01 <catrope@deploy1001> Synchronized php-1.35.0-wmf.41/includes/Storage/PageUpdater.php: Fix handling of null edits (T257766) (duration: 01m 11s) [production]
18:33 <jhuneidi@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.1 (duration: 41m 22s) [production]
18:27 <ejegg> restored new URL for TY page in payments-wiki settings [production]
18:22 <mforns@deploy1001> Finished deploy [analytics/refinery@0c25de1] (thin): Redeploying to unbreak unique devices per domain monthly THIN [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] (duration: 00m 07s) [production]
18:22 <mforns@deploy1001> Started deploy [analytics/refinery@0c25de1] (thin): Redeploying to unbreak unique devices per domain monthly THIN [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] [production]
18:21 <mforns@deploy1001> Finished deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly - third try [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] (duration: 00m 12s) [production]
18:21 <mforns@deploy1001> Started deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly - third try [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] [production]
18:17 <mforns@deploy1001> Finished deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly - second try [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] (duration: 00m 17s) [production]
18:16 <mforns@deploy1001> Started deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly - second try [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] [production]
18:13 <mforns@deploy1001> Finished deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] (duration: 05m 32s) [production]
18:08 <mforns@deploy1001> Started deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] [production]
17:52 <jhuneidi@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.1 [production]
17:50 <volans@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
17:45 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
17:10 <jhuneidi@deploy1001> Pruned MediaWiki: 1.35.0-wmf.39 (duration: 16m 25s) [production]
16:32 <ppchelko@deploy1001> Finished deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase, take 2 (duration: 04m 54s) [production]
16:27 <ppchelko@deploy1001> Started deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase, take 2 [production]
16:27 <ppchelko@deploy1001> Finished deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase (duration: 10m 37s) [production]
16:21 <longma> 1.36.0-wmf.1 was branched at 3a1faac3764ecae8dde813bd67a5a8e8f4975a85 for T257969 [production]
16:16 <ppchelko@deploy1001> Started deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase [production]
15:16 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
15:12 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
15:10 <moritzm> draining restbase1027 for eventual reboot for kernel security update [production]
15:09 <godog> poweroff ms-be1024 for bbu replacement - T257949 [production]
15:08 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:08 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:04 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
15:01 <vgutierrez> show a synthetic warning for traffic using ECDHE-RSA-AES128-SHA - T258405 [production]
15:01 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
15:00 <moritzm> draining restbase1026 for eventual reboot for kernel security update [production]
14:57 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:57 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
14:56 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
14:52 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
14:51 <moritzm> draining restbase1025 for eventual reboot for kernel security update [production]
14:48 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
14:44 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
14:35 <akosiaris@cumin1001> conftool action : set/weight=10; selector: dc=codfw,service=mobileapps,name=scb.* [production]
14:35 <akosiaris> decrease codfw mobileapps kubernetes traffic to 72% T218733. Weird latency patterns exhibited when 92% was reached. See https://grafana.wikimedia.org/d/5CmeRcnMz/mobileapps?panelId=34&fullscreen&orgId=1&from=1595338489749&to=1595342071227&var-dc=codfw%20prometheus%2Fk8s&var-service=mobileapps&var-container_name=All [production]