1401-1450 of 10000 results (30ms)
2020-07-22 §
08:25 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1084 and db1107', diff saved to https://phabricator.wikimedia.org/P12012 and previous config saved to /var/cache/conftool/dbconfig/20200722-082309-marostegui.json [production]
08:22 <kartik@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
08:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1126', diff saved to https://phabricator.wikimedia.org/P12010 and previous config saved to /var/cache/conftool/dbconfig/20200722-082023-marostegui.json [production]
08:19 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
08:16 <akosiaris> increase codfw mobileapps kubernetes traffic to 96% T218733. Take #2. Let's see if I can reproduce the weird increases in p99 latencies and figure out their cause [production]
08:15 <akosiaris@cumin1001> conftool action : set/weight=1; selector: dc=codfw,service=mobileapps,name=scb.* [production]
08:14 <kormat@cumin1001> dbctl commit (dc=all): 'Increase es1020 to 75% pooled in es4, reduce es1021 to weight 25 T257284', diff saved to https://phabricator.wikimedia.org/P12009 and previous config saved to /var/cache/conftool/dbconfig/20200722-081457-kormat.json [production]
08:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1084 and db1107', diff saved to https://phabricator.wikimedia.org/P12008 and previous config saved to /var/cache/conftool/dbconfig/20200722-081330-marostegui.json [production]
08:12 <moritzm> Turnilo switched to CAS [production]
08:05 <jayme> updated docker-report to 0.0.6-1 on deneb [production]
07:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1084 and db1107', diff saved to https://phabricator.wikimedia.org/P12007 and previous config saved to /var/cache/conftool/dbconfig/20200722-075749-marostegui.json [production]
07:53 <kormat@cumin1001> dbctl commit (dc=all): 'Increase es1020 to 50% pooled in es4 T257284', diff saved to https://phabricator.wikimedia.org/P12006 and previous config saved to /var/cache/conftool/dbconfig/20200722-075312-kormat.json [production]
07:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1084 to s1, depooled T253217', diff saved to https://phabricator.wikimedia.org/P12005 and previous config saved to /var/cache/conftool/dbconfig/20200722-075040-marostegui.json [production]
07:49 <jayme> import docker-report 0.0.6-1 to buster-wikimedia [production]
07:40 <jynus> stop db1145 for hw maintenance T258249 [production]
06:47 <elukey> update analytics-in4/6 filters on cr1/cr2 eqiad (ref https://gerrit.wikimedia.org/r/c/operations/homer/public/+/614702) [production]
06:26 <marostegui> Stop MySQL on db1107 [production]
06:11 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:09 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1107 to clone db1084', diff saved to https://phabricator.wikimedia.org/P12003 and previous config saved to /var/cache/conftool/dbconfig/20200722-060432-marostegui.json [production]
05:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1126', diff saved to https://phabricator.wikimedia.org/P12002 and previous config saved to /var/cache/conftool/dbconfig/20200722-051607-marostegui.json [production]
2020-07-21 §
23:37 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Bump cirrus MLR models to latest (duration: 01m 06s) [production]
23:13 <Urbanecm> Evening backport window done [production]
23:12 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: 7a50168d54b5e86834606fb8d7880eb3a923ffd5: Updating UploadWizard template: PD-old-70-1923->PD-old-70-expired (T258523) (duration: 01m 06s) [production]
23:06 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 7acc9d966a07d589bb6aed5f801c9e1defc75fe1: Enable $wgWatchlistExpiry on testwiki (T257506) (duration: 01m 08s) [production]
19:10 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.1 [production]
19:02 <catrope@deploy1001> Synchronized php-1.36.0-wmf.1/includes/Storage/PageUpdater.php: Fix handling of null edits (T257766) (duration: 01m 06s) [production]
19:01 <catrope@deploy1001> Synchronized php-1.35.0-wmf.41/includes/Storage/PageUpdater.php: Fix handling of null edits (T257766) (duration: 01m 11s) [production]
18:33 <jhuneidi@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.1 (duration: 41m 22s) [production]
18:27 <ejegg> restored new URL for TY page in payments-wiki settings [production]
18:22 <mforns@deploy1001> Finished deploy [analytics/refinery@0c25de1] (thin): Redeploying to unbreak unique devices per domain monthly THIN [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] (duration: 00m 07s) [production]
18:22 <mforns@deploy1001> Started deploy [analytics/refinery@0c25de1] (thin): Redeploying to unbreak unique devices per domain monthly THIN [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] [production]
18:21 <mforns@deploy1001> Finished deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly - third try [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] (duration: 00m 12s) [production]
18:21 <mforns@deploy1001> Started deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly - third try [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] [production]
18:17 <mforns@deploy1001> Finished deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly - second try [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] (duration: 00m 17s) [production]
18:16 <mforns@deploy1001> Started deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly - second try [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] [production]
18:13 <mforns@deploy1001> Finished deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] (duration: 05m 32s) [production]
18:08 <mforns@deploy1001> Started deploy [analytics/refinery@0c25de1]: Redeploying to unbreak unique devices per domain monthly [analytics/refinery@0c25de19a3a309276654b4463cca4f574336d8fd] [production]
17:52 <jhuneidi@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.1 [production]
17:50 <volans@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
17:45 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
17:10 <jhuneidi@deploy1001> Pruned MediaWiki: 1.35.0-wmf.39 (duration: 16m 25s) [production]
16:32 <ppchelko@deploy1001> Finished deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase, take 2 (duration: 04m 54s) [production]
16:27 <ppchelko@deploy1001> Started deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase, take 2 [production]
16:27 <ppchelko@deploy1001> Finished deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase (duration: 10m 37s) [production]
16:21 <longma> 1.36.0-wmf.1 was branched at 3a1faac3764ecae8dde813bd67a5a8e8f4975a85 for T257969 [production]
16:16 <ppchelko@deploy1001> Started deploy [restbase/deploy@4f3cb41]: Add new wikis to RESTBase [production]
15:16 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
15:12 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]