601-650 of 10000 results (57ms)
2020-03-25 ยง
12:56 <marostegui> Deploy schema change on db1139:3316 [production]
12:45 <marostegui> Stop MySQL on db2115 for reimage to buster [production]
11:50 <cdanis> cr1-codfw: `set chassis fpc 5 inline-services flex-flow-sizing` and `request chassis fpc restart slot 5` T248394 [production]
11:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2115 for upgrade', diff saved to https://phabricator.wikimedia.org/P10763 and previous config saved to /var/cache/conftool/dbconfig/20200325-114655-marostegui.json [production]
11:39 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
11:37 <mutante> decom mw1250 - mw1253 [production]
11:37 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
11:35 <cdanis> depool codfw for router maintenance T248394 [production]
11:33 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
11:32 <mutante> decom mw1232 - mw1235 [production]
11:31 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
11:27 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw125[0-3].eqiad.wmnet [production]
11:26 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw123[2-5].eqiad.wmnet [production]
11:22 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:22 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:21 <Urbanecm> EU SWAT done [production]
11:21 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:21 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:20 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw123[2-5].eqiad.wmnet [production]
11:20 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw125[0-3].eqiad.wmnet [production]
11:19 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: SWAT: 59412db: Add gwtoolset to available rights to allow granting to global groups (duration: 01m 07s) [production]
11:12 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: SWAT: 7b8d7c5: TwoColConflict: Limited default deployment CommonSettings.php (T244863) (duration: 01m 06s) [production]
11:10 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 81cda0f: TwoColConflict: Limited default deployment InitialiseSettings.php (T244863; take II) (duration: 01m 06s) [production]
11:08 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 81cda0f: TwoColConflict: Limited default deployment InitialiseSettings.php (T244863) (duration: 01m 17s) [production]
11:08 <jynus@cumin1001> dbctl commit (dc=all): 'Reduce db1091 load, increase main traffic on all other s4 instances', diff saved to https://phabricator.wikimedia.org/P10762 and previous config saved to /var/cache/conftool/dbconfig/20200325-110821-jynus.json [production]
10:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1137', diff saved to https://phabricator.wikimedia.org/P10761 and previous config saved to /var/cache/conftool/dbconfig/20200325-105503-marostegui.json [production]
10:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1137', diff saved to https://phabricator.wikimedia.org/P10760 and previous config saved to /var/cache/conftool/dbconfig/20200325-103938-marostegui.json [production]
10:37 <XioNoX> change aggregate policy for 2620:0:862::/48 on cr3-knams - T236785 [production]
10:19 <XioNoX> change aggregate policy for v4 prefixes on cr2-eqdfw - T236785 [production]
10:04 <oblivian@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-main' for release 'canary' . [production]
10:04 <oblivian@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-main' for release 'production' . [production]
09:56 <XioNoX> change aggregate policy for 2620:0:860::/46 on cr2-eqdfw - T236785 [production]
09:54 <vgutierrez> Enable inbound TLSv1.3 on upload@eqsin - T170567 [production]
09:27 <jmm@cumin2001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
09:23 <vgutierrez> upgrade ATS to 8.0.6-1wm3 on upload@eqsin - T170567 [production]
09:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1137', diff saved to https://phabricator.wikimedia.org/P10759 and previous config saved to /var/cache/conftool/dbconfig/20200325-091421-marostegui.json [production]
09:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1137', diff saved to https://phabricator.wikimedia.org/P10758 and previous config saved to /var/cache/conftool/dbconfig/20200325-090227-marostegui.json [production]
08:55 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:53 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:38 <marostegui> Reimage db1137 [production]
08:18 <marostegui> Reboot db1117 for full-upgrade [production]
08:15 <oblivian@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-main' for release 'canary' . [production]
08:15 <oblivian@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-main' for release 'production' . [production]
08:14 <_joe_> upgrading all eventgate-main to envoy 1.13.1 T246868 [production]
08:12 <marostegui> Stop all mysql daemons on db1117 [production]
07:50 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-main' for release 'canary' . [production]
07:50 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-main' for release 'production' . [production]
07:42 <XioNoX> reboot scs-eqsin for CPU usage [production]
07:20 <jmm@cumin2001> START - Cookbook sre.ganeti.makevm [production]
07:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1137 for upgrade', diff saved to https://phabricator.wikimedia.org/P10757 and previous config saved to /var/cache/conftool/dbconfig/20200325-070946-marostegui.json [production]