401-450 of 10000 results (59ms)
2020-03-25 ยง
18:05 <ppchelko@deploy1001> Started deploy [changeprop/deploy@4bdf55b]: Stop rerendering experimental PCS endpoints [production]
17:43 <mvolz@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'citoid' for release 'production' . [production]
17:38 <mvolz@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'citoid' for release 'production' . [production]
17:33 <mvolz@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'citoid' for release 'staging' . [production]
16:50 <moritzm> installing python-bleach security updates [production]
16:47 <moritzm> updated jenkins packages on apt.wikimedia.org to 2.222.1 [production]
16:33 <rzl@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'blubberoid' for release 'production' . [production]
16:32 <sukhe> upload cescout 0.1.0-1 to apt.wm.o (buster) - T247273 [production]
16:17 <rzl@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'blubberoid' for release 'production' . [production]
16:15 <rzl@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
16:07 <rlazarus> updating blubberoid to envoy 1.13.1 T246868 [production]
15:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2115 after reimage to Buster', diff saved to https://phabricator.wikimedia.org/P10767 and previous config saved to /var/cache/conftool/dbconfig/20200325-152148-marostegui.json [production]
15:14 <moritzm> installing deneb.codfw.wmnet T248165 [production]
14:51 <cdanis> repool codfw T248394 [production]
14:46 <mutante> closed port 80 for caching servers on misc backends https://gerrit.wikimedia.org/r/q/topic:%22applayer-tls%22+(status:open%20OR%20status:merged) as final step per service on T210411 [production]
14:39 <mutante> static microsites (annual.wikimedia.org, research.wikimedia.org, static-bugzilla etc). closed port 80 for caching servers, finalizing switch to https behind caching servers [production]
14:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:53 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
13:48 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
13:26 <_joe_> cumin A:puppetmaster 'apt-get -y install puppet-common' [production]
13:03 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
13:02 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:56 <marostegui> Deploy schema change on db1139:3316 [production]
12:45 <marostegui> Stop MySQL on db2115 for reimage to buster [production]
11:50 <cdanis> cr1-codfw: `set chassis fpc 5 inline-services flex-flow-sizing` and `request chassis fpc restart slot 5` T248394 [production]
11:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2115 for upgrade', diff saved to https://phabricator.wikimedia.org/P10763 and previous config saved to /var/cache/conftool/dbconfig/20200325-114655-marostegui.json [production]
11:39 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
11:37 <mutante> decom mw1250 - mw1253 [production]
11:37 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
11:35 <cdanis> depool codfw for router maintenance T248394 [production]
11:33 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
11:32 <mutante> decom mw1232 - mw1235 [production]
11:31 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
11:27 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw125[0-3].eqiad.wmnet [production]
11:26 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw123[2-5].eqiad.wmnet [production]
11:22 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:22 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:21 <Urbanecm> EU SWAT done [production]
11:21 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:21 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:20 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw123[2-5].eqiad.wmnet [production]
11:20 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw125[0-3].eqiad.wmnet [production]
11:19 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: SWAT: 59412db: Add gwtoolset to available rights to allow granting to global groups (duration: 01m 07s) [production]
11:12 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: SWAT: 7b8d7c5: TwoColConflict: Limited default deployment CommonSettings.php (T244863) (duration: 01m 06s) [production]
11:10 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 81cda0f: TwoColConflict: Limited default deployment InitialiseSettings.php (T244863; take II) (duration: 01m 06s) [production]
11:08 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 81cda0f: TwoColConflict: Limited default deployment InitialiseSettings.php (T244863) (duration: 01m 17s) [production]
11:08 <jynus@cumin1001> dbctl commit (dc=all): 'Reduce db1091 load, increase main traffic on all other s4 instances', diff saved to https://phabricator.wikimedia.org/P10762 and previous config saved to /var/cache/conftool/dbconfig/20200325-110821-jynus.json [production]
10:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1137', diff saved to https://phabricator.wikimedia.org/P10761 and previous config saved to /var/cache/conftool/dbconfig/20200325-105503-marostegui.json [production]
10:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1137', diff saved to https://phabricator.wikimedia.org/P10760 and previous config saved to /var/cache/conftool/dbconfig/20200325-103938-marostegui.json [production]