2101-2150 of 10000 results (31ms)
2020-06-10 §
15:38 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . [production]
15:37 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
15:36 <ppchelko@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Send kafka purges everywhere, gerrit:603654 (duration: 01m 05s) [production]
15:35 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:32 <ema> remaining-cp (non-ulsfo): rolling ats-tls-restart to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604305/ T255015 [production]
15:29 <ppchelko@deploy1001> Synchronized wmf-config/CommonSettings.php: Make kafka purges config more robust, gerrit:603649, CS.php (duration: 01m 05s) [production]
15:27 <ppchelko@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Make kafka purges config more robust, gerrit:603649, IS.php (duration: 01m 08s) [production]
15:21 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
15:19 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:08 <godog> roll-restart prometheus k8s to enable thanos upload [production]
15:02 <ema> A:cp-ulsfo: rolling ats-tls-restart to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604305/ T255015 [production]
14:43 <ema> A:cp rolling systemctl restart trafficserver [production]
14:28 <ema> systemctl restart trafficserver for instances critical in icinga [production]
14:21 <ema> cp3056: ats-backend-restart [production]
14:09 <ema> A:cp rolling ats-be/ats-tls restarts to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604305/ T255015 [production]
14:08 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:06 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:02 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:59 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094 into s7', diff saved to https://phabricator.wikimedia.org/P11458 and previous config saved to /var/cache/conftool/dbconfig/20200610-135753-marostegui.json [production]
13:50 <ema> cp3050: ats-tls-restart to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604305/ T255015 [production]
13:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094 into s7', diff saved to https://phabricator.wikimedia.org/P11457 and previous config saved to /var/cache/conftool/dbconfig/20200610-135039-marostegui.json [production]
13:40 <ema> cp3050: ats-backend-restart to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/604305/ T255015 [production]
13:36 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=99) [production]
13:06 <liw@deploy1001> Synchronized php: group1 wikis to 1.35.0-wmf.36 (duration: 01m 04s) [production]
13:05 <liw@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.36 [production]
12:33 <ayounsi@cumin1001> START - Cookbook sre.network.prepare-upgrade [production]
12:32 <ayounsi@cumin1001> END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=99) [production]
12:32 <ayounsi@cumin1001> START - Cookbook sre.network.prepare-upgrade [production]
12:13 <akosiaris> pool thumbor2002, thumbor2001. T251570 [production]
12:12 <akosiaris@cumin1001> conftool action : set/pooled=yes; selector: name=thumbor2002.codfw.wmnet [production]
12:12 <akosiaris@cumin1001> conftool action : set/pooled=yes; selector: name=thumbor2001.codfw.wmnet [production]
11:50 <marostegui> Deploy schema change on commonswiki codfw T255003 [production]
11:41 <moritzm> upgrading remaining app servers in codfw to PHP 7.2.31 [production]
11:38 <marostegui> Deploy schema change on testcommonswiki T255003 [production]
11:37 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 52091b8: Grant cswiki accountcreators tboverride-account and override-antispoof (T254927) (duration: 01m 06s) [production]
11:13 <moritzm> upgrading remaining job runners in codfw to PHP 7.2.31 [production]
11:02 <marostegui> Stop MySQL on db1094 to clone db1127 [production]
11:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1094 moving to clone db1127 T253217', diff saved to https://phabricator.wikimedia.org/P11453 and previous config saved to /var/cache/conftool/dbconfig/20200610-110204-marostegui.json [production]
10:57 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:54 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1127 moving it to s7 T253217', diff saved to https://phabricator.wikimedia.org/P11452 and previous config saved to /var/cache/conftool/dbconfig/20200610-103742-marostegui.json [production]
10:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1103,db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11451 and previous config saved to /var/cache/conftool/dbconfig/20200610-102805-marostegui.json [production]
10:24 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T254036 Undeploy CollaborationKit: IV – Drop flag to load (duration: 01m 05s) [production]
10:23 <jayme> T254581 re-enabled puppet on all mw, api and jobrunner servers [production]
10:20 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: T254036 Undeploy CollaborationKit: III – Drop ability to load (duration: 01m 05s) [production]
10:16 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T254036 Undeploy CollaborationKit: II – Disable on Test Wikipedia (duration: 01m 37s) [production]
10:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1103,db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11450 and previous config saved to /var/cache/conftool/dbconfig/20200610-101407-marostegui.json [production]
10:12 <moritzm> upgrading remaining API servers in codfw to PHP 7.2.31 [production]
10:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1103,db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11449 and previous config saved to /var/cache/conftool/dbconfig/20200610-100834-marostegui.json [production]