2020-07-14
ยง
|
08:30 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
08:30 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
08:30 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
08:30 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
08:30 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
08:30 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
08:13 |
<jforrester@deploy1001> |
Started scap: Re-re-start full scap to push out wmf.41 and switch testwikis to it T256669 |
[production] |
08:05 |
<akosiaris> |
restart pybal on lvs2009 |
[production] |
08:03 |
<_joe_> |
restart pybal on lvs1016 |
[production] |
08:02 |
<akosiaris> |
restart pybal on lvs2007 |
[production] |
08:01 |
<akosiaris@cumin1001> |
conftool action : set/pooled=inactive; selector: name=restbase2009.codfw.wmnet |
[production] |
08:00 |
<_joe_> |
restart pybal on lvs1015 |
[production] |
08:00 |
<akosiaris> |
restart pybal on lvs2010 after merging https://gerrit.wikimedia.org/r/612487 |
[production] |
07:52 |
<jforrester@deploy1001> |
sync aborted: Re-start full scap to push out wmf.41 and switch testwikis to it T256669 (duration: 02m 14s) |
[production] |
07:50 |
<jforrester@deploy1001> |
Started scap: Re-start full scap to push out wmf.41 and switch testwikis to it T256669 |
[production] |
07:48 |
<oblivian@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: revert forcehttps in an attempt to fix T257887 (duration: 01m 06s) |
[production] |
07:32 |
<oblivian@deploy1001> |
sync-file aborted: revert forcehttps in an attempt to fix T257887 (duration: 00m 20s) |
[production] |
07:31 |
<oblivian@deploy1001> |
Scap failed!: 7/9 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
07:27 |
<moritzm> |
installing libtasn1-6 security updates |
[production] |
07:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1075', diff saved to https://phabricator.wikimedia.org/P11894 and previous config saved to /var/cache/conftool/dbconfig/20200714-071233-marostegui.json |
[production] |
07:04 |
<marostegui> |
Drop gerrit, gerritro, gerrittest users from m2 databases - T255715 |
[production] |
06:58 |
<marostegui> |
Stop mysql on db1131 for HW maintenance |
[production] |
06:56 |
<oblivian@deploy2001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
06:54 |
<jforrester@deploy1001> |
scap failed: RuntimeError Scap failed!: 9/9 canaries failed their endpoint checks(http://en.wikipedia.org) (duration: 24m 59s) |
[production] |
06:54 |
<jforrester@deploy1001> |
Scap failed!: 9/9 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
06:53 |
<oblivian@deploy2001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
06:53 |
<marostegui> |
Deploy MCR schema change on s5 primary master T238966 |
[production] |
06:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1078', diff saved to https://phabricator.wikimedia.org/P11893 and previous config saved to /var/cache/conftool/dbconfig/20200714-065229-marostegui.json |
[production] |
06:29 |
<jforrester@deploy1001> |
Started scap: testwikis wikis to 1.35.0-wmf.41 |
[production] |
05:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Decrease a bit db1088 load', diff saved to https://phabricator.wikimedia.org/P11891 and previous config saved to /var/cache/conftool/dbconfig/20200714-051551-marostegui.json |
[production] |
05:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1131 for HW maintenance', diff saved to https://phabricator.wikimedia.org/P11890 and previous config saved to /var/cache/conftool/dbconfig/20200714-050931-marostegui.json |
[production] |
05:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1093 from api', diff saved to https://phabricator.wikimedia.org/P11889 and previous config saved to /var/cache/conftool/dbconfig/20200714-050912-marostegui.json |
[production] |
05:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1093 to s6 master and remove read-only from s6 T257253', diff saved to https://phabricator.wikimedia.org/P11888 and previous config saved to /var/cache/conftool/dbconfig/20200714-050157-marostegui.json |
[production] |
05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s6 as read-only for maintenance T257253', diff saved to https://phabricator.wikimedia.org/P11887 and previous config saved to /var/cache/conftool/dbconfig/20200714-050039-marostegui.json |
[production] |
05:00 |
<marostegui> |
Starting s6 failover from db1131 to db1093 - T257253 |
[production] |
04:59 |
<James_F> |
1.35.0-wmf.41 branched at 7d04152db4f8ea9a459511bed8117101d9bb4602 |
[production] |
04:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1078', diff saved to https://phabricator.wikimedia.org/P11886 and previous config saved to /var/cache/conftool/dbconfig/20200714-043907-marostegui.json |
[production] |
04:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1093 in preparation for failover', diff saved to https://phabricator.wikimedia.org/P11885 and previous config saved to /var/cache/conftool/dbconfig/20200714-041548-marostegui.json |
[production] |
04:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1130', diff saved to https://phabricator.wikimedia.org/P11884 and previous config saved to /var/cache/conftool/dbconfig/20200714-041440-marostegui.json |
[production] |
01:23 |
<ryankemper> |
Started long-running Elasticsearch reindex of `eqiad`, `codfw`, and `cloudelastic`. tmux session `reindex` under `ryankemper` on `mwmaint1002` |
[production] |
01:20 |
<cdanis> |
โcdanis@lvs1015.eqiad.wmnet ~ ๐ค๐บ sudo systemctl restart pybal.service |
[production] |
01:15 |
<cdanis> |
โ๏ธ cdanis@lvs1016.eqiad.wmnet ~ ๐๐บ sudo systemctl restart pybal.service |
[production] |
01:14 |
<cdanis> |
โ๏ธ cdanis@lvs2009.codfw.wmnet ~ ๐๐บ sudo systemctl restart pybal.service |
[production] |
01:01 |
<cdanis> |
โ๏ธ cdanis@lvs2010.codfw.wmnet ~ ๐๐บ sudo systemctl restart pybal.service |
[production] |