|
2020-07-14
ยง
|
| 11:25 |
<_joe_> |
restart pybal on lvs1015 T257887 |
[production] |
| 11:22 |
<_joe_> |
restart pybal on lvs1016 |
[production] |
| 11:15 |
<jayme@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 11:03 |
<jayme@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 10:59 |
<jayme@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 10:56 |
<volans@cumin1001> |
conftool action : set/pooled=inactive; selector: name=wtp2005.codfw.wmnet |
[production] |
| 10:52 |
<volans> |
powerdown wtp2005, hardware issue - T257903 |
[production] |
| 10:47 |
<volans@cumin1001> |
conftool action : set/pooled=no; selector: name=wtp2005.codfw.wmnet |
[production] |
| 10:45 |
<jiji@cumin1001> |
conftool action : set/pooled=no; selector: name=wtp2005.codfw.wmnet,service=parsoid-php |
[production] |
| 10:45 |
<jiji@cumin1001> |
conftool action : set/pooled=no; selector: name=wtp2005.codfw.wmnet,service=parsoid |
[production] |
| 10:45 |
<effie> |
depool wtp2005 |
[production] |
| 10:42 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:42 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:39 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.change-distro |
[production] |
| 10:39 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.stop-cluster (exit_code=0) |
[production] |
| 10:32 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.stop-cluster |
[production] |
| 10:18 |
<oblivian@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'termbox' for release 'production' . |
[production] |
| 10:14 |
<James_F> |
Running AbuseFilter's updateVarDumps for group1 T246539 |
[production] |
| 10:13 |
<oblivian@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'termbox' for release 'production' . |
[production] |
| 10:10 |
<oblivian@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'staging' . |
[production] |
| 10:10 |
<oblivian@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'test' . |
[production] |
| 09:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1112', diff saved to https://phabricator.wikimedia.org/P11897 and previous config saved to /var/cache/conftool/dbconfig/20200714-094449-marostegui.json |
[production] |
| 09:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1075', diff saved to https://phabricator.wikimedia.org/P11896 and previous config saved to /var/cache/conftool/dbconfig/20200714-094354-marostegui.json |
[production] |
| 09:39 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: ExtensionDistributor: Add REL1_35 as a candidate release (duration: 01m 06s) |
[production] |
| 09:05 |
<jforrester@deploy1001> |
Finished scap: Re-re-start full scap to push out wmf.41 and switch testwikis to it T256669 (duration: 51m 41s) |
[production] |
| 08:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1135 for PDU upgrade T257871', diff saved to https://phabricator.wikimedia.org/P11895 and previous config saved to /var/cache/conftool/dbconfig/20200714-084033-marostegui.json |
[production] |
| 08:30 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:30 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:30 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:30 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:30 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 08:30 |
<akosiaris@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:13 |
<jforrester@deploy1001> |
Started scap: Re-re-start full scap to push out wmf.41 and switch testwikis to it T256669 |
[production] |
| 08:05 |
<akosiaris> |
restart pybal on lvs2009 |
[production] |
| 08:03 |
<_joe_> |
restart pybal on lvs1016 |
[production] |
| 08:02 |
<akosiaris> |
restart pybal on lvs2007 |
[production] |
| 08:01 |
<akosiaris@cumin1001> |
conftool action : set/pooled=inactive; selector: name=restbase2009.codfw.wmnet |
[production] |
| 08:00 |
<_joe_> |
restart pybal on lvs1015 |
[production] |
| 08:00 |
<akosiaris> |
restart pybal on lvs2010 after merging https://gerrit.wikimedia.org/r/612487 |
[production] |
| 07:52 |
<jforrester@deploy1001> |
sync aborted: Re-start full scap to push out wmf.41 and switch testwikis to it T256669 (duration: 02m 14s) |
[production] |
| 07:50 |
<jforrester@deploy1001> |
Started scap: Re-start full scap to push out wmf.41 and switch testwikis to it T256669 |
[production] |
| 07:48 |
<oblivian@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: revert forcehttps in an attempt to fix T257887 (duration: 01m 06s) |
[production] |
| 07:32 |
<oblivian@deploy1001> |
sync-file aborted: revert forcehttps in an attempt to fix T257887 (duration: 00m 20s) |
[production] |
| 07:31 |
<oblivian@deploy1001> |
Scap failed!: 7/9 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 07:27 |
<moritzm> |
installing libtasn1-6 security updates |
[production] |
| 07:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1075', diff saved to https://phabricator.wikimedia.org/P11894 and previous config saved to /var/cache/conftool/dbconfig/20200714-071233-marostegui.json |
[production] |
| 07:04 |
<marostegui> |
Drop gerrit, gerritro, gerrittest users from m2 databases - T255715 |
[production] |
| 06:58 |
<marostegui> |
Stop mysql on db1131 for HW maintenance |
[production] |
| 06:56 |
<oblivian@deploy2001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 06:54 |
<jforrester@deploy1001> |
scap failed: RuntimeError Scap failed!: 9/9 canaries failed their endpoint checks(http://en.wikipedia.org) (duration: 24m 59s) |
[production] |