|
2021-03-11
§
|
| 08:50 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1031.eqiad.wmnet |
[production] |
| 08:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1079 (re)pooling @ 60%: Repool db1079 after schema change', diff saved to https://phabricator.wikimedia.org/P14765 and previous config saved to /var/cache/conftool/dbconfig/20210311-084809-root.json |
[production] |
| 08:43 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ms-be1031.eqiad.wmnet |
[production] |
| 08:41 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1030.eqiad.wmnet |
[production] |
| 08:34 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ms-be1030.eqiad.wmnet |
[production] |
| 08:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1079 (re)pooling @ 30%: Repool db1079 after schema change', diff saved to https://phabricator.wikimedia.org/P14764 and previous config saved to /var/cache/conftool/dbconfig/20210311-083305-root.json |
[production] |
| 08:30 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1029.eqiad.wmnet |
[production] |
| 08:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2109', diff saved to https://phabricator.wikimedia.org/P14762 and previous config saved to /var/cache/conftool/dbconfig/20210311-082546-marostegui.json |
[production] |
| 08:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2074', diff saved to https://phabricator.wikimedia.org/P14761 and previous config saved to /var/cache/conftool/dbconfig/20210311-082528-marostegui.json |
[production] |
| 08:24 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2074', diff saved to https://phabricator.wikimedia.org/P14760 and previous config saved to /var/cache/conftool/dbconfig/20210311-082445-marostegui.json |
[production] |
| 08:24 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ms-be1029.eqiad.wmnet |
[production] |
| 08:22 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1028.eqiad.wmnet |
[production] |
| 08:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1079 (re)pooling @ 10%: Repool db1079 after schema change', diff saved to https://phabricator.wikimedia.org/P14759 and previous config saved to /var/cache/conftool/dbconfig/20210311-081801-root.json |
[production] |
| 08:16 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ms-be1028.eqiad.wmnet |
[production] |
| 08:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2108 T275633', diff saved to https://phabricator.wikimedia.org/P14758 and previous config saved to /var/cache/conftool/dbconfig/20210311-081010-marostegui.json |
[production] |
| 08:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db2148 to s2 T275633', diff saved to https://phabricator.wikimedia.org/P14757 and previous config saved to /var/cache/conftool/dbconfig/20210311-080944-marostegui.json |
[production] |
| 07:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P14756 and previous config saved to /var/cache/conftool/dbconfig/20210311-074352-marostegui.json |
[production] |
| 07:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 100%: Repool db1136 after schema change', diff saved to https://phabricator.wikimedia.org/P14755 and previous config saved to /var/cache/conftool/dbconfig/20210311-073741-root.json |
[production] |
| 07:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 60%: Repool db1136 after schema change', diff saved to https://phabricator.wikimedia.org/P14754 and previous config saved to /var/cache/conftool/dbconfig/20210311-072237-root.json |
[production] |
| 07:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 30%: Repool db1136 after schema change', diff saved to https://phabricator.wikimedia.org/P14753 and previous config saved to /var/cache/conftool/dbconfig/20210311-070734-root.json |
[production] |
| 06:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1136 (re)pooling @ 10%: Repool db1136 after schema change', diff saved to https://phabricator.wikimedia.org/P14752 and previous config saved to /var/cache/conftool/dbconfig/20210311-065230-root.json |
[production] |
| 06:48 |
<marostegui> |
Stop mysql on db2108 to clone db2148 T275633 |
[production] |
| 06:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2108 T275633', diff saved to https://phabricator.wikimedia.org/P14750 and previous config saved to /var/cache/conftool/dbconfig/20210311-064821-marostegui.json |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1136', diff saved to https://phabricator.wikimedia.org/P14749 and previous config saved to /var/cache/conftool/dbconfig/20210311-063814-marostegui.json |
[production] |
| 06:36 |
<marostegui> |
Drop testreduce from m5 - T276787 |
[production] |
| 05:34 |
<thcipriani> |
restarted apache2 on gerrit1001 |
[production] |
| 00:50 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2219.codfw.wmnet |
[production] |
| 00:49 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2218.codfw.wmnet |
[production] |
| 00:49 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2217.codfw.wmnet |
[production] |
| 00:48 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2216.codfw.wmnet |
[production] |
| 00:48 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2215.codfw.wmnet |
[production] |
|
2021-03-10
§
|
| 23:49 |
<mholloway-shell@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/EventLogging: EventLogging: Stream always in sample if the user is in debugMode (T276515) (duration: 01m 23s) |
[production] |
| 23:41 |
<dwisehaupt> |
disabled silverpop daily run in process-control until utf8mb4 conversion completes on frdev1001 |
[production] |
| 23:12 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:10 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:10 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry1002.eqiad.wmnet |
[production] |
| 23:01 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts registry1002.eqiad.wmnet |
[production] |
| 22:55 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry[2001-2002].codfw.wmnet |
[production] |
| 22:51 |
<andrewbogott> |
updating puppet compiler facts to catch up with a new custom fact |
[production] |
| 22:44 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts registry[2001-2002].codfw.wmnet |
[production] |
| 22:40 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry1001.eqiad.wmnet |
[production] |
| 22:32 |
<brennen@deploy1002> |
Synchronized php: group1 wikis to 1.36.0-wmf.34 (duration: 01m 30s) |
[production] |
| 22:30 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.34 |
[production] |
| 22:27 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts registry1001.eqiad.wmnet |
[production] |
| 22:26 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): T277094 believed resolved, promoting to group1. |
[production] |
| 22:25 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670535|Fix client error logging (T277094)]] (duration: 01m 09s) |
[production] |
| 21:53 |
<mutante> |
ferm/iptables docker NAT rules applied by puppet on releases servers after breaking out fules into their own profile class (T276869) |
[production] |
| 21:51 |
<dwisehaupt> |
upgraded mariadb and keeping replication stopped on frdb1002 to start the utf8mb4 table alters under a root screen session |
[production] |
| 21:43 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): client errors may still be missing for group0; continuing to hold for T277094 until we know what's broken. |
[production] |
| 21:40 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670533|Revert "Error in shouldLog logic drops most errors" (T277094)]] (duration: 01m 08s) |
[production] |