|
2021-02-16
§
|
| 11:54 |
<kharlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 11:52 |
<kharlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 11:52 |
<kharlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 11:47 |
<kharlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 11:47 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1023.eqiad.wmnet |
[production] |
| 11:45 |
<marostegui> |
Failover m2-master back from dbproxy1015 to dbproxy1013 |
[production] |
| 11:42 |
<effie> |
upgrade mc2037 to memcached 1.6 - T270315 |
[production] |
| 11:41 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host mc1023.eqiad.wmnet |
[production] |
| 11:40 |
<marostegui> |
Reboot dbproxy1013 for kernel upgrade |
[production] |
| 11:29 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |
| 11:28 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster |
[production] |
| 11:27 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) for Hadoop test cluster |
[production] |
| 10:53 |
<marostegui> |
Reboot es2023, es2024 and es2025 for kernel upgrade |
[production] |
| 10:46 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.reboot-workers for Hadoop test cluster |
[production] |
| 10:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 100%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14373 and previous config saved to /var/cache/conftool/dbconfig/20210216-103730-root.json |
[production] |
| 10:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 80%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14372 and previous config saved to /var/cache/conftool/dbconfig/20210216-102227-root.json |
[production] |
| 10:19 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 10:18 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'staging' . |
[production] |
| 10:18 |
<marostegui> |
Reboot pc1010 for kernel upgrade |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1075 from dbctl T274235', diff saved to https://phabricator.wikimedia.org/P14371 and previous config saved to /var/cache/conftool/dbconfig/20210216-101710-marostegui.json |
[production] |
| 10:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 60%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14370 and previous config saved to /var/cache/conftool/dbconfig/20210216-100723-root.json |
[production] |
| 09:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 40%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14369 and previous config saved to /var/cache/conftool/dbconfig/20210216-095220-root.json |
[production] |
| 09:40 |
<akosiaris> |
deploy new certs for apertium |
[production] |
| 09:40 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 09:40 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'staging' . |
[production] |
| 09:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 20%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14368 and previous config saved to /var/cache/conftool/dbconfig/20210216-093716-root.json |
[production] |
| 09:28 |
<marostegui> |
Failover m2-master from dbproxy1013 to dbproxy1015 |
[production] |
| 09:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 10%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14367 and previous config saved to /var/cache/conftool/dbconfig/20210216-092213-root.json |
[production] |
| 08:37 |
<godog> |
swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836 |
[production] |
| 08:30 |
<marostegui> |
Deploy schema change on s6 codfw - T273359 |
[production] |
| 07:40 |
<dcausse> |
restarting blazegraph on wdqs1013 |
[production] |
| 07:27 |
<marostegui> |
Reboot dbproxy1021 for kernel upgrade |
[production] |
| 07:21 |
<marostegui> |
Reboot dbproxy1012, 1015, 1016, 1017 for kernel upgrade |
[production] |
| 07:18 |
<marostegui> |
Reboot dbproxy2* for kernel upgrade |
[production] |
| 06:49 |
<marostegui> |
Reboot pc2010 pc2009 pc2008 pc2007 for kernel upgrade |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1092 to clone db1172 T258361', diff saved to https://phabricator.wikimedia.org/P14365 and previous config saved to /var/cache/conftool/dbconfig/20210216-064602-marostegui.json |
[production] |
| 06:43 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:37 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1093 from dbctl T273955', diff saved to https://phabricator.wikimedia.org/P14364 and previous config saved to /var/cache/conftool/dbconfig/20210216-063250-marostegui.json |
[production] |
| 04:17 |
<jforrester@deploy1001> |
Finished deploy [integration/docroot@864afdb]: Update docroot with changes from this weekend. (duration: 00m 17s) |
[production] |
| 04:17 |
<jforrester@deploy1001> |
Started deploy [integration/docroot@864afdb]: Update docroot with changes from this weekend. |
[production] |
|
2021-02-15
§
|
| 21:33 |
<eileen> |
civicrm revision changed from dfbb8f41bc to c535ac603a, config revision is ba9b2380b1 |
[production] |
| 16:46 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestage1002.eqiad.wmnet |
[production] |
| 16:39 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kubestage1002.eqiad.wmnet |
[production] |
| 16:33 |
<volans> |
restarted netbox on netbox1001 |
[production] |
| 16:32 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestage1001.eqiad.wmnet |
[production] |
| 16:27 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kubestage1001.eqiad.wmnet |
[production] |
| 16:26 |
<jayme> |
rolled back linkrecommendation helm releases to the most recent revision running chart verion linkrecommendation-0.0.4 on clusters codfw and eqiad (cc: kostajh) |
[production] |
| 16:22 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: name=mwdebug1002.eqiad.wmnet |
[production] |
| 16:18 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestage2002.codfw.wmnet |
[production] |