|
2021-02-16
ยง
|
| 12:46 |
<aborrero@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 12:41 |
<jayme@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 12:39 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/Wikibase.php: Config: [[gerrit:664507|Enable Wikibase Repo ID generator rate limiting on Test Wikidata (T272032)]] 2/2 (duration: 01m 06s) |
[production] |
| 12:38 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:664507|Enable Wikibase Repo ID generator rate limiting on Test Wikidata (T272032)]] 1/2 (duration: 01m 12s) |
[production] |
| 12:29 |
<jayme@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |
| 12:08 |
<jayme@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |
| 12:06 |
<marostegui> |
Deploy schema change on s5 codfw - T273359 |
[production] |
| 11:54 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.30/extensions/DiscussionTools/includes/CommentFormatter.php: 5f4f516177a355b42b896ee142d66c0c969e20f1: CommentFormatter: Fix problems with editsection and quotes (T274709) (duration: 01m 12s) |
[production] |
| 11:54 |
<kharlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 11:54 |
<kharlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 11:52 |
<kharlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 11:52 |
<kharlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 11:47 |
<kharlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 11:47 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1023.eqiad.wmnet |
[production] |
| 11:45 |
<marostegui> |
Failover m2-master back from dbproxy1015 to dbproxy1013 |
[production] |
| 11:42 |
<effie> |
upgrade mc2037 to memcached 1.6 - T270315 |
[production] |
| 11:41 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host mc1023.eqiad.wmnet |
[production] |
| 11:40 |
<marostegui> |
Reboot dbproxy1013 for kernel upgrade |
[production] |
| 11:29 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |
| 11:28 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster |
[production] |
| 11:27 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) for Hadoop test cluster |
[production] |
| 10:53 |
<marostegui> |
Reboot es2023, es2024 and es2025 for kernel upgrade |
[production] |
| 10:46 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.reboot-workers for Hadoop test cluster |
[production] |
| 10:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 100%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14373 and previous config saved to /var/cache/conftool/dbconfig/20210216-103730-root.json |
[production] |
| 10:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 80%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14372 and previous config saved to /var/cache/conftool/dbconfig/20210216-102227-root.json |
[production] |
| 10:19 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 10:18 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'staging' . |
[production] |
| 10:18 |
<marostegui> |
Reboot pc1010 for kernel upgrade |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1075 from dbctl T274235', diff saved to https://phabricator.wikimedia.org/P14371 and previous config saved to /var/cache/conftool/dbconfig/20210216-101710-marostegui.json |
[production] |
| 10:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 60%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14370 and previous config saved to /var/cache/conftool/dbconfig/20210216-100723-root.json |
[production] |
| 09:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 40%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14369 and previous config saved to /var/cache/conftool/dbconfig/20210216-095220-root.json |
[production] |
| 09:40 |
<akosiaris> |
deploy new certs for apertium |
[production] |
| 09:40 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 09:40 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'staging' . |
[production] |
| 09:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 20%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14368 and previous config saved to /var/cache/conftool/dbconfig/20210216-093716-root.json |
[production] |
| 09:28 |
<marostegui> |
Failover m2-master from dbproxy1013 to dbproxy1015 |
[production] |
| 09:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 10%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14367 and previous config saved to /var/cache/conftool/dbconfig/20210216-092213-root.json |
[production] |
| 08:37 |
<godog> |
swift eqiad-prod: decrease weight for SSDs on ms-be[1019-1026] - T272836 |
[production] |
| 08:30 |
<marostegui> |
Deploy schema change on s6 codfw - T273359 |
[production] |
| 07:40 |
<dcausse> |
restarting blazegraph on wdqs1013 |
[production] |
| 07:27 |
<marostegui> |
Reboot dbproxy1021 for kernel upgrade |
[production] |
| 07:21 |
<marostegui> |
Reboot dbproxy1012, 1015, 1016, 1017 for kernel upgrade |
[production] |
| 07:18 |
<marostegui> |
Reboot dbproxy2* for kernel upgrade |
[production] |
| 06:49 |
<marostegui> |
Reboot pc2010 pc2009 pc2008 pc2007 for kernel upgrade |
[production] |
| 06:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1092 to clone db1172 T258361', diff saved to https://phabricator.wikimedia.org/P14365 and previous config saved to /var/cache/conftool/dbconfig/20210216-064602-marostegui.json |
[production] |
| 06:43 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:37 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1093 from dbctl T273955', diff saved to https://phabricator.wikimedia.org/P14364 and previous config saved to /var/cache/conftool/dbconfig/20210216-063250-marostegui.json |
[production] |
| 04:17 |
<jforrester@deploy1001> |
Finished deploy [integration/docroot@864afdb]: Update docroot with changes from this weekend. (duration: 00m 17s) |
[production] |
| 04:17 |
<jforrester@deploy1001> |
Started deploy [integration/docroot@864afdb]: Update docroot with changes from this weekend. |
[production] |