|
2021-02-16
§
|
| 15:14 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 15:14 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 15:13 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 15:13 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 15:12 |
<cdanis> |
previous message was re: T274888 |
[production] |
| 15:11 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕙☕ sudo cumin 'A:cp-upload and A:eqsin' 'disable-puppet "cdanis deploying Iab4d211 T263496"' |
[production] |
| 14:38 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.30 refs T271344 bfc73b6e8b33e49e916d9d93cf5cdb7624297d44 |
[production] |
| 14:24 |
<twentyafterfour> |
MediaWiki train: prepare to promote all wikis to 1.36.0-wmf.30 refs T271344 |
[production] |
| 14:07 |
<akosiaris> |
rolling restart of cp500[1-6] |
[production] |
| 13:40 |
<marostegui> |
Deploy schema change on s2 codfw - T273359 |
[production] |
| 13:13 |
<urbanecm@deploy1001> |
Synchronized static/images/cswiki-black-ribbon.png: 5d5b5c41d889f6f30566f23bd9f71d16337b9d6d: Temporarily add cswiki-black-ribbon.png as a static resource (duration: 01m 07s) |
[production] |
| 13:02 |
<jayme@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 12:53 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 12:46 |
<aborrero@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 12:41 |
<jayme@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 12:39 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/Wikibase.php: Config: [[gerrit:664507|Enable Wikibase Repo ID generator rate limiting on Test Wikidata (T272032)]] 2/2 (duration: 01m 06s) |
[production] |
| 12:38 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:664507|Enable Wikibase Repo ID generator rate limiting on Test Wikidata (T272032)]] 1/2 (duration: 01m 12s) |
[production] |
| 12:29 |
<jayme@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |
| 12:08 |
<jayme@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |
| 12:06 |
<marostegui> |
Deploy schema change on s5 codfw - T273359 |
[production] |
| 11:54 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.30/extensions/DiscussionTools/includes/CommentFormatter.php: 5f4f516177a355b42b896ee142d66c0c969e20f1: CommentFormatter: Fix problems with editsection and quotes (T274709) (duration: 01m 12s) |
[production] |
| 11:54 |
<kharlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 11:54 |
<kharlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 11:52 |
<kharlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 11:52 |
<kharlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 11:47 |
<kharlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 11:47 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1023.eqiad.wmnet |
[production] |
| 11:45 |
<marostegui> |
Failover m2-master back from dbproxy1015 to dbproxy1013 |
[production] |
| 11:42 |
<effie> |
upgrade mc2037 to memcached 1.6 - T270315 |
[production] |
| 11:41 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host mc1023.eqiad.wmnet |
[production] |
| 11:40 |
<marostegui> |
Reboot dbproxy1013 for kernel upgrade |
[production] |
| 11:29 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |
| 11:28 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster |
[production] |
| 11:27 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) for Hadoop test cluster |
[production] |
| 10:53 |
<marostegui> |
Reboot es2023, es2024 and es2025 for kernel upgrade |
[production] |
| 10:46 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.reboot-workers for Hadoop test cluster |
[production] |
| 10:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 100%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14373 and previous config saved to /var/cache/conftool/dbconfig/20210216-103730-root.json |
[production] |
| 10:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 80%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14372 and previous config saved to /var/cache/conftool/dbconfig/20210216-102227-root.json |
[production] |
| 10:19 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 10:18 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'staging' . |
[production] |
| 10:18 |
<marostegui> |
Reboot pc1010 for kernel upgrade |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1075 from dbctl T274235', diff saved to https://phabricator.wikimedia.org/P14371 and previous config saved to /var/cache/conftool/dbconfig/20210216-101710-marostegui.json |
[production] |
| 10:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 60%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14370 and previous config saved to /var/cache/conftool/dbconfig/20210216-100723-root.json |
[production] |
| 09:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 40%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14369 and previous config saved to /var/cache/conftool/dbconfig/20210216-095220-root.json |
[production] |
| 09:40 |
<akosiaris> |
deploy new certs for apertium |
[production] |
| 09:40 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 09:40 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'staging' . |
[production] |
| 09:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 20%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14368 and previous config saved to /var/cache/conftool/dbconfig/20210216-093716-root.json |
[production] |
| 09:28 |
<marostegui> |
Failover m2-master from dbproxy1013 to dbproxy1015 |
[production] |
| 09:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 10%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14367 and previous config saved to /var/cache/conftool/dbconfig/20210216-092213-root.json |
[production] |