|
2021-02-16
§
|
| 15:44 |
<klausman@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ml-etcd[1001-1003].eqiad.wmnet with reason: klausman: Pushing new etcd changes from T273071 |
[production] |
| 15:27 |
<cdanis> |
re-enabling Puppet on cp-upload@eqsin to deploy Iab4d211 T274888 |
[production] |
| 15:26 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 15:25 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 15:25 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 15:25 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 15:17 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 15:17 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 15:16 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 15:16 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 15:15 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: WikimediaEvents: Sample mediawiki.client.session_tick at 1:100 (T274172) (duration: 01m 00s) |
[production] |
| 15:14 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 15:14 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 15:13 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 15:13 |
<akosiaris@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 15:12 |
<cdanis> |
previous message was re: T274888 |
[production] |
| 15:11 |
<cdanis> |
✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕙☕ sudo cumin 'A:cp-upload and A:eqsin' 'disable-puppet "cdanis deploying Iab4d211 T263496"' |
[production] |
| 14:38 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.30 refs T271344 bfc73b6e8b33e49e916d9d93cf5cdb7624297d44 |
[production] |
| 14:24 |
<twentyafterfour> |
MediaWiki train: prepare to promote all wikis to 1.36.0-wmf.30 refs T271344 |
[production] |
| 14:07 |
<akosiaris> |
rolling restart of cp500[1-6] |
[production] |
| 13:40 |
<marostegui> |
Deploy schema change on s2 codfw - T273359 |
[production] |
| 13:13 |
<urbanecm@deploy1001> |
Synchronized static/images/cswiki-black-ribbon.png: 5d5b5c41d889f6f30566f23bd9f71d16337b9d6d: Temporarily add cswiki-black-ribbon.png as a static resource (duration: 01m 07s) |
[production] |
| 13:02 |
<jayme@deploy1001> |
helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 12:53 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 12:46 |
<aborrero@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 12:41 |
<jayme@deploy1001> |
helmfile [staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 12:39 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/Wikibase.php: Config: [[gerrit:664507|Enable Wikibase Repo ID generator rate limiting on Test Wikidata (T272032)]] 2/2 (duration: 01m 06s) |
[production] |
| 12:38 |
<lucaswerkmeister-wmde@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:664507|Enable Wikibase Repo ID generator rate limiting on Test Wikidata (T272032)]] 1/2 (duration: 01m 12s) |
[production] |
| 12:29 |
<jayme@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |
| 12:08 |
<jayme@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |
| 12:06 |
<marostegui> |
Deploy schema change on s5 codfw - T273359 |
[production] |
| 11:54 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.30/extensions/DiscussionTools/includes/CommentFormatter.php: 5f4f516177a355b42b896ee142d66c0c969e20f1: CommentFormatter: Fix problems with editsection and quotes (T274709) (duration: 01m 12s) |
[production] |
| 11:54 |
<kharlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 11:54 |
<kharlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 11:52 |
<kharlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 11:52 |
<kharlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 11:47 |
<kharlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 11:47 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc1023.eqiad.wmnet |
[production] |
| 11:45 |
<marostegui> |
Failover m2-master back from dbproxy1015 to dbproxy1013 |
[production] |
| 11:42 |
<effie> |
upgrade mc2037 to memcached 1.6 - T270315 |
[production] |
| 11:41 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host mc1023.eqiad.wmnet |
[production] |
| 11:40 |
<marostegui> |
Reboot dbproxy1013 for kernel upgrade |
[production] |
| 11:29 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'eventrouter' . |
[production] |
| 11:28 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster |
[production] |
| 11:27 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) for Hadoop test cluster |
[production] |
| 10:53 |
<marostegui> |
Reboot es2023, es2024 and es2025 for kernel upgrade |
[production] |
| 10:46 |
<elukey@cumin1001> |
START - Cookbook sre.hadoop.reboot-workers for Hadoop test cluster |
[production] |
| 10:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 100%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14373 and previous config saved to /var/cache/conftool/dbconfig/20210216-103730-root.json |
[production] |
| 10:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1092 (re)pooling @ 80%: Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P14372 and previous config saved to /var/cache/conftool/dbconfig/20210216-102227-root.json |
[production] |
| 10:19 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |