|
2024-04-25
§
|
| 05:36 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2161 (T352010)', diff saved to https://phabricator.wikimedia.org/P61191 and previous config saved to /var/cache/conftool/dbconfig/20240425-053608-ladsgroup.json |
[production] |
| 05:36 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance |
[production] |
| 05:35 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance |
[production] |
| 05:35 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2154 (T352010)', diff saved to https://phabricator.wikimedia.org/P61190 and previous config saved to /var/cache/conftool/dbconfig/20240425-053545-ladsgroup.json |
[production] |
| 05:23 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P61189 and previous config saved to /var/cache/conftool/dbconfig/20240425-052354-ladsgroup.json |
[production] |
| 05:20 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P61188 and previous config saved to /var/cache/conftool/dbconfig/20240425-052038-ladsgroup.json |
[production] |
| 05:08 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2153 (T352010)', diff saved to https://phabricator.wikimedia.org/P61187 and previous config saved to /var/cache/conftool/dbconfig/20240425-050845-ladsgroup.json |
[production] |
| 05:05 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P61186 and previous config saved to /var/cache/conftool/dbconfig/20240425-050531-ladsgroup.json |
[production] |
| 04:50 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2154 (T352010)', diff saved to https://phabricator.wikimedia.org/P61185 and previous config saved to /var/cache/conftool/dbconfig/20240425-045023-ladsgroup.json |
[production] |
|
2024-04-24
§
|
| 21:52 |
<dzahn@cumin2002> |
END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: security release T363349 |
[production] |
| 21:36 |
<ryankemper> |
[Elastic] T361268 Pooled new hosts: `elastic110[3-7]` |
[production] |
| 21:35 |
<ryankemper@puppetmaster1001> |
conftool action : set/weight=10:pooled=yes; selector: name=elastic110[3-7]\.eqiad\.wmnet |
[production] |
| 20:38 |
<denisse> |
Disabling Puppet on the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414 |
[production] |
| 20:38 |
<denisse@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on prometheus6002.drmrs.wmnet,prometheus5002.eqsin.wmnet,prometheus3003.esams.wmnet,prometheus4002.ulsfo.wmnet with reason: Downtiming the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414 |
[production] |
| 20:37 |
<denisse@cumin2002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on prometheus6002.drmrs.wmnet,prometheus5002.eqsin.wmnet,prometheus3003.esams.wmnet,prometheus4002.ulsfo.wmnet with reason: Downtiming the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414 |
[production] |
| 20:37 |
<denisse> |
Downtiming the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414 |
[production] |
| 20:24 |
<ebernhardson@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 20:24 |
<ebernhardson@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 19:32 |
<dzahn@cumin2002> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: security release T363349 |
[production] |
| 19:27 |
<cstone> |
payments-wiki upgraded from 1895e43b to c7ab847d |
[production] |
| 19:15 |
<bking@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 19:15 |
<bking@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 19:14 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Apply truststore changes — T352647 - eevans@cumin1002 |
[production] |
| 19:08 |
<inflatador> |
bking@deploy1002 stop `consumer-cloudelastic` release to test alerting T359213 |
[production] |
| 19:07 |
<bking@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 19:06 |
<bking@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 19:03 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2153 (T352010)', diff saved to https://phabricator.wikimedia.org/P61181 and previous config saved to /var/cache/conftool/dbconfig/20240424-190237-ladsgroup.json |
[production] |
| 19:03 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance |
[production] |
| 19:02 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance |
[production] |
| 19:02 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2146 (T352010)', diff saved to https://phabricator.wikimedia.org/P61180 and previous config saved to /var/cache/conftool/dbconfig/20240424-190214-ladsgroup.json |
[production] |
| 18:57 |
<amastilovic@deploy1002> |
Finished deploy [airflow-dags/analytics@3f994d5]: (no justification provided) (duration: 00m 28s) |
[production] |
| 18:57 |
<amastilovic@deploy1002> |
Started deploy [airflow-dags/analytics@3f994d5]: (no justification provided) |
[production] |
| 18:47 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P61179 and previous config saved to /var/cache/conftool/dbconfig/20240424-184707-ladsgroup.json |
[production] |
| 18:32 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P61178 and previous config saved to /var/cache/conftool/dbconfig/20240424-183200-ladsgroup.json |
[production] |
| 18:20 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.43.0-wmf.2 refs T361396 |
[production] |
| 18:16 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2146 (T352010)', diff saved to https://phabricator.wikimedia.org/P61177 and previous config saved to /var/cache/conftool/dbconfig/20240424-181653-ladsgroup.json |
[production] |
| 18:14 |
<cstone> |
payments-wiki upgraded from fb0367a4 to 1895e43b |
[production] |
| 18:03 |
<brennen> |
train 1.43.0-wmf.2 (T361396) status: no current blockers, rolling to group1 |
[production] |
| 17:41 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) |
[production] |
| 17:20 |
<btullis@cumin1002> |
START - Cookbook sre.wdqs.restart |
[production] |
| 17:05 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db2154 (T352010)', diff saved to https://phabricator.wikimedia.org/P61176 and previous config saved to /var/cache/conftool/dbconfig/20240424-170444-ladsgroup.json |
[production] |
| 17:04 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance |
[production] |
| 17:04 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance |
[production] |
| 17:04 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2152 (T352010)', diff saved to https://phabricator.wikimedia.org/P61175 and previous config saved to /var/cache/conftool/dbconfig/20240424-170421-ladsgroup.json |
[production] |
| 17:03 |
<hmonroy@deploy1002> |
Finished scap: Backport for [[gerrit:1023501|[hewiki] enable CodeMirrorV6 and CodeMirrorLineNumberingNamespaces (T357795 T347211)]] (duration: 20m 36s) |
[production] |
| 16:51 |
<eevans@cumin1002> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Apply truststore changes — T352647 - eevans@cumin1002 |
[production] |
| 16:50 |
<hmonroy@deploy1002> |
musikanimal and hmonroy: Continuing with sync |
[production] |
| 16:49 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1010.eqiad.wmnet |
[production] |
| 16:49 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P61174 and previous config saved to /var/cache/conftool/dbconfig/20240424-164914-ladsgroup.json |
[production] |
| 16:45 |
<hmonroy@deploy1002> |
musikanimal and hmonroy: Backport for [[gerrit:1023501|[hewiki] enable CodeMirrorV6 and CodeMirrorLineNumberingNamespaces (T357795 T347211)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |