|
2021-11-22
ยง
|
| 17:00 |
<ryankemper@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation restart with plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic plugin upgrade + restart - ryankemper@cumin1001 |
[production] |
| 16:58 |
<ryankemper> |
[Elastic] T295705 Rolling restart w/ plugin upgrade of `relforge` is complete |
[production] |
| 16:55 |
<ryankemper> |
[Elastic] T295705 Restarting second and final relforge host: `ryankemper@relforge1003:~$ sudo systemctl restart elasticsearch_6@relforge-eqiad.service elasticsearch_6@relforge-eqiad-small-alpha.service logstash.service` |
[production] |
| 16:55 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp4032.ulsfo.wmnet with OS buster |
[production] |
| 16:52 |
<ryankemper> |
[Elastic] T295705 Restarting first relforge host: `ryankemper@relforge1004:~$ sudo systemctl restart elasticsearch_6@relforge-eqiad.service elasticsearch_6@relforge-eqiad-small-alpha.service logstash.service` |
[production] |
| 16:51 |
<jayme> |
fleet wide updated wmf-certificates to 0~20211122-1 |
[production] |
| 16:50 |
<vgutierrez> |
depol cp4032 to be reimaged as cache::text_haproxy - T290005 |
[production] |
| 16:49 |
<ryankemper> |
[Elastic] T295705 Downtimed relforge* for 2 hours in order to performing a manual rolling restart of the two hosts `relforge1003` and `relforge1004` |
[production] |
| 16:44 |
<ryankemper> |
T295705 Upgrading `relforge` elasticsearch packages: `ryankemper@cumin1001:~$ sudo cumin -b 2 'relforge*' 'DEBIAN_FRONTEND=noninteractive sudo apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install elasticsearch-oss wmf-elasticsearch-search-plugins'` |
[production] |
| 16:39 |
<ryankemper@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation restart with plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw plugin upgrade + restart - ryankemper@cumin1001 - T295705 |
[production] |
| 16:15 |
<urbanecm> |
Password reset for Miraki@arbcom_dewiki per private request |
[production] |
| 16:15 |
<moritzm> |
installing postgresql-13 security updates on bullseye |
[production] |
| 15:56 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:55 |
<XioNoX> |
Telia DDoS auto-mitigation enabled on all circuits - T288926 |
[production] |
| 15:51 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:28 |
<Amir1> |
revoking DROP for wikiadmin from db1100 (T249683) |
[production] |
| 15:27 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host prometheus2006.codfw.wmnet with OS bullseye |
[production] |
| 15:17 |
<moritzm> |
set kvm:machine_version=pc-i440fx-2.8 for Ganeti cluster in codfw T294119 |
[production] |
| 15:16 |
<jayme> |
imported wmf-certificates 0~20211122-1 to stretch-wikimedia,buster-wikimedia,bullseye-wikimedia |
[production] |
| 15:13 |
<_joe_> |
restarting pybal low-traffic in codfw, eqiad |
[production] |
| 15:07 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:58 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host gitlab-runner1001.wikimedia.org |
[production] |
| 14:55 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:734426|Disable DPL on opt-in wikis where not in use (T287916)]] (duration: 00m 56s) |
[production] |
| 14:54 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host prometheus2006.codfw.wmnet with OS bullseye |
[production] |
| 14:53 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:51 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:734425|Disable DPL on Wikiversities where not in use (T287916)]] (duration: 00m 56s) |
[production] |
| 14:49 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:48 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:734424|Disable DPL on Wikisources where not in use (T287916)]] (duration: 00m 56s) |
[production] |
| 14:44 |
<jelto@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host gitlab-runner1001.wikimedia.org |
[production] |
| 14:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:24 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:23 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Restarting to pick up Java security updates - hnowlan@cumin1001 |
[production] |
| 14:06 |
<akosiaris> |
repool wtp1025, wtp1041 to parsoid cluster. T296098 |
[production] |
| 14:05 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: cluster=parsoid,name=wtp1041.eqiad.wmnet |
[production] |
| 14:05 |
<akosiaris@cumin1001> |
conftool action : set/pooled=yes; selector: cluster=parsoid,name=wtp1025.eqiad.wmnet |
[production] |
| 13:58 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host prometheus2005.codfw.wmnet with OS bullseye |
[production] |
| 13:32 |
<XioNoX> |
re-enable pybal on lvs2007 - T295118 |
[production] |
| 13:31 |
<XioNoX> |
re-enable puppet on lvs2007 |
[production] |
| 13:30 |
<XioNoX> |
re-enabling V6 between cr2-codfw and asw-b-codfw - T295118 |
[production] |
| 13:28 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 13:25 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host prometheus2005.codfw.wmnet with OS bullseye |
[production] |
| 13:24 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 13:20 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.9 |
[production] |
| 13:04 |
<XioNoX> |
asw-b-codfw# set virtual-chassis member 7 mastership-priority 255 - T295118 |
[production] |
| 12:53 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 12:51 |
<Lucas_WMDE> |
UTC morning backport+config window done |
[production] |
| 12:51 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/ProofreadPage/includes/ProofreadPageLuaLibrary.php: Backport: [[gerrit:740556|Lua: use LinkBatch to speed up the template dependencies (T296092)]] (2/2) (duration: 01m 03s) |
[production] |
| 12:49 |
<lucaswerkmeister-wmde@deploy1002> |
Synchronized php-1.38.0-wmf.9/extensions/ProofreadPage/includes/Pagination/Pagination.php: Backport: [[gerrit:740556|Lua: use LinkBatch to speed up the template dependencies (T296092)]] (1/2) (duration: 01m 04s) |
[production] |
| 12:49 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |