|
2023-06-19
ยง
|
| 10:52 |
<moritzm> |
imported megacli and ssacli to thirdparty/hwraid for bookworm-wikimedia T339847 |
[production] |
| 10:48 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts analytics1059.eqiad.wmnet |
[production] |
| 10:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1135 (re)pooling @ 75%: Maint over (T338354)', diff saved to https://phabricator.wikimedia.org/P49448 and previous config saved to /var/cache/conftool/dbconfig/20230619-104702-ladsgroup.json |
[production] |
| 10:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1135 (re)pooling @ 25%: Maint over (T338354)', diff saved to https://phabricator.wikimedia.org/P49447 and previous config saved to /var/cache/conftool/dbconfig/20230619-103157-ladsgroup.json |
[production] |
| 10:17 |
<gmodena@deploy1002> |
helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 10:16 |
<gmodena@deploy1002> |
helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 10:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1135 (re)pooling @ 10%: Maint over (T338354)', diff saved to https://phabricator.wikimedia.org/P49446 and previous config saved to /var/cache/conftool/dbconfig/20230619-101653-ladsgroup.json |
[production] |
| 10:16 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'db1135 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P49445 and previous config saved to /var/cache/conftool/dbconfig/20230619-101623-ladsgroup.json |
[production] |
| 10:15 |
<gmodena@deploy1002> |
helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 10:15 |
<gmodena@deploy1002> |
helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 10:04 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1124.eqiad.wmnet with reason: host reimage |
[production] |
| 10:01 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1124.eqiad.wmnet with reason: host reimage |
[production] |
| 10:00 |
<claime> |
Switching test.wikipedia.org to mw-on-k8s - T337489 |
[production] |
| 09:51 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db1124.eqiad.wmnet with OS bookworm |
[production] |
| 09:43 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:931231|Enable new spam block page in all wikis except meta, commons, wikidata (T337431)]] (duration: 10m 45s) |
[production] |
| 09:40 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts analytics1058.eqiad.wmnet |
[production] |
| 09:40 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 09:40 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1058.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001" |
[production] |
| 09:34 |
<gmodena@deploy1002> |
helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 09:34 |
<gmodena@deploy1002> |
helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply |
[production] |
| 09:33 |
<ladsgroup@deploy1002> |
ladsgroup: Backport for [[gerrit:931231|Enable new spam block page in all wikis except meta, commons, wikidata (T337431)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 09:32 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:931231|Enable new spam block page in all wikis except meta, commons, wikidata (T337431)]] |
[production] |
| 09:30 |
<stevemunene@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1058.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001" |
[production] |
| 09:30 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:931067|Blocked domains: Fix removing a domain via the special page (T337431)]] (duration: 08m 24s) |
[production] |
| 09:27 |
<stevemunene@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 09:22 |
<ladsgroup@deploy1002> |
ladsgroup: Backport for [[gerrit:931067|Blocked domains: Fix removing a domain via the special page (T337431)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet |
[production] |
| 09:21 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:931067|Blocked domains: Fix removing a domain via the special page (T337431)]] |
[production] |
| 09:21 |
<stevemunene@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts analytics1058.eqiad.wmnet |
[production] |
| 09:15 |
<kart_> |
Updated MinT to 2023-06-16-042302-production, Updated people egress (T339271, T335491) |
[production] |
| 09:12 |
<kartik@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 09:12 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:931066|blocked domains: Make sure users can't bypass the list by using uppercase (T337431)]] (duration: 09m 53s) |
[production] |
| 09:07 |
<kartik@deploy1002> |
helmfile [eqiad] START helmfile.d/services/machinetranslation: apply |
[production] |
| 09:06 |
<kartik@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 09:03 |
<ladsgroup@deploy1002> |
ladsgroup: Backport for [[gerrit:931066|blocked domains: Make sure users can't bypass the list by using uppercase (T337431)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 09:02 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:931066|blocked domains: Make sure users can't bypass the list by using uppercase (T337431)]] |
[production] |
| 09:01 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:931087|Temporarily bring back legacy encoding in four wikis (T128150)]] (duration: 07m 31s) |
[production] |
| 09:00 |
<kartik@deploy1002> |
helmfile [codfw] START helmfile.d/services/machinetranslation: apply |
[production] |
| 08:55 |
<ladsgroup@deploy1002> |
ladsgroup: Backport for [[gerrit:931087|Temporarily bring back legacy encoding in four wikis (T128150)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 08:53 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 08:53 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:931087|Temporarily bring back legacy encoding in four wikis (T128150)]] |
[production] |
| 08:51 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/machinetranslation: apply |
[production] |
| 08:49 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1124.eqiad.wmnet with OS bookworm |
[production] |
| 08:45 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:930925|moveToExternal: First decompress gziped entries before iconv (T128150)]] (duration: 08m 52s) |
[production] |
| 08:38 |
<fabfur@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp3050.esams.wmnet |
[production] |
| 08:38 |
<fabfur@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp3051.esams.wmnet |
[production] |
| 08:37 |
<ladsgroup@deploy1002> |
ladsgroup: Backport for [[gerrit:930925|moveToExternal: First decompress gziped entries before iconv (T128150)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 08:36 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:930925|moveToExternal: First decompress gziped entries before iconv (T128150)]] |
[production] |
| 08:30 |
<fabfur> |
rebooting cp3051 and cp3051 for kernel upgrade (T335835) |
[production] |
| 08:29 |
<fabfur@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cp3050.esams.wmnet |
[production] |
| 08:29 |
<fabfur@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cp3051.esams.wmnet |
[production] |