1051-1100 of 10000 results (79ms)
2023-06-19 ยง
10:58 <stevemunene@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1059.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001" [production]
10:56 <stevemunene@cumin1001> START - Cookbook sre.dns.netbox [production]
10:55 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1124.eqiad.wmnet with OS bookworm [production]
10:52 <moritzm> imported megacli and ssacli to thirdparty/hwraid for bookworm-wikimedia T339847 [production]
10:48 <stevemunene@cumin1001> START - Cookbook sre.hosts.decommission for hosts analytics1059.eqiad.wmnet [production]
10:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1135 (re)pooling @ 75%: Maint over (T338354)', diff saved to https://phabricator.wikimedia.org/P49448 and previous config saved to /var/cache/conftool/dbconfig/20230619-104702-ladsgroup.json [production]
10:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1135 (re)pooling @ 25%: Maint over (T338354)', diff saved to https://phabricator.wikimedia.org/P49447 and previous config saved to /var/cache/conftool/dbconfig/20230619-103157-ladsgroup.json [production]
10:17 <gmodena@deploy1002> helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
10:16 <gmodena@deploy1002> helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
10:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1135 (re)pooling @ 10%: Maint over (T338354)', diff saved to https://phabricator.wikimedia.org/P49446 and previous config saved to /var/cache/conftool/dbconfig/20230619-101653-ladsgroup.json [production]
10:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1135 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P49445 and previous config saved to /var/cache/conftool/dbconfig/20230619-101623-ladsgroup.json [production]
10:15 <gmodena@deploy1002> helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
10:15 <gmodena@deploy1002> helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
10:04 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1124.eqiad.wmnet with reason: host reimage [production]
10:01 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1124.eqiad.wmnet with reason: host reimage [production]
10:00 <claime> Switching test.wikipedia.org to mw-on-k8s - T337489 [production]
09:51 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1124.eqiad.wmnet with OS bookworm [production]
09:43 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:931231|Enable new spam block page in all wikis except meta, commons, wikidata (T337431)]] (duration: 10m 45s) [production]
09:40 <stevemunene@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts analytics1058.eqiad.wmnet [production]
09:40 <stevemunene@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:40 <stevemunene@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1058.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001" [production]
09:34 <gmodena@deploy1002> helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply [production]
09:34 <gmodena@deploy1002> helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply [production]
09:33 <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:931231|Enable new spam block page in all wikis except meta, commons, wikidata (T337431)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
09:32 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:931231|Enable new spam block page in all wikis except meta, commons, wikidata (T337431)]] [production]
09:30 <stevemunene@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: analytics1058.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1001" [production]
09:30 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:931067|Blocked domains: Fix removing a domain via the special page (T337431)]] (duration: 08m 24s) [production]
09:27 <stevemunene@cumin1001> START - Cookbook sre.dns.netbox [production]
09:22 <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:931067|Blocked domains: Fix removing a domain via the special page (T337431)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
09:21 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:931067|Blocked domains: Fix removing a domain via the special page (T337431)]] [production]
09:21 <stevemunene@cumin1001> START - Cookbook sre.hosts.decommission for hosts analytics1058.eqiad.wmnet [production]
09:15 <kart_> Updated MinT to 2023-06-16-042302-production, Updated people egress (T339271, T335491) [production]
09:12 <kartik@deploy1002> helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply [production]
09:12 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:931066|blocked domains: Make sure users can't bypass the list by using uppercase (T337431)]] (duration: 09m 53s) [production]
09:07 <kartik@deploy1002> helmfile [eqiad] START helmfile.d/services/machinetranslation: apply [production]
09:06 <kartik@deploy1002> helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply [production]
09:03 <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:931066|blocked domains: Make sure users can't bypass the list by using uppercase (T337431)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
09:02 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:931066|blocked domains: Make sure users can't bypass the list by using uppercase (T337431)]] [production]
09:01 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:931087|Temporarily bring back legacy encoding in four wikis (T128150)]] (duration: 07m 31s) [production]
09:00 <kartik@deploy1002> helmfile [codfw] START helmfile.d/services/machinetranslation: apply [production]
08:55 <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:931087|Temporarily bring back legacy encoding in four wikis (T128150)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet [production]
08:53 <cgoubert@deploy1002> helmfile [staging] DONE helmfile.d/services/machinetranslation: apply [production]
08:53 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:931087|Temporarily bring back legacy encoding in four wikis (T128150)]] [production]
08:51 <cgoubert@deploy1002> helmfile [staging] START helmfile.d/services/machinetranslation: apply [production]
08:49 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db1124.eqiad.wmnet with OS bookworm [production]
08:45 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:930925|moveToExternal: First decompress gziped entries before iconv (T128150)]] (duration: 08m 52s) [production]
08:38 <fabfur@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp3050.esams.wmnet [production]
08:38 <fabfur@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp3051.esams.wmnet [production]
08:37 <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:930925|moveToExternal: First decompress gziped entries before iconv (T128150)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
08:36 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:930925|moveToExternal: First decompress gziped entries before iconv (T128150)]] [production]