151-200 of 10000 results (72ms)
2023-03-09 ยง
15:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2172 (T329260)', diff saved to https://phabricator.wikimedia.org/P45713 and previous config saved to /var/cache/conftool/dbconfig/20230309-155520-marostegui.json [production]
15:55 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2172.codfw.wmnet with reason: Maintenance [production]
15:55 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2172.codfw.wmnet with reason: Maintenance [production]
15:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T329260)', diff saved to https://phabricator.wikimedia.org/P45712 and previous config saved to /var/cache/conftool/dbconfig/20230309-155459-marostegui.json [production]
15:40 <marostegui@cumin1001> dbctl commit (dc=all): 'db2163 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P45711 and previous config saved to /var/cache/conftool/dbconfig/20230309-154053-root.json [production]
15:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P45710 and previous config saved to /var/cache/conftool/dbconfig/20230309-153953-marostegui.json [production]
15:29 <brett@cumin2002> START - Cookbook sre.ganeti.reimage for host acmechief1001.eqiad.wmnet with OS bullseye [production]
15:27 <brett> Enable puppet on R:acme_chief::cert - T321309 [production]
15:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P45709 and previous config saved to /var/cache/conftool/dbconfig/20230309-152447-marostegui.json [production]
15:15 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:15 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for codfw cr links to cloudsw-b1-codfw. - cmooney@cumin1001" [production]
15:15 <moritzm> installing PHP 7.3 security updates (as shipped in Debian) [production]
15:14 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS records for codfw cr links to cloudsw-b1-codfw. - cmooney@cumin1001" [production]
15:14 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
15:13 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
15:12 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
15:11 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
15:11 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
15:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2163 (T329203)', diff saved to https://phabricator.wikimedia.org/P45707 and previous config saved to /var/cache/conftool/dbconfig/20230309-151100-marostegui.json [production]
15:10 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2163.codfw.wmnet with reason: Maintenance [production]
15:10 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2163.codfw.wmnet with reason: Maintenance [production]
15:10 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
15:10 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
15:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T329260)', diff saved to https://phabricator.wikimedia.org/P45706 and previous config saved to /var/cache/conftool/dbconfig/20230309-150940-marostegui.json [production]
15:06 <brett> Disable puppet on R:acme_chief::cert for acmechief maintenance - T321309 [production]
15:04 <zabe@deploy2002> Finished scap: Backport for [[gerrit:790707|Drop unused FlaggedRevs threshold level names (T277883)]] (duration: 10m 48s) [production]
15:04 <TheresNoTime> close UTC afternoon backport window [production]
15:02 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db[2135,2160].codfw.wmnet,db[1117,1176,1183].eqiad.wmnet with reason: m5 master switch T330847 [production]
15:02 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:30:00 on db[2135,2160].codfw.wmnet,db[1117,1176,1183].eqiad.wmnet with reason: m5 master switch T330847 [production]
15:01 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
15:01 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
15:00 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
15:00 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
14:56 <hnowlan@deploy2002> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
14:55 <zabe@deploy2002> awight and zabe: Backport for [[gerrit:790707|Drop unused FlaggedRevs threshold level names (T277883)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
14:55 <hnowlan@deploy2002> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
14:54 <zabe@deploy2002> Started scap: Backport for [[gerrit:790707|Drop unused FlaggedRevs threshold level names (T277883)]] [production]
14:34 <moritzm> installing apr security updates [production]
14:31 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
14:31 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
14:30 <jgiannelos@deploy2002> Finished deploy [restbase/deploy@f774711]: (no justification provided) (duration: 19m 03s) [production]
14:13 <samtar@deploy2002> Finished scap: Backport for [[gerrit:886905|Bump parsoid parser cache writes to 50%. (T320534)]] (duration: 07m 28s) [production]
14:11 <jgiannelos@deploy2002> Started deploy [restbase/deploy@f774711]: (no justification provided) [production]
14:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2155 (T329260)', diff saved to https://phabricator.wikimedia.org/P45705 and previous config saved to /var/cache/conftool/dbconfig/20230309-140915-marostegui.json [production]
14:09 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
14:09 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
14:09 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
14:08 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
14:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2147 (T329260)', diff saved to https://phabricator.wikimedia.org/P45704 and previous config saved to /var/cache/conftool/dbconfig/20230309-140850-marostegui.json [production]
14:08 <Emperor> testing disk-swap in ms-be1066 T329305 [production]