301-350 of 10000 results (75ms)
2023-09-19 §
06:09 <XioNoX> push new pfw policy - T346705 [production]
05:48 <fnegri@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudservices2004-dev.codfw.wmnet with OS bookworm [production]
05:46 <marostegui@cumin1001> START - Cookbook sre.mysql.clone of db1134.eqiad.wmnet onto db1128.eqiad.wmnet [production]
05:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1134', diff saved to https://phabricator.wikimedia.org/P52522 and previous config saved to /var/cache/conftool/dbconfig/20230919-054539-root.json [production]
04:35 <isaranto@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
04:06 <mwpresync@deploy1002> Pruned MediaWiki: 1.41.0-wmf.25 (duration: 02m 10s) [production]
04:03 <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.41.0-wmf.27 refs T345888 (duration: 61m 05s) [production]
03:02 <mwpresync@deploy1002> Started scap: testwikis wikis to 1.41.0-wmf.27 refs T345888 [production]
00:56 <eileen> civicrm upgraded from 0a36997d to f0e9d3f6 [production]
2023-09-18 §
22:09 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wdqs1004.eqiad.wmnet [production]
22:09 <ryankemper@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:08 <ryankemper@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1004.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin1001" [production]
22:07 <ryankemper@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1004.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin1001" [production]
21:59 <ryankemper@cumin1001> START - Cookbook sre.dns.netbox [production]
21:51 <ryankemper@cumin1001> START - Cookbook sre.hosts.decommission for hosts wdqs1004.eqiad.wmnet [production]
21:49 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wdqs1003.eqiad.wmnet [production]
21:49 <ryankemper@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:49 <ryankemper@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin1001" [production]
21:45 <ryankemper@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin1001" [production]
21:40 <ryankemper@cumin1001> START - Cookbook sre.dns.netbox [production]
21:19 <maryum> Deployed patch for T344359 [production]
21:13 <ryankemper@cumin1001> START - Cookbook sre.hosts.decommission for hosts wdqs1003.eqiad.wmnet [production]
20:49 <cjming> end of UTC late backport window [production]
20:36 <cjming@deploy1002> Finished scap: Backport for [[gerrit:957871|Link recommendations: prevent too large offsets in cirrus queries (T345713)]] (duration: 11m 40s) [production]
20:30 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbstore1008.eqiad.wmnet with OS bullseye [production]
20:30 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
20:29 <cjming@deploy1002> urbanecm and cjming: Continuing with sync [production]
20:29 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
20:27 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbstore1009.eqiad.wmnet with OS bullseye [production]
20:27 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
20:26 <cjming@deploy1002> urbanecm and cjming: Backport for [[gerrit:957871|Link recommendations: prevent too large offsets in cirrus queries (T345713)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:24 <cjming@deploy1002> Started scap: Backport for [[gerrit:957871|Link recommendations: prevent too large offsets in cirrus queries (T345713)]] [production]
20:24 <cjming@deploy1002> Finished scap: Backport for [[gerrit:958025|clienthints: Enable purging of data on all wikis (T257893)]] (duration: 09m 24s) [production]
20:24 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
20:16 <cjming@deploy1002> cjming and dreamyjazz: Continuing with sync [production]
20:16 <cjming@deploy1002> cjming and dreamyjazz: Backport for [[gerrit:958025|clienthints: Enable purging of data on all wikis (T257893)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:16 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbstore1008.eqiad.wmnet with reason: host reimage [production]
20:15 <cjming@deploy1002> Started scap: Backport for [[gerrit:958025|clienthints: Enable purging of data on all wikis (T257893)]] [production]
20:13 <cjming@deploy1002> Finished scap: Backport for [[gerrit:958024|clienthints: Pin wgCheckUserDisplayClientHints to false (T337942)]] (duration: 08m 18s) [production]
20:12 <jhancock@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dbstore1008.eqiad.wmnet with reason: host reimage [production]
20:09 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbstore1009.eqiad.wmnet with reason: host reimage [production]
20:06 <cjming@deploy1002> cjming and dreamyjazz: Continuing with sync [production]
20:06 <cjming@deploy1002> cjming and dreamyjazz: Backport for [[gerrit:958024|clienthints: Pin wgCheckUserDisplayClientHints to false (T337942)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:06 <jhancock@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dbstore1009.eqiad.wmnet with reason: host reimage [production]
20:05 <cjming@deploy1002> Started scap: Backport for [[gerrit:958024|clienthints: Pin wgCheckUserDisplayClientHints to false (T337942)]] [production]
19:46 <isaranto@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
19:43 <isaranto@deploy1002> helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
19:22 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host dbstore1009.eqiad.wmnet with OS bullseye [production]
19:22 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host dbstore1008.eqiad.wmnet with OS bullseye [production]
18:02 <ejegg> re-enabled donor thank you mail send jobs [production]