1-50 of 10000 results (65ms)
2023-08-30 ยง
22:28 <krinkle@deploy1002> Synchronized php-1.41.0-wmf.24/extensions/WikimediaEvents/: 697ab03ae9a5d5ddb6 (duration: 06m 26s) [production]
22:09 <krinkle@deploy1002> Finished scap: Backport for [[gerrit:953649|mediawiki.util: Investigate when mw.util is compromised by third-party script (T343944)]] (duration: 35m 08s) [production]
21:57 <krinkle@deploy1002> krinkle: Continuing with sync [production]
21:55 <krinkle@deploy1002> krinkle: Backport for [[gerrit:953649|mediawiki.util: Investigate when mw.util is compromised by third-party script (T343944)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
21:34 <krinkle@deploy1002> Started scap: Backport for [[gerrit:953649|mediawiki.util: Investigate when mw.util is compromised by third-party script (T343944)]] [production]
21:31 <Krinkle> krinkle@deploy1002: running `sudo /usr/local/sbin/fix-staging-perms` two fix permissions under /srv/patches/1.41.0-wmf.24 where 2 of the 3 patch files are read-only by jnuche:deployment [production]
20:44 <hmonroy@deploy1002> Finished scap: Backport for [[gerrit:952925|Add comment about mirroring of wgMobileUrlTemplate (T344185)]] (duration: 07m 11s) [production]
20:37 <hmonroy@deploy1002> Started scap: Backport for [[gerrit:952925|Add comment about mirroring of wgMobileUrlTemplate (T344185)]] [production]
20:33 <hmonroy@deploy1002> Finished scap: Backport for [[gerrit:953648|Omit 'target' in the body of review REST API requests]] (duration: 08m 18s) [production]
20:27 <hmonroy@deploy1002> matmarex and hmonroy: Continuing with sync [production]
20:26 <hmonroy@deploy1002> matmarex and hmonroy: Backport for [[gerrit:953648|Omit 'target' in the body of review REST API requests]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:24 <hmonroy@deploy1002> Started scap: Backport for [[gerrit:953648|Omit 'target' in the body of review REST API requests]] [production]
20:14 <hmonroy@deploy1002> Finished scap: Backport for [[gerrit:952940|wikidiff2: set maxSplitSize = 10 by default (T341754)]] (duration: 09m 13s) [production]
20:08 <hmonroy@deploy1002> hmonroy: Continuing with sync [production]
20:06 <hmonroy@deploy1002> hmonroy: Backport for [[gerrit:952940|wikidiff2: set maxSplitSize = 10 by default (T341754)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
20:05 <hmonroy@deploy1002> Started scap: Backport for [[gerrit:952940|wikidiff2: set maxSplitSize = 10 by default (T341754)]] [production]
19:19 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply security updates - bking@cumin1001 - T344587 [production]
19:17 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts wdqs1005.eqiad.wmnet [production]
18:41 <cmooney@cumin1001> END (FAIL) - Cookbook sre.network.provision (exit_code=99) for device ssw1-a1-codfw.mgmt.codfw.wmnet [production]
18:41 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:41 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove management record for ssw1-a1-codfw - cmooney@cumin1001" [production]
18:39 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove management record for ssw1-a1-codfw - cmooney@cumin1001" [production]
18:37 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
18:28 <ryankemper@cumin1001> START - Cookbook sre.hosts.decommission for hosts wdqs1005.eqiad.wmnet [production]
18:19 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host doh2001.wikimedia.org with OS bookworm [production]
18:15 <jhuneidi@deploy1002> Synchronized php: group1 wikis to 1.41.0-wmf.24 refs T343726 (duration: 06m 13s) [production]
18:08 <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.41.0-wmf.24 refs T343726 [production]
18:04 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on doh2001.wikimedia.org with reason: host reimage [production]
18:03 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply security updates - bking@cumin1001 - T344587 [production]
18:03 <bking@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: apply security updates - bking@cumin1001 - T344587 [production]
18:03 <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_eqiad: apply security updates - bking@cumin1001 - T344587 [production]
18:01 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on doh2001.wikimedia.org with reason: host reimage [production]
17:46 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host doh2001.wikimedia.org with OS bookworm [production]
16:36 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:948542|ores-extension: replace thresholds with numeric values (T343308)]] (duration: 10m 09s) [production]
16:30 <ladsgroup@deploy1002> ladsgroup and isaranto: Continuing with sync [production]
16:28 <ladsgroup@deploy1002> ladsgroup and isaranto: Backport for [[gerrit:948542|ores-extension: replace thresholds with numeric values (T343308)]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
16:26 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:948542|ores-extension: replace thresholds with numeric values (T343308)]] [production]
16:19 <bking@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply security updates - bking@cumin1001 - T344587 [production]
16:08 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
16:08 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
15:56 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/rdf-streaming-updater: apply [production]
15:56 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/rdf-streaming-updater: apply [production]
15:56 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/rdf-streaming-updater: apply [production]
15:56 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/rdf-streaming-updater: apply [production]
15:54 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubemaster2001.codfw.wmnet [production]
15:50 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single for host kubemaster2001.codfw.wmnet [production]
15:49 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host doh4001.wikimedia.org with OS bookworm [production]
15:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
15:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
15:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1222 (T343718)', diff saved to https://phabricator.wikimedia.org/P52113 and previous config saved to /var/cache/conftool/dbconfig/20230830-154437-ladsgroup.json [production]