601-650 of 10000 results (114ms)
2026-04-06 §
14:53 <taavi@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: allocate lvs vip for dumps-lb.eqiad - taavi@cumin1003" [production]
14:49 <taavi@cumin1003> START - Cookbook sre.dns.netbox [production]
14:47 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db1157 (T419635)', diff saved to https://phabricator.wikimedia.org/P90263 and previous config saved to /var/cache/conftool/dbconfig/20260406-144734-fceratto.json [production]
14:47 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
14:40 <vgutierrez@cumin1003> END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[6001,6009].*} and A:cp - 3.2.15 upgrade (T421402) [production]
14:28 <vgutierrez@cumin1003> START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[6001,6009].*} and A:cp - 3.2.15 upgrade (T421402) [production]
14:27 <vgutierrez> fetch haproxy 3.2.15 on thirdparty/haproxy32 (trixie-wikimedia) - T421402 [production]
14:26 <eevans@cumin1003> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Actually upgrade Cassandra to 4.1.11 — T418417 - eevans@cumin1003 [production]
14:24 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
13:20 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Maintenance [production]
13:06 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1023.eqiad.wmnet with reason: Maintenance [production]
12:13 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [production]
12:13 <marostegui@cumin1003> START - Cookbook sre.mysql.parsercache [production]
12:11 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2015.codfw.wmnet with OS trixie [production]
12:06 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1015.eqiad.wmnet with OS trixie [production]
11:53 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2015.codfw.wmnet with reason: host reimage [production]
11:49 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1015.eqiad.wmnet with reason: host reimage [production]
11:48 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on pc2015.codfw.wmnet with reason: host reimage [production]
11:43 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on pc1015.eqiad.wmnet with reason: host reimage [production]
11:29 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host pc2015.codfw.wmnet with OS trixie [production]
11:28 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host pc1015.eqiad.wmnet with OS trixie [production]
11:26 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on pc2015.codfw.wmnet,pc1015.eqiad.wmnet with reason: Maintenance [production]
11:26 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on pc2015.codfw.wmnet with reason: Maintenance [production]
11:24 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [production]
11:24 <marostegui@cumin1003> START - Cookbook sre.mysql.parsercache [production]
11:14 <marostegui@cumin1003> conftool action : set/pooled=yes; selector: name=clouddb1022.eqiad.wmnet,service=s3 [production]
10:15 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [production]
10:15 <marostegui@cumin1003> START - Cookbook sre.mysql.parsercache [production]
10:12 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2142.codfw.wmnet with OS trixie [production]
09:54 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2142.codfw.wmnet with reason: host reimage [production]
09:48 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db2142.codfw.wmnet with reason: host reimage [production]
09:30 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db2142.codfw.wmnet with OS trixie [production]
09:27 <urbanecm@deploy1003> Finished scap sync-world: Backport for [[gerrit:1268196|SECURITY: Protect ApiEchoNotifications with a new user right (T420154)]], [[gerrit:1268197|[i18n] Correct the action message (T420154)]], [[gerrit:1268198|refactor: Use a trait to check for reading permissions (T420154)]], [[gerrit:1268199|Create a new grant for the echo-read-notifications (T420154)]] (duration: 31m 47s) [production]
09:27 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2142.codfw.wmnet,db1152.eqiad.wmnet with reason: Upgrade [production]
09:15 <urbanecm@deploy1003> urbanecm: Continuing with sync [production]
09:15 <urbanecm@deploy1003> urbanecm: Backport for [[gerrit:1268196|SECURITY: Protect ApiEchoNotifications with a new user right (T420154)]], [[gerrit:1268197|[i18n] Correct the action message (T420154)]], [[gerrit:1268198|refactor: Use a trait to check for reading permissions (T420154)]], [[gerrit:1268199|Create a new grant for the echo-read-notifications (T420154)]] synced to the testservers (see https://wikitech.wikimedia [production]
09:10 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [production]
09:10 <marostegui@cumin1003> START - Cookbook sre.mysql.parsercache [production]
09:01 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Upgrade [production]
09:01 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [production]
09:01 <marostegui@cumin1003> START - Cookbook sre.mysql.parsercache [production]
09:01 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool db1152: Upgrade [production]
09:00 <marostegui@cumin1003> dbctl commit (dc=all): 'Pool db1169', diff saved to https://phabricator.wikimedia.org/P90257 and previous config saved to /var/cache/conftool/dbconfig/20260406-090040-marostegui.json [production]
09:00 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1169: test [production]
08:59 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool db1169: test [production]
08:55 <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1268196|SECURITY: Protect ApiEchoNotifications with a new user right (T420154)]], [[gerrit:1268197|[i18n] Correct the action message (T420154)]], [[gerrit:1268198|refactor: Use a trait to check for reading permissions (T420154)]], [[gerrit:1268199|Create a new grant for the echo-read-notifications (T420154)]] [production]
08:48 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1023.eqiad.wmnet with reason: Maintenance [production]
08:46 <urbanecm@deploy1003> Finished scap sync-world: Backport for [[gerrit:1268061|[Growth] Decrease user impact limits back to the defaults (T422288 T341599)]] (duration: 10m 50s) [production]
08:40 <urbanecm@deploy1003> urbanecm: Continuing with sync [production]
08:37 <urbanecm@deploy1003> urbanecm: Backport for [[gerrit:1268061|[Growth] Decrease user impact limits back to the defaults (T422288 T341599)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]