1251-1300 of 10000 results (81ms)
2023-05-03 ยง
08:04 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org [production]
07:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 4%: Pooling db1213:3316 T326669', diff saved to https://phabricator.wikimedia.org/P47308 and previous config saved to /var/cache/conftool/dbconfig/20230503-075828-root.json [production]
07:58 <marostegui@cumin1001> dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 4%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47307 and previous config saved to /var/cache/conftool/dbconfig/20230503-075818-root.json [production]
07:57 <jelto@cumin1001> START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org [production]
07:48 <jelto@cumin1001> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Install software version upgrade [production]
07:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 3%: Pooling db1213:3316 T326669', diff saved to https://phabricator.wikimedia.org/P47306 and previous config saved to /var/cache/conftool/dbconfig/20230503-074323-root.json [production]
07:43 <marostegui@cumin1001> dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 3%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47305 and previous config saved to /var/cache/conftool/dbconfig/20230503-074313-root.json [production]
07:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1110 T335011', diff saved to https://phabricator.wikimedia.org/P47304 and previous config saved to /var/cache/conftool/dbconfig/20230503-073602-root.json [production]
07:28 <taavi@deploy1002> Finished scap: Backport for [[gerrit:914429|Remove duplicated diff-mode selector in save dialog (T324759)]] (duration: 10m 14s) [production]
07:28 <marostegui@cumin1001> dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 2%: Pooling db1213:3316 T326669', diff saved to https://phabricator.wikimedia.org/P47303 and previous config saved to /var/cache/conftool/dbconfig/20230503-072818-root.json [production]
07:28 <marostegui@cumin1001> dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 2%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47302 and previous config saved to /var/cache/conftool/dbconfig/20230503-072808-root.json [production]
07:26 <elukey@deploy1002> helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' . [production]
07:20 <taavi@deploy1002> taavi and samwilson: Backport for [[gerrit:914429|Remove duplicated diff-mode selector in save dialog (T324759)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
07:18 <taavi@deploy1002> Started scap: Backport for [[gerrit:914429|Remove duplicated diff-mode selector in save dialog (T324759)]] [production]
07:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db1213:3316 (re)pooling @ 1%: Pooling db1213:3316 T326669', diff saved to https://phabricator.wikimedia.org/P47299 and previous config saved to /var/cache/conftool/dbconfig/20230503-071313-root.json [production]
07:13 <marostegui@cumin1001> dbctl commit (dc=all): 'db1213:3315 (re)pooling @ 1%: Pooling db1213:3315 T326669', diff saved to https://phabricator.wikimedia.org/P47298 and previous config saved to /var/cache/conftool/dbconfig/20230503-071303-root.json [production]
07:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1213 (s5,s6) to dbctl T326669', diff saved to https://phabricator.wikimedia.org/P47297 and previous config saved to /var/cache/conftool/dbconfig/20230503-071046-marostegui.json [production]
07:09 <moritzm> installing glibc bugfix updates from bullseye point release [production]
07:02 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1117.eqiad.wmnet [production]
07:02 <marostegui@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
07:02 <marostegui@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1117.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" [production]
07:01 <marostegui@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1117.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1001" [production]
06:56 <marostegui@cumin1001> START - Cookbook sre.dns.netbox [production]
06:50 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission for hosts db1117.eqiad.wmnet [production]
06:46 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 38 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
06:46 <marostegui> Disconnect codfw -> eqiad replication on s1 T335267 [production]
06:46 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on 38 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
06:29 <ayounsi@cumin1001> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling update on A:netbox [production]
06:28 <ayounsi@cumin1001> START - Cookbook sre.netbox.update-extras rolling update on A:netbox [production]
06:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 34 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
06:14 <marostegui> Disconnect codfw -> eqiad replication on s8 T335267 [production]
06:14 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on 34 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
06:09 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 35 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
06:09 <marostegui> Disconnect codfw -> eqiad replication on s4 T335267 [production]
06:09 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on 35 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
06:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 28 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
06:06 <marostegui> Disconnect codfw -> eqiad replication on s7 T335267 [production]
06:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on 28 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
06:02 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 24 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
06:01 <marostegui> Disconnect codfw -> eqiad replication on s3 T335267 [production]
06:01 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on 24 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
05:59 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 26 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
05:59 <marostegui> Disconnect codfw -> eqiad replication on s5 T335267 [production]
05:59 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on 26 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
05:58 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 27 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
05:57 <marostegui> Disconnect codfw -> eqiad replication on s2 T335267 [production]
05:57 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on 27 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
05:54 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 27 hosts with reason: Disconnecting codfw > eqiad T335267 [production]
05:54 <marostegui> Disconnect codfw -> eqiad replication on s6 T335267 [production]
05:54 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on 27 hosts with reason: Disconnecting codfw > eqiad T335267 [production]