2551-2600 of 10000 results (83ms)
2023-05-17 §
05:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1112 from dbctl T336332', diff saved to https://phabricator.wikimedia.org/P48263 and previous config saved to /var/cache/conftool/dbconfig/20230517-052007-marostegui.json [production]
05:16 <marostegui> Optimize s7 on dbstore1003 T336733 [production]
00:21 <krinkle@deploy1002> Synchronized src/: I4cfa4a2474b4e (duration: 06m 01s) [production]
00:15 <krinkle@deploy1002> Synchronized wmf-config/: I4cfa4a2474b4e (duration: 06m 14s) [production]
00:07 <krinkle@deploy1002> Synchronized lib/: I4cfa4a2474b4e (duration: 06m 51s) [production]
2023-05-16 §
20:59 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:920237|Add maint script to opt out active users from the new topic tool (T317375)]] (duration: 07m 18s) [production]
20:53 <jdrewniak@deploy1002> jdrewniak and matmarex: Backport for [[gerrit:920237|Add maint script to opt out active users from the new topic tool (T317375)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
20:52 <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:920237|Add maint script to opt out active users from the new topic tool (T317375)]] [production]
20:49 <volans@cumin2002> END (PASS) - Cookbook sre.network.provision (exit_code=0) for device ssw1-a8-codfw.mgmt.codfw.wmnet [production]
20:49 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:920242|Consolidate watchstar icon updating logic under watchstar.js (T336640 T336641)]] (duration: 09m 19s) [production]
20:41 <jdrewniak@deploy1002> jdrewniak: Backport for [[gerrit:920242|Consolidate watchstar icon updating logic under watchstar.js (T336640 T336641)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
20:39 <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:920242|Consolidate watchstar icon updating logic under watchstar.js (T336640 T336641)]] [production]
20:36 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:920240|Ensure mw-watchlink is used for the sticky header watchlink (T336640 T336641)]] (duration: 07m 44s) [production]
20:30 <jdrewniak@deploy1002> jdrewniak: Backport for [[gerrit:920240|Ensure mw-watchlink is used for the sticky header watchlink (T336640 T336641)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet [production]
20:30 <brett> Rolling out maglev LVS scheduler in drmrs (for real this time) - T263797 [production]
20:29 <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:920240|Ensure mw-watchlink is used for the sticky header watchlink (T336640 T336641)]] [production]
19:13 <volans@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:13 <volans@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for ssw1-a8-codfw - volans@cumin2002" [production]
19:12 <volans@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for ssw1-a8-codfw - volans@cumin2002" [production]
19:10 <volans@cumin2002> START - Cookbook sre.dns.netbox [production]
19:10 <volans@cumin2002> START - Cookbook sre.network.provision for device ssw1-a8-codfw.mgmt.codfw.wmnet [production]
19:04 <sukhe> dummry run of authdns-update to confirm new hosts [production]
19:00 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dns2003.wikimedia.org [production]
19:00 <sukhe@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:00 <sukhe@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns2003.wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" [production]
18:59 <sukhe@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dns2003.wikimedia.org decommissioned, removing all IPs except the asset tag one - sukhe@cumin2002" [production]
18:57 <sukhe@cumin2002> START - Cookbook sre.dns.netbox [production]
18:54 <ryankemper@cumin1001> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_codfw [production]
18:53 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_codfw [production]
18:52 <sukhe@cumin2002> START - Cookbook sre.hosts.decommission for hosts dns2003.wikimedia.org [production]
18:50 <ryankemper@puppetmaster1001> conftool action : set/weight=0:pooled=inactive; selector: name=wdqs2022.* [production]
18:50 <ryankemper@puppetmaster1001> conftool action : set/weight=0:pooled=inactive; selector: name=wdqs2021.* [production]
18:50 <volans@cumin2002> END (FAIL) - Cookbook sre.network.provision (exit_code=99) for device ssw1-a8-codfw.mgmt.codfw.wmnet [production]
18:50 <volans@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:50 <volans@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove management record for ssw1-a8-codfw - volans@cumin2002" [production]
18:49 <volans@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove management record for ssw1-a8-codfw - volans@cumin2002" [production]
18:47 <ryankemper> [WDQS] Pooled `wdqs2012` [production]
18:46 <ryankemper> [WDQS] Pooled `wdqs2006` (not sure why was depooled) [production]
18:46 <sukhe> homer "cr*-codfw*" commit "Gerrit: 920363 remove to-be decommissioned host dns2003": T335777 [production]
18:46 <volans@cumin2002> START - Cookbook sre.dns.netbox [production]
18:43 <volans@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:43 <volans@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for ssw1-a8-codfw - volans@cumin2002" [production]
18:42 <volans@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for ssw1-a8-codfw - volans@cumin2002" [production]
18:41 <volans@cumin2002> START - Cookbook sre.dns.netbox [production]
18:41 <volans@cumin2002> START - Cookbook sre.network.provision for device ssw1-a8-codfw.mgmt.codfw.wmnet [production]
18:36 <sukhe> set routing-options static route 208.80.153.231/32 next-hop [ 208.80.153.48 208.80.153.74 208.80.153.107 ]: T326688 [production]
18:34 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.9 refs T330215 [production]
18:28 <sukhe> homer "cr*-codfw*" commit "Gerrit: 920358 add new DNS host dns2006": T326688 [production]
18:22 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns2006.wikimedia.org with OS bullseye [production]
18:05 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns2006.wikimedia.org with reason: host reimage [production]