1651-1700 of 10000 results (71ms)
2023-01-26 ยง
14:08 <marostegui@cumin1001> dbctl commit (dc=all): 'db2123 (re)pooling @ 5%: After switchover', diff saved to https://phabricator.wikimedia.org/P43412 and previous config saved to /var/cache/conftool/dbconfig/20230126-140804-root.json [production]
14:07 <lucaswerkmeister-wmde@deploy1002> dreamyjazz and lucaswerkmeister-wmde: Backport for [[gerrit:883122|Enable write new for CheckUserLog comment fields on group 0 and 1 (T233004)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
14:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2123 T328023', diff saved to https://phabricator.wikimedia.org/P43411 and previous config saved to /var/cache/conftool/dbconfig/20230126-140716-root.json [production]
14:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db2113 to s5 primary T328023', diff saved to https://phabricator.wikimedia.org/P43410 and previous config saved to /var/cache/conftool/dbconfig/20230126-140630-root.json [production]
14:06 <marostegui> Starting s5 codfw failover from db2123 to db2113 - T328023 [production]
14:06 <lucaswerkmeister-wmde@deploy1002> Started scap: Backport for [[gerrit:883122|Enable write new for CheckUserLog comment fields on group 0 and 1 (T233004)]] [production]
14:00 <moritzm> restarting etherpad-lite to pick up nodejs security update [production]
13:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove vslow from db2113, future s5 codfw master T328023', diff saved to https://phabricator.wikimedia.org/P43409 and previous config saved to /var/cache/conftool/dbconfig/20230126-135509-marostegui.json [production]
13:52 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s5 T328023 [production]
13:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db2113 with weight 0 T328023', diff saved to https://phabricator.wikimedia.org/P43408 and previous config saved to /var/cache/conftool/dbconfig/20230126-135215-root.json [production]
13:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s5 T328023 [production]
13:45 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
13:45 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
13:44 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
13:38 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:38 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove DNS records for removed esams eqiad GRE tunnel link IPs. - cmooney@cumin1001" [production]
13:37 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove DNS records for removed esams eqiad GRE tunnel link IPs. - cmooney@cumin1001" [production]
13:32 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:883723|Change time zone setting on gorwiktionary (T327986)]] (duration: 12m 02s) [production]
13:32 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
13:25 <moritzm> restarting turnilo for nodejs security update [production]
13:22 <ladsgroup@deploy1002> superpes and ladsgroup: Backport for [[gerrit:883723|Change time zone setting on gorwiktionary (T327986)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
13:20 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:883723|Change time zone setting on gorwiktionary (T327986)]] [production]
13:10 <moritzm> installing nodejs security updates on bullseye [production]
13:09 <hashar> Rebooting gerrit2002.wikimedia.org host to validate Apache 2 services starts AFTER network went online | T326125 [production]
13:06 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
13:04 <btullis@cumin1001> END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) for Hadoop analytics cluster [production]
12:47 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flerovium.eqiad.wmnet [production]
12:43 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on cp3051.esams.wmnet with reason: T323717 [production]
12:42 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on cp3051.esams.wmnet with reason: T323717 [production]
12:42 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3051.esams.wmnet,service=ats-be [production]
12:42 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3051.esams.wmnet,service=cdn [production]
12:41 <sukhe> depool cp3051.esams.wmnet for firmware update testing: T323717 [production]
12:41 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host flerovium.eqiad.wmnet [production]
12:40 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host furud.codfw.wmnet [production]
12:29 <mvernon@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-proxies (exit_code=0) rolling restart_daemons on A:eqiad and not A:thanos-fe and A:swift-fe or A:thanos-fe [production]
12:15 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host furud.codfw.wmnet [production]
12:10 <hnowlan@puppetmaster1001> conftool action : set/pooled=inactive; selector: service=thumbor,name=kubernetes101[0123].eqiad.wmnet [production]
12:10 <mvernon@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-proxies rolling restart_daemons on A:eqiad and not A:thanos-fe and A:swift-fe or A:thanos-fe [production]
12:03 <jbond> enable profile::base::firewall::defs_from_etcd: true globally [production]
11:56 <jbond@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-client-ssl._tcp.wikimedia.org on all recursors [production]
11:56 <jbond@cumin1001> START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.wikimedia.org on all recursors [production]
11:49 <hnowlan@puppetmaster1001> conftool action : set/pooled=inactive; selector: service=thumbor,name=kubernetes1014.eqiad.wmnet [production]
11:49 <hnowlan@puppetmaster1001> conftool action : set/weight=10; selector: service=thumbor,name=kubernetes1010.eqiad.wmnet [production]
11:48 <ayounsi@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts flowspec1001 [production]
11:48 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:48 <ayounsi@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: flowspec1001 decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1001" [production]
11:46 <ayounsi@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: flowspec1001 decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1001" [production]
11:44 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
11:40 <ayounsi@cumin1001> START - Cookbook sre.hosts.decommission for hosts flowspec1001 [production]
11:36 <cgoubert@cumin1001> conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-aux [production]