1001-1050 of 10000 results (72ms)
2023-01-26 ยง
14:31 <ladsgroup@deploy1002> Synchronized private/PrivateSettings.php: Rotating wikiadmin password (T326802) (duration: 07m 04s) [production]
14:27 <moritzm> installing containerd security updates [production]
14:23 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes; selector: service=thumbor,name=kubernetes101[0123].eqiad.wmnet [production]
14:23 <marostegui@cumin1001> dbctl commit (dc=all): 'db2123 (re)pooling @ 10%: After switchover', diff saved to https://phabricator.wikimedia.org/P43413 and previous config saved to /var/cache/conftool/dbconfig/20230126-142309-root.json [production]
14:16 <Lucas_WMDE> UTC afternoon backport+config window done [production]
14:15 <lucaswerkmeister-wmde@deploy1002> Finished scap: Backport for [[gerrit:883122|Enable write new for CheckUserLog comment fields on group 0 and 1 (T233004)]] (duration: 09m 16s) [production]
14:11 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [production]
14:11 <jbond> disable puppet fleet wide to role out etcd ferm change gerrit:883888 [production]
14:11 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [production]
14:09 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [production]
14:08 <marostegui@cumin1001> dbctl commit (dc=all): 'db2123 (re)pooling @ 5%: After switchover', diff saved to https://phabricator.wikimedia.org/P43412 and previous config saved to /var/cache/conftool/dbconfig/20230126-140804-root.json [production]
14:07 <lucaswerkmeister-wmde@deploy1002> dreamyjazz and lucaswerkmeister-wmde: Backport for [[gerrit:883122|Enable write new for CheckUserLog comment fields on group 0 and 1 (T233004)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
14:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2123 T328023', diff saved to https://phabricator.wikimedia.org/P43411 and previous config saved to /var/cache/conftool/dbconfig/20230126-140716-root.json [production]
14:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db2113 to s5 primary T328023', diff saved to https://phabricator.wikimedia.org/P43410 and previous config saved to /var/cache/conftool/dbconfig/20230126-140630-root.json [production]
14:06 <marostegui> Starting s5 codfw failover from db2123 to db2113 - T328023 [production]
14:06 <lucaswerkmeister-wmde@deploy1002> Started scap: Backport for [[gerrit:883122|Enable write new for CheckUserLog comment fields on group 0 and 1 (T233004)]] [production]
14:00 <moritzm> restarting etherpad-lite to pick up nodejs security update [production]
13:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove vslow from db2113, future s5 codfw master T328023', diff saved to https://phabricator.wikimedia.org/P43409 and previous config saved to /var/cache/conftool/dbconfig/20230126-135509-marostegui.json [production]
13:52 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s5 T328023 [production]
13:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db2113 with weight 0 T328023', diff saved to https://phabricator.wikimedia.org/P43408 and previous config saved to /var/cache/conftool/dbconfig/20230126-135215-root.json [production]
13:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s5 T328023 [production]
13:45 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [production]
13:45 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [production]
13:44 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [production]
13:38 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:38 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove DNS records for removed esams eqiad GRE tunnel link IPs. - cmooney@cumin1001" [production]
13:37 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove DNS records for removed esams eqiad GRE tunnel link IPs. - cmooney@cumin1001" [production]
13:32 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:883723|Change time zone setting on gorwiktionary (T327986)]] (duration: 12m 02s) [production]
13:32 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
13:25 <moritzm> restarting turnilo for nodejs security update [production]
13:22 <ladsgroup@deploy1002> superpes and ladsgroup: Backport for [[gerrit:883723|Change time zone setting on gorwiktionary (T327986)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet [production]
13:20 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:883723|Change time zone setting on gorwiktionary (T327986)]] [production]
13:10 <moritzm> installing nodejs security updates on bullseye [production]
13:09 <hashar> Rebooting gerrit2002.wikimedia.org host to validate Apache 2 services starts AFTER network went online | T326125 [production]
13:06 <elukey@deploy1002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [production]
13:04 <btullis@cumin1001> END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) for Hadoop analytics cluster [production]
12:47 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flerovium.eqiad.wmnet [production]
12:43 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on cp3051.esams.wmnet with reason: T323717 [production]
12:42 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on cp3051.esams.wmnet with reason: T323717 [production]
12:42 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3051.esams.wmnet,service=ats-be [production]
12:42 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3051.esams.wmnet,service=cdn [production]
12:41 <sukhe> depool cp3051.esams.wmnet for firmware update testing: T323717 [production]
12:41 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host flerovium.eqiad.wmnet [production]
12:40 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host furud.codfw.wmnet [production]
12:29 <mvernon@cumin2002> END (PASS) - Cookbook sre.swift.roll-restart-reboot-proxies (exit_code=0) rolling restart_daemons on A:eqiad and not A:thanos-fe and A:swift-fe or A:thanos-fe [production]
12:15 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host furud.codfw.wmnet [production]
12:10 <hnowlan@puppetmaster1001> conftool action : set/pooled=inactive; selector: service=thumbor,name=kubernetes101[0123].eqiad.wmnet [production]
12:10 <mvernon@cumin2002> START - Cookbook sre.swift.roll-restart-reboot-proxies rolling restart_daemons on A:eqiad and not A:thanos-fe and A:swift-fe or A:thanos-fe [production]
12:03 <jbond> enable profile::base::firewall::defs_from_etcd: true globally [production]
11:56 <jbond@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-client-ssl._tcp.wikimedia.org on all recursors [production]