5901-5950 of 10000 results (94ms)
2023-02-27 §
11:07 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye [production]
11:05 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye [production]
11:04 <jbond@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1003'] [production]
10:59 <jbond@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cloudcephosd1003'] [production]
10:54 <jbond@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1003'] [production]
10:48 <elukey@deploy1002> helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [production]
10:48 <elukey@deploy1002> helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [production]
10:43 <root@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudcephosd1003'] [production]
10:39 <root@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1003'] [production]
10:32 <marostegui> Restart eqiad sanitarium hosts T330502 [production]
10:31 <root@cumin1001> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcephosd1003'] [production]
10:26 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye [production]
10:26 <marostegui> Restart codfw sanitarium hosts T330502 [production]
10:23 <root@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcephosd1003'] [production]
10:23 <dcaro@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1003.eqiad.wmnet with OS bullseye [production]
10:22 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=0) [production]
10:19 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches [production]
10:19 <claime> live testing cache warmup cookbook [production]
10:18 <eoghan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab1004.wikimedia.org with reason: Running failover to gitlab2002- T329931 [production]
10:17 <eoghan@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab1004.wikimedia.org with reason: Running failover to gitlab2002- T329931 [production]
10:08 <marostegui> Enable replication codfw -> eqiad on s4 T330619 [production]
09:47 <dcaro@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcephosd1003.eqiad.wmnet with OS bullseye [production]
09:46 <dcaro@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1003.eqiad.wmnet with OS bullseye [production]
09:46 <dcaro@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcephosd1003.eqiad.wmnet with OS bullseye [production]
09:44 <marostegui> Enable replication codfw -> eqiad on s1 T330619 [production]
09:39 <marostegui> Enable replication codfw -> eqiad on s5 T330619 [production]
09:36 <dcaro@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1003.eqiad.wmnet with OS bullseye [production]
09:36 <dcaro@cumin1001> START - Cookbook sre.hosts.reimage for host cloudcephosd1003.eqiad.wmnet with OS bullseye [production]
09:34 <marostegui> Enable replication codfw -> eqiad on s6 T330619 [production]
09:32 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye [production]
09:27 <marostegui> Enable replication codfw -> eqiad on s7 T330619 [production]
09:26 <marostegui> Enable replication codfw -> eqiad on s8 T330619 [production]
09:20 <hashar> Restarting CI Jenkins T330045 [production]
09:19 <elukey@cumin1001> START - Cookbook sre.hosts.reimage for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye [production]
09:16 <hashar@deploy1002> Finished deploy [releng/jenkins-deploy@0e465ac] (releasing): (no justification provided) (duration: 00m 46s) [production]
09:15 <hashar@deploy1002> Started deploy [releng/jenkins-deploy@0e465ac] (releasing): (no justification provided) [production]
09:12 <marostegui> Enable replication codfw -> eqiad on s3 T330619 [production]
08:56 <moritzm> updating mw/codfw to PHP 1:7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u2 T330270 [production]
08:54 <vgutierrez> test haproxy hardening in cp4045 - T323944 [production]
08:51 <marostegui> Disable GTID on es% x1 and s% on codfw masters T330619 [production]
08:51 <marostegui> Enable replication codfw -> eqiad on s2 T330619 [production]
08:32 <marostegui> Enable replication codfw -> eqiad on es4 and es5 T330619 [production]
07:57 <marostegui> Enable replication codfw -> eqiad on x1 T330619 [production]
07:42 <marostegui> Enable replication codfw -> eqiad on pcX T330619 [production]
2023-02-26 §
02:07 <Amir1> foreachwikiindblist s5 maintenance/migrateExternallinks.php --batch-size=100 --sleep 1 (T326314) [production]
2023-02-25 §
15:30 <apergos> resized lvm and filesystem for /data on dumpsdata1004,5,7; was <100G, now is 38T usable (left some room for growth later) [production]
11:03 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on dse-k8s-worker[1001-1004,1007].eqiad.wmnet with reason: Downtime DSE workers for cluster upgrade [production]
11:02 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on dse-k8s-worker[1001-1004,1007].eqiad.wmnet with reason: Downtime DSE workers for cluster upgrade [production]
09:38 <elukey> delete knative pods on ml-serve-codfw to clear latency alerts [production]
2023-02-24 §
23:15 <mutante> people2002 - for each user who has a public_html dir that is not empty (for pubdir in $(find . -name public_html -type d -not -empty); ..); rsync it from people1003 with --delete (rsync -avp rsync://people1003.eqiad.wmnet/people-home/${pubdiruser}/public_html/ /home/${pubdiruser}/public_html/); T330091 [production]