1-50 of 10000 results (56ms)
2023-02-15 §
23:30 <dduvall@deploy1002> Synchronized php: group1 wikis to 1.40.0-wmf.23 refs T325586 (duration: 06m 43s) [production]
23:23 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.23 refs T325586 [production]
23:15 <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:889608|Change linter maintenance scripts to use existing config varaibles (T329342)]] (duration: 08m 12s) [production]
23:08 <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:889608|Change linter maintenance scripts to use existing config varaibles (T329342)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
23:06 <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:889608|Change linter maintenance scripts to use existing config varaibles (T329342)]] [production]
23:04 <dduvall@deploy1002> Finished scap: Backport for [[gerrit:889607|Bump wikimedia/parsoid to 0.17.0-a16 (T329740)]] (duration: 08m 47s) [production]
23:01 <Amir1> running linter migrate namespace on all wikis (T329764) [production]
22:57 <dduvall@deploy1002> cscott and dduvall: Backport for [[gerrit:889607|Bump wikimedia/parsoid to 0.17.0-a16 (T329740)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet [production]
22:55 <dduvall@deploy1002> Started scap: Backport for [[gerrit:889607|Bump wikimedia/parsoid to 0.17.0-a16 (T329740)]] [production]
22:22 <eevans@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in codfw: maintenance [production]
22:17 <eevans@cumin1001> START - Cookbook sre.discovery.service-route pool sessionstore in codfw: maintenance [production]
21:15 <urandom> rebooting sessionstore2001 w/o cookbook — T327954 [production]
21:10 <dancy@deploy1002> Installation of scap version "4.35.0" completed for 563 hosts [production]
21:09 <dancy@deploy1002> Installing scap version "4.35.0" for 563 hosts [production]
20:39 <urandom> rebooting sessionstore2001 w/o cookbook — T327954 [production]
20:33 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore2001.codfw.wmnet [production]
20:23 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore2001.codfw.wmnet [production]
20:04 <urandom> setting Cassandra query trace probability to 0 (disabled) on sessionstore cluster, codfw datacenter — T327954 [production]
20:02 <eevans@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore2001.codfw.wmnet [production]
19:57 <eevans@cumin1001> START - Cookbook sre.hosts.reboot-single for host sessionstore2001.codfw.wmnet [production]
19:48 <urandom> setting Cassandra query trace probability to 0.25 on sessionstore cluster, codfw datacenter — T327954 [production]
19:42 <eevans@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool sessionstore in codfw: Depooling while we attempt to reproduce errors — T327954 [production]
19:36 <eevans@cumin1001> START - Cookbook sre.discovery.service-route depool sessionstore in codfw: Depooling while we attempt to reproduce errors — T327954 [production]
19:36 <eevans@cumin1001> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check 2 services: maintenance [production]
19:36 <eevans@cumin1001> START - Cookbook sre.discovery.service-route check 2 services: maintenance [production]
19:32 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.40.0-wmf.23" [production]
19:23 <dduvall> rolling back due to spike in parsoid errors (T325586) [production]
19:23 <dduvall@deploy1002> Synchronized php: group1 wikis to 1.40.0-wmf.23 refs T325586 (duration: 06m 36s) [production]
19:18 <dduvall> correction: spike may be temporary. holding (T325586) [production]
19:17 <dduvall> large spike in undefined property errors. rolling back (T325586) [production]
19:16 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.23 refs T325586 [production]
18:41 <akosiaris> dummy entry [production]
18:24 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns4004.wikimedia.org with OS buster [production]
18:12 <moritzm> installing curl security updates on bullseye (not buster) [production]
18:12 <moritzm> installing curl security updates on buster [production]
17:52 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-ui1001.eqiad.wmnet [production]
17:51 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns4004.wikimedia.org with reason: host reimage [production]
17:49 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on dns4004.wikimedia.org with reason: host reimage [production]
17:47 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-test-ui1001.eqiad.wmnet [production]
17:31 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host dns4004.wikimedia.org with OS buster [production]
17:24 <sukhe> [done] disable puppet on A:dns-auth: merging CR 889560 [production]
17:18 <sukhe> disable puppet on A:dns-auth: merging CR 889560 [production]
16:52 <moritzm> installing postgresql-11 security updates [production]
16:45 <jmm@cumin2002> END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling restart_daemons on A:ldap-replicas [production]
16:43 <moritzm> restarting Exim on MXes to pick up gnutls security updates [production]
16:42 <jmm@cumin2002> START - Cookbook sre.ldap.roll-restart-reboot-replica rolling restart_daemons on A:ldap-replicas [production]
16:24 <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-airflow1005.eqiad.wmnet with reason: host reimage [production]
16:21 <bking@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-airflow1005.eqiad.wmnet with reason: host reimage [production]
16:16 <moritzm> installing gnutls28 security updates [production]
16:12 <bking@cumin1001> START - Cookbook sre.ganeti.reimage for host an-airflow1005.eqiad.wmnet with OS buster [production]