3201-3250 of 10000 results (33ms)
2024-09-30 §
16:27 <swfrench-wmf> imported php8.1_8.1.30-1+wmf11u1 into component/php81 - T376036 [production]
2024-09-26 §
23:41 <swfrench-wmf> silenced KubernetesAPILatency for 24h for resource=~"(blockaffinities|ipamblocks)" site="eqiad" (f33e2c42-d921-4686-a761-59d43ad14d84) - T369744 [production]
19:30 <brennen@deploy2002> rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.24 refs T373643 [production]
17:08 <swfrench-wmf> cleared contents of a stale scap dsh group resource on deploy2002 - T370962 [production]
16:27 <swfrench-wmf> done with switchover day 3 deployment server switch - T370962 [production]
15:04 <swfrench-wmf> running authdns-update for deployment CNAME switch - T370962 [production]
15:01 <swfrench-wmf> starting switchover day 3 deployment server switch - T370962 [production]
2024-09-25 §
18:14 <brennen@deploy1003> rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.24 refs T373643 [production]
18:05 <brennen> 1.43.0-wmf.24 train status: no current blockers, logs clean-ish, rolling to group1 [production]
15:57 <swfrench-wmf> switchover day 2 manual steps are complete - T370962 [production]
14:54 <swfrench-wmf> reset failed units on mwmaint1002 to unblock 01-stop-maintenance - T370962 [production]
13:50 <swfrench-wmf> kartotherian repooled in eqiad due load issues - T370962 [production]
2024-09-24 §
18:21 <brennen@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.24 refs T373643 [production]
17:09 <brennen@deploy1003> Finished scap sync-world: testwikis to 1.43.0-wmf.24 refs T373643 (duration: 27m 45s) [production]
16:41 <brennen@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.24 refs T373643 [production]
16:38 <swfrench-wmf> switched logstash.discovery.wmnet back to eqiad due to reports of stale dashboards - T370962 [production]
16:18 <swfrench-wmf> repooled swift in eqiad to potentially mitigate issues with thumbor and swift - T370962 [production]
16:06 <swfrench-wmf> repooled swift-ro in eqiad to potentially mitigate issues with thumbor - T370962 [production]
15:50 <swfrench-wmf> switchover day 1 actions are complete - T370962 [production]
15:01 <swfrench-wmf> starting switchover day 1 - T370962 [production]
14:21 <swfrench-wmf> scaled up mw-web ahead of traffic+services switchover - T370962 [production]
14:19 <swfrench-wmf> scaled up mw-api-ext ahead of traffic+services switchover - T370962 [production]
10:10 <jnuche@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.24 refs T373643 [production]
08:43 <jnuche@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.24 refs T373643 [production]
08:30 <jnuche@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.24 refs T373643 [production]
04:06 <mwpresync@deploy1003> Pruned MediaWiki: 1.43.0-wmf.21 (duration: 06m 02s) [production]
03:02 <mwpresync@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.24 refs T373643 [production]
2024-09-23 §
09:22 <jayme@cumin1002> conftool action : set/pooled=no; selector: name=registry200(3|4).codfw.wmnet [production]
2024-09-20 §
13:09 <jiji@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2425.codfw.wmnet with reason: reimage [production]
13:09 <jiji@cumin1002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2425.codfw.wmnet with reason: reimage [production]
2024-09-19 §
18:02 <swfrench-wmf> scaling down mw-api-ext in eqiad after pre-switchover testing - T371273 [production]
18:02 <swfrench-wmf> scaling down mw-web in eqiad after pre-switchover testing - T371273 [production]
16:50 <swfrench-wmf> scaling up mw-web in eqiad for pre-switchover testing - T371273 [production]
16:48 <swfrench-wmf> scaling up mw-api-ext in eqiad for pre-switchover testing - T371273 [production]
08:16 <jnuche@deploy1003> rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.23 refs T373642 [production]
00:41 <swfrench-wmf> force-reboot of puppetserver1001 via ipmitool (unresponsive for over 30m) [production]
2024-09-18 §
12:28 <jnuche@deploy1003> rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.23 refs T373642 [production]
08:30 <jnuche@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.23 refs T373642 [production]
08:15 <jnuche@deploy1003> rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.23 refs T373642 [production]
2024-09-17 §
12:50 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2161.codfw.wmnet with reason: Maintenance [production]
12:50 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2161.codfw.wmnet with reason: Maintenance [production]
08:12 <jnuche@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.23 refs T373642 [production]
04:00 <mwpresync@deploy1003> Pruned MediaWiki: 1.43.0-wmf.20 (duration: 00m 58s) [production]
03:53 <mwpresync@deploy1003> Finished scap sync-world: testwikis to 1.43.0-wmf.23 refs T373642 (duration: 50m 47s) [production]
03:02 <mwpresync@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.23 refs T373642 [production]
2024-09-16 §
15:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2238 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69154 and previous config saved to /var/cache/conftool/dbconfig/20240916-155637-arnaudb.json [production]
15:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2237 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69153 and previous config saved to /var/cache/conftool/dbconfig/20240916-155632-arnaudb.json [production]
15:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2227 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69152 and previous config saved to /var/cache/conftool/dbconfig/20240916-155627-arnaudb.json [production]
15:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2225 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69151 and previous config saved to /var/cache/conftool/dbconfig/20240916-155622-arnaudb.json [production]
15:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2224 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69150 and previous config saved to /var/cache/conftool/dbconfig/20240916-155617-arnaudb.json [production]