1701-1750 of 10000 results (35ms)
2024-09-19 §
08:16 <jnuche@deploy1003> rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.23 refs T373642 [production]
00:41 <swfrench-wmf> force-reboot of puppetserver1001 via ipmitool (unresponsive for over 30m) [production]
2024-09-18 §
12:28 <jnuche@deploy1003> rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.23 refs T373642 [production]
08:30 <jnuche@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.23 refs T373642 [production]
08:15 <jnuche@deploy1003> rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.23 refs T373642 [production]
2024-09-17 §
12:50 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2161.codfw.wmnet with reason: Maintenance [production]
12:50 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2161.codfw.wmnet with reason: Maintenance [production]
08:12 <jnuche@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.23 refs T373642 [production]
04:00 <mwpresync@deploy1003> Pruned MediaWiki: 1.43.0-wmf.20 (duration: 00m 58s) [production]
03:53 <mwpresync@deploy1003> Finished scap sync-world: testwikis to 1.43.0-wmf.23 refs T373642 (duration: 50m 47s) [production]
03:02 <mwpresync@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.23 refs T373642 [production]
2024-09-16 §
15:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2238 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69154 and previous config saved to /var/cache/conftool/dbconfig/20240916-155637-arnaudb.json [production]
15:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2237 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69153 and previous config saved to /var/cache/conftool/dbconfig/20240916-155632-arnaudb.json [production]
15:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2227 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69152 and previous config saved to /var/cache/conftool/dbconfig/20240916-155627-arnaudb.json [production]
15:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2225 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69151 and previous config saved to /var/cache/conftool/dbconfig/20240916-155622-arnaudb.json [production]
15:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2224 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69150 and previous config saved to /var/cache/conftool/dbconfig/20240916-155617-arnaudb.json [production]
15:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2222 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69149 and previous config saved to /var/cache/conftool/dbconfig/20240916-155612-arnaudb.json [production]
15:56 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2221 (re)pooling @ 4%: T374623', diff saved to https://phabricator.wikimedia.org/P69148 and previous config saved to /var/cache/conftool/dbconfig/20240916-155607-arnaudb.json [production]
2024-09-13 §
16:57 <swfrench-wmf> running homer cr*codfw* commit 'T372878' [production]
16:38 <swfrench-wmf> running homer lsw1-b3-codfw* commit 'T372878' [production]
10:44 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on db1172.eqiad.wmnet with reason: Depooled recovering replag [production]
10:44 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on db1172.eqiad.wmnet with reason: Depooled recovering replag [production]
2024-09-12 §
19:03 <swfrench-wmf> rebuilt php8.1 production images to pick up php-uuid - T372602 [production]
18:20 <swfrench-wmf> ran systemctl reset-failed mediawiki_job_MachineVision_prioritize_uncategorized.service on mwmaint1002 to clear failed state for turned down job - T352884 [production]
18:18 <dduvall@deploy1003> rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.22 refs T373641 [production]
13:51 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1209.eqiad.wmnet with reason: Maintenance [production]
13:51 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1209.eqiad.wmnet with reason: Maintenance [production]
13:03 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1192.eqiad.wmnet with reason: Maintenance [production]
13:03 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1192.eqiad.wmnet with reason: Maintenance [production]
12:36 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
12:36 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
08:23 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2129 (re)pooling @ 4%: post db2229 bootstrap', diff saved to https://phabricator.wikimedia.org/P69022 and previous config saved to /var/cache/conftool/dbconfig/20240912-082301-arnaudb.json [production]
2024-09-11 §
18:25 <dduvall@deploy1003> rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.22 refs T373641 [production]
17:39 <swfrench-wmf> imported php-uuid_1.2.0-12+wmf11u1 into component/php81 - T372507 [production]
2024-09-10 §
18:38 <swfrench-wmf> ran authdns-update on dns1004 (18:25 UTC) for T372604 [production]
18:14 <dduvall@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.22 refs T373641 [production]
18:04 <swfrench-wmf> ran sre.dns.netbox after adding mwdebug-next LVS VIPs for T372604 [production]
17:39 <tzatziki> removing 4 files for legal compliance [production]
15:02 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2127 (re)pooling @ 4%: post reimage && maintenance repool', diff saved to https://phabricator.wikimedia.org/P68809 and previous config saved to /var/cache/conftool/dbconfig/20240910-150222-arnaudb.json [production]
04:01 <mwpresync@deploy1003> Pruned MediaWiki: 1.43.0-wmf.19 (duration: 00m 58s) [production]
03:47 <mwpresync@deploy1003> Finished scap sync-world: testwikis to 1.43.0-wmf.22 refs T373641 (duration: 45m 06s) [production]
03:02 <mwpresync@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.22 refs T373641 [production]
2024-09-07 §
10:28 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on db1246.eqiad.wmnet with reason: https://phabricator.wikimedia.org/T374215 → server depooled has hardware issues [production]
10:28 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on db1246.eqiad.wmnet with reason: https://phabricator.wikimedia.org/T374215 → server depooled has hardware issues [production]
2024-09-06 §
10:33 <akosiaris@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1246.eqiad.wmnet with reason: Server failed, rebooted in emergency/single user mode [production]
10:32 <akosiaris@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db1246.eqiad.wmnet with reason: Server failed, rebooted in emergency/single user mode [production]
2024-09-05 §
18:37 <dancy@deploy1003> rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.21 refs T373640 [production]
16:38 <dancy@deploy1003> rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.21 refs T373640 [production]
2024-09-04 §
21:03 <dancy@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.21 refs T373640 [production]
17:59 <dancy@deploy1003> Finished scap sync-world: testwikis to 1.43.0-wmf.21 refs T373640 (duration: 06m 48s) [production]