|
2025-11-20
§
|
| 23:10 |
<swfrench-wmf> |
restarted postfix on mx-in1001, mx-in2001 at ~ 23:00 UTC for config change |
[production] |
| 19:16 |
<brennen@deploy2002> |
rebuilt and synchronized wikiversions files: group2 to 1.46.0-wmf.3 refs T408273 |
[production] |
| 19:04 |
<brennen> |
1.46.0-wmf.3 train status (T408273): no current blockers, rolling to all wikis |
[production] |
| 15:19 |
<ladsgroup@deploy2002> |
sync-world failed: <CalledProcessError> Command '['sudo', '-u', 'mwbuilder', '-n', '--', '/usr/bin/scap', 'mwscript', '--no-local-config', '--directory', '/srv/mediawiki-staging', '--user', 'www-data', '--', 'mergeMessageFileList.php', '--wiki=aawiki', '--force-version', '1.46.0-wmf.3', '--list-file', '/srv/mediawiki-staging/wmf-config/extension-list', '--output', '/tmp/tmp.RV8eoygq6j']' returned |
[production] |
| 01:45 |
<ladsgroup@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1185* gradually with 4 steps - Work done |
[production] |
| 01:14 |
<mwpresync@deploy2002> |
Finished scap build-images: Publishing wmf/next image (duration: 13m 32s) |
[production] |
| 01:00 |
<mwpresync@deploy2002> |
Started scap build-images: Publishing wmf/next image |
[production] |
| 00:59 |
<ladsgroup@cumin1003> |
START - Cookbook sre.mysql.pool db1185* gradually with 4 steps - Work done |
[production] |
|
2025-11-19
§
|
| 19:23 |
<brennen@deploy2002> |
rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.3 refs T408273 |
[production] |
| 19:13 |
<brennen> |
1.46.0-wmf.3 train status (T408273): no current blockers, logs clean, rolling to group1 |
[production] |
| 16:39 |
<sukhe@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 39 hosts with reason: site depool |
[production] |
| 15:22 |
<ladsgroup@deploy2002> |
sync-world failed: <CalledProcessError> Command '['sudo', '-u', 'mwbuilder', '-n', '--', '/usr/bin/scap', 'mwscript', '--no-local-config', '--directory', '/srv/mediawiki-staging', '--user', 'www-data', '--', 'mergeMessageFileList.php', '--wiki=aawiki', '--force-version', '1.46.0-wmf.3', '--list-file', '/srv/mediawiki-staging/wmf-config/extension-list', '--output', '/tmp/tmp.IxZM23pYxK']' returned |
[production] |
| 14:29 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1187 gradually with 4 steps - repool after schema change test |
[production] |
| 13:43 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool db1187 gradually with 4 steps - repool after schema change test |
[production] |
| 13:37 |
<ladsgroup@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1187.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:33 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1187.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:33 |
<marostegui@cumin1003> |
DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 4:00:00 on db1187.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:37 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1189 gradually with 4 steps - Repooling after switchover |
[production] |
| 06:52 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool db1189 gradually with 4 steps - Repooling after switchover |
[production] |
| 06:52 |
<marostegui@cumin1003> |
END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) db1189 gradually with 4 steps - Repooling after switchover |
[production] |
| 06:48 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool db1189 gradually with 4 steps - Repooling after switchover |
[production] |
| 01:14 |
<mwpresync@deploy2002> |
Finished scap build-images: Publishing wmf/next image (duration: 13m 18s) |
[production] |
| 01:00 |
<mwpresync@deploy2002> |
Started scap build-images: Publishing wmf/next image |
[production] |
|
2025-11-18
§
|
| 19:16 |
<swfrench-wmf> |
disable puppet on A:lvs-codfw for pybal config change - T352245 |
[production] |
| 19:14 |
<brennen@deploy2002> |
rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.3 refs T408273 |
[production] |
| 19:02 |
<brennen> |
1.46.0-wmf.3 train status (T408273): no current blockers, rolling to group0 |
[production] |
| 16:55 |
<swfrench-wmf> |
deleted EtcdReplicationDown silence db7447af-851f-4faa-a4fd-b535ee9fbcdb - T352245 |
[production] |
| 16:38 |
<swfrench-wmf> |
transferred etcd-mirror replication back to conf2005 - T352245 |
[production] |
| 16:28 |
<swfrench-wmf> |
restarted navtiming on webperf2003 - T352245 |
[production] |
| 16:25 |
<swfrench-wmf> |
begin rolling restarts of codfw-associated confds - T352245 |
[production] |
| 16:22 |
<swfrench-wmf> |
migrating etcd to PKI certs on conf2005 - T352245 |
[production] |
| 16:10 |
<swfrench-wmf> |
migrating etcd to PKI certs on conf2004 - T352245 |
[production] |
| 15:48 |
<swfrench-wmf> |
transferred etcd-mirror replication to conf2006 - T352245 |
[production] |
| 15:39 |
<swfrench-wmf> |
silenced EtcdReplicationDown db7447af-851f-4faa-a4fd-b535ee9fbcdb - T352245 |
[production] |
| 15:35 |
<swfrench-wmf> |
disable puppet on A:conf-codfw - T352245 |
[production] |
| 14:16 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) es1033 gradually with 4 steps - Repooling after upgrade |
[production] |
| 13:31 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool es1033 gradually with 4 steps - Repooling after upgrade |
[production] |
| 13:31 |
<marostegui@cumin1003> |
END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) es1033 gradually with 4 steps - Repooling after upgrade |
[production] |
| 13:31 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool es1033 gradually with 4 steps - Repooling after upgrade |
[production] |
| 07:13 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1184 gradually with 4 steps - Repooling after switchover |
[production] |
| 06:27 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool db1184 gradually with 4 steps - Repooling after switchover |
[production] |