|
2021-03-10
ยง
|
| 22:44 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts registry[2001-2002].codfw.wmnet |
[production] |
| 22:40 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry1001.eqiad.wmnet |
[production] |
| 22:32 |
<brennen@deploy1002> |
Synchronized php: group1 wikis to 1.36.0-wmf.34 (duration: 01m 30s) |
[production] |
| 22:30 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.34 |
[production] |
| 22:27 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts registry1001.eqiad.wmnet |
[production] |
| 22:26 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): T277094 believed resolved, promoting to group1. |
[production] |
| 22:25 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670535|Fix client error logging (T277094)]] (duration: 01m 09s) |
[production] |
| 21:53 |
<mutante> |
ferm/iptables docker NAT rules applied by puppet on releases servers after breaking out fules into their own profile class (T276869) |
[production] |
| 21:51 |
<dwisehaupt> |
upgraded mariadb and keeping replication stopped on frdb1002 to start the utf8mb4 table alters under a root screen session |
[production] |
| 21:43 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): client errors may still be missing for group0; continuing to hold for T277094 until we know what's broken. |
[production] |
| 21:40 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670533|Revert "Error in shouldLog logic drops most errors" (T277094)]] (duration: 01m 08s) |
[production] |
| 21:38 |
<dwisehaupt> |
stopping mysql replication on frdev1001 and starting utf8mb4 table alters under a root screen session |
[production] |
| 21:38 |
<dwisehaupt> |
stopping mysql replication on frdb1003 and starting utf8mb4 table alters under a root screen session |
[production] |
| 21:30 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): logstash client error board was set up incorrectly; reverting earlier patch for T277094 and will proceed to group1. |
[production] |
| 21:19 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: cdc47f3e35e815081f787def2d51f3fd337ecf6c: jawiki: Growth features: Add help panel links (T276830) (duration: 01m 08s) |
[production] |
| 21:16 |
<eileen> |
civicrm revision changed from b13e70d968 to 550be50105, config revision is 970b10b0b3 |
[production] |
| 21:13 |
<cdanis@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 21:00 |
<cdanis@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:57 |
<cdanis@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:56 |
<Urbanecm> |
Fixing wrong sync message: urbanecm@deploy1002 Synchronized dblists/growthexperiments.dblist f72c3d6c4fcbda692c5bf8c37a38667c3ba12d80: jawiki: Enable Growth features in stealth mode (T276830) (duration: 01m 08s) |
[production] |
| 20:56 |
<Urbanecm> |
Fixing wrong sync message: urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: f72c3d6c4fcbda692c5bf8c37a38667c3ba12d80: jawiki: Enable Growth features in stealth mode (T276830) (duration: 01m 07s) |
[production] |
| 20:54 |
<urbanecm@deploy1002> |
Synchronized dblists/growthexperiments.dblist: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 08s) |
[production] |
| 20:53 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 07s) |
[production] |
| 20:50 |
<cdanis@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 20:48 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 08s) |
[production] |
| 20:41 |
<brennen@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670529|Error in shouldLog logic drops most errors (T277094)]] (duration: 01m 14s) |
[production] |
| 20:36 |
<cdanis@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 19:58 |
<brennen> |
train status: 1.36.0-wmf.34 (T274938): currently blocked at group0 as client error logging is broken (UBN ticket incoming), will hold for patch. |
[production] |
| 19:37 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: a130e9f2eab6dec12aec4380efdfd6bde1767aeb: Enable Growth features on eowiki in stealth mode (T276123) (duration: 01m 08s) |
[production] |
| 19:35 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:33 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:32 |
<ryankemper> |
T266470 `ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-all' 'sudo enable-puppet "revoking old cert and generating new one with new alt_names - T266470 - root"'` && `ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-all' 'sudo run-puppet-agent'` |
[production] |
| 19:29 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 84271f616081e28e48676a2dd498bd904d5c0b76: Enable DiscussionTools beta features on frwiktionary (T276189) (duration: 01m 09s) |
[production] |
| 19:28 |
<ryankemper> |
T266470 `ryankemper@wdqs1004:~$ sudo enable-puppet "revoking old cert and generating new one with new alt_names - T266470 - root"` && `sudo run-puppet-agent` |
[production] |
| 19:27 |
<ryankemper> |
T266470 `/srv/private` commit SHA for this change is `45852086679616bccb5bba3dd6396082b0f25a3d` |
[production] |
| 19:26 |
<ryankemper> |
T266470 `sudo chown -Rv gitpuppet:gitpuppet /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/` && `sudo chown -v gitpuppet:gitpuppet /srv/private/modules/secret/secrets/ssl/wdqs.discovery.wmnet.key` |
[production] |
| 19:25 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 5093618d5069dd287a4f33c1d49b5e5c8a05a13c: Enable DiscussionTools beta feature for newtopictool on most wikis (T275827) (duration: 01m 08s) |
[production] |
| 19:23 |
<ryankemper> |
T266470 Deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/670562 (copies over new pubkey) |
[production] |
| 19:23 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 4824679d79d462459eba6b77a5af787817f186d2: Disable DiscussionTools Reply Tool A/B test (T276967) (duration: 01m 07s) |
[production] |
| 19:22 |
<urbanecm@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/DiscussionTools/includes/Hooks/HookUtils.php: 9cb48f08f452a124868e1bf9d700a45c1d7255f4: Allow users to continue using reply tool after disabling A/B test (T276967) (duration: 01m 07s) |
[production] |
| 19:20 |
<urbanecm@deploy1002> |
Synchronized php-1.36.0-wmf.33/extensions/DiscussionTools/includes/Hooks/HookUtils.php: 4193ff71df421f2fe2ed3e1f2fa1c54334e722e2: Allow users to continue using reply tool after disabling A/B test (T276967) (duration: 01m 09s) |
[production] |
| 19:18 |
<urbanecm@deploy1002> |
Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: e998086f7cf7839d2c9aa917776509b3198c3142: searchSatisfaction: Allow for async initialisation (T274869) (duration: 01m 08s) |
[production] |
| 19:18 |
<ryankemper> |
T266470 `sudo cergen -c 'wdqs.*' --generate --base-path /srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d` |
[production] |
| 19:17 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1002.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:16 |
<urbanecm@deploy1002> |
Synchronized php-1.36.0-wmf.33/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: d9bad12cdb02e13517cecd1775162fde88af48eb: searchSatisfaction: Allow for async initialisation (T274869) (duration: 01m 08s) |
[production] |
| 19:16 |
<ryankemper> |
T266470 `sudo rm -fv certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.csr.pem certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12 certificates/wdqs.discovery.wmnet/truststore.jks` (full paths not provided to fit the IRC line) |
[production] |
| 19:15 |
<ryankemper> |
T266470 `sudo puppet cert clean wdqs.discovery.wmnet` |
[production] |
| 19:15 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1002.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:14 |
<ryankemper> |
T266470 on `ryankemper@cumin1001`: `sudo -E cumin 'A:wdqs-all' 'sudo disable-puppet "revoking old cert and generating new one with new alt_names - T266470"'` |
[production] |
| 19:14 |
<ryankemper> |
T266470 Temporarily disabling puppet on all `wdqs*` hosts in preparation for `wdqs.discovery.wmnet` certificate revocation |
[production] |