2751-2800 of 10000 results (36ms)
2021-03-10 ยง
21:38 <dwisehaupt> stopping mysql replication on frdev1001 and starting utf8mb4 table alters under a root screen session [production]
21:38 <dwisehaupt> stopping mysql replication on frdb1003 and starting utf8mb4 table alters under a root screen session [production]
21:30 <brennen> train status: 1.36.0-wmf.34 (T274938): logstash client error board was set up incorrectly; reverting earlier patch for T277094 and will proceed to group1. [production]
21:19 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: cdc47f3e35e815081f787def2d51f3fd337ecf6c: jawiki: Growth features: Add help panel links (T276830) (duration: 01m 08s) [production]
21:16 <eileen> civicrm revision changed from b13e70d968 to 550be50105, config revision is 970b10b0b3 [production]
21:13 <cdanis@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
21:00 <cdanis@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:57 <cdanis@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
20:56 <Urbanecm> Fixing wrong sync message: urbanecm@deploy1002 Synchronized dblists/growthexperiments.dblist f72c3d6c4fcbda692c5bf8c37a38667c3ba12d80: jawiki: Enable Growth features in stealth mode (T276830) (duration: 01m 08s) [production]
20:56 <Urbanecm> Fixing wrong sync message: urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: f72c3d6c4fcbda692c5bf8c37a38667c3ba12d80: jawiki: Enable Growth features in stealth mode (T276830) (duration: 01m 07s) [production]
20:54 <urbanecm@deploy1002> Synchronized dblists/growthexperiments.dblist: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 08s) [production]
20:53 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 07s) [production]
20:50 <cdanis@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:48 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 08s) [production]
20:41 <brennen@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670529|Error in shouldLog logic drops most errors (T277094)]] (duration: 01m 14s) [production]
20:36 <cdanis@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
19:58 <brennen> train status: 1.36.0-wmf.34 (T274938): currently blocked at group0 as client error logging is broken (UBN ticket incoming), will hold for patch. [production]
19:37 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: a130e9f2eab6dec12aec4380efdfd6bde1767aeb: Enable Growth features on eowiki in stealth mode (T276123) (duration: 01m 08s) [production]
19:35 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: REIMAGE [production]
19:33 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: REIMAGE [production]
19:32 <ryankemper> T266470 `ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-all' 'sudo enable-puppet "revoking old cert and generating new one with new alt_names - T266470 - root"'` && `ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-all' 'sudo run-puppet-agent'` [production]
19:29 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 84271f616081e28e48676a2dd498bd904d5c0b76: Enable DiscussionTools beta features on frwiktionary (T276189) (duration: 01m 09s) [production]
19:28 <ryankemper> T266470 `ryankemper@wdqs1004:~$ sudo enable-puppet "revoking old cert and generating new one with new alt_names - T266470 - root"` && `sudo run-puppet-agent` [production]
19:27 <ryankemper> T266470 `/srv/private` commit SHA for this change is `45852086679616bccb5bba3dd6396082b0f25a3d` [production]
19:26 <ryankemper> T266470 `sudo chown -Rv gitpuppet:gitpuppet /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/` && `sudo chown -v gitpuppet:gitpuppet /srv/private/modules/secret/secrets/ssl/wdqs.discovery.wmnet.key` [production]
19:25 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 5093618d5069dd287a4f33c1d49b5e5c8a05a13c: Enable DiscussionTools beta feature for newtopictool on most wikis (T275827) (duration: 01m 08s) [production]
19:23 <ryankemper> T266470 Deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/670562 (copies over new pubkey) [production]
19:23 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 4824679d79d462459eba6b77a5af787817f186d2: Disable DiscussionTools Reply Tool A/B test (T276967) (duration: 01m 07s) [production]
19:22 <urbanecm@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/DiscussionTools/includes/Hooks/HookUtils.php: 9cb48f08f452a124868e1bf9d700a45c1d7255f4: Allow users to continue using reply tool after disabling A/B test (T276967) (duration: 01m 07s) [production]
19:20 <urbanecm@deploy1002> Synchronized php-1.36.0-wmf.33/extensions/DiscussionTools/includes/Hooks/HookUtils.php: 4193ff71df421f2fe2ed3e1f2fa1c54334e722e2: Allow users to continue using reply tool after disabling A/B test (T276967) (duration: 01m 09s) [production]
19:18 <urbanecm@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: e998086f7cf7839d2c9aa917776509b3198c3142: searchSatisfaction: Allow for async initialisation (T274869) (duration: 01m 08s) [production]
19:18 <ryankemper> T266470 `sudo cergen -c 'wdqs.*' --generate --base-path /srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d` [production]
19:17 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1002.eqiad.wmnet with reason: REIMAGE [production]
19:16 <urbanecm@deploy1002> Synchronized php-1.36.0-wmf.33/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: d9bad12cdb02e13517cecd1775162fde88af48eb: searchSatisfaction: Allow for async initialisation (T274869) (duration: 01m 08s) [production]
19:16 <ryankemper> T266470 `sudo rm -fv certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.csr.pem certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12 certificates/wdqs.discovery.wmnet/truststore.jks` (full paths not provided to fit the IRC line) [production]
19:15 <ryankemper> T266470 `sudo puppet cert clean wdqs.discovery.wmnet` [production]
19:15 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1002.eqiad.wmnet with reason: REIMAGE [production]
19:14 <ryankemper> T266470 on `ryankemper@cumin1001`: `sudo -E cumin 'A:wdqs-all' 'sudo disable-puppet "revoking old cert and generating new one with new alt_names - T266470"'` [production]
19:14 <ryankemper> T266470 Temporarily disabling puppet on all `wdqs*` hosts in preparation for `wdqs.discovery.wmnet` certificate revocation [production]
19:06 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: fe99c312b3ce635342cbd690c34e2610184b74b0: Remove unused config for InukaPageView (T265921) (duration: 01m 26s) [production]
18:56 <dwisehaupt> all fundraising servers are now running buster - T254198 [production]
18:37 <mforns@deploy1002> Finished deploy [analytics/refinery@7fbc3c7] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@7fbc3c700ccb3c598690da9a38990ef7cb187656] (duration: 04m 12s) [production]
18:33 <mforns@deploy1002> Started deploy [analytics/refinery@7fbc3c7] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@7fbc3c700ccb3c598690da9a38990ef7cb187656] [production]
18:33 <mforns@deploy1002> Finished deploy [analytics/refinery@7fbc3c7] (thin): Regular analytics weekly train THIN [analytics/refinery@7fbc3c700ccb3c598690da9a38990ef7cb187656] (duration: 00m 07s) [production]
18:33 <mforns@deploy1002> Started deploy [analytics/refinery@7fbc3c7] (thin): Regular analytics weekly train THIN [analytics/refinery@7fbc3c700ccb3c598690da9a38990ef7cb187656] [production]
18:32 <mforns@deploy1002> Finished deploy [analytics/refinery@7fbc3c7]: Regular analytics weekly train [analytics/refinery@7fbc3c700ccb3c598690da9a38990ef7cb187656] (duration: 14m 30s) [production]
18:18 <mforns@deploy1002> Started deploy [analytics/refinery@7fbc3c7]: Regular analytics weekly train [analytics/refinery@7fbc3c700ccb3c598690da9a38990ef7cb187656] [production]
17:48 <mutante> new Wikimedia project language "trv" added - Seediq is an Atayalic language spoken in the mountains of Northern Taiwan by the Seediq and Taroko people. [production]
17:45 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on kafka-logging2003.codfw.wmnet with reason: REIMAGE [production]
17:42 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging2003.codfw.wmnet with reason: REIMAGE [production]