1-50 of 10000 results (24ms)
2021-03-10 ยง
23:49 <mholloway-shell@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/EventLogging: EventLogging: Stream always in sample if the user is in debugMode (T276515) (duration: 01m 23s) [production]
23:41 <dwisehaupt> disabled silverpop daily run in process-control until utf8mb4 conversion completes on frdev1001 [production]
23:12 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: REIMAGE [production]
23:10 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: REIMAGE [production]
23:10 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry1002.eqiad.wmnet [production]
23:01 <legoktm@cumin1001> START - Cookbook sre.hosts.decommission for hosts registry1002.eqiad.wmnet [production]
22:55 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry[2001-2002].codfw.wmnet [production]
22:51 <andrewbogott> updating puppet compiler facts to catch up with a new custom fact [production]
22:44 <legoktm@cumin1001> START - Cookbook sre.hosts.decommission for hosts registry[2001-2002].codfw.wmnet [production]
22:40 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry1001.eqiad.wmnet [production]
22:32 <brennen@deploy1002> Synchronized php: group1 wikis to 1.36.0-wmf.34 (duration: 01m 30s) [production]
22:30 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.34 [production]
22:27 <legoktm@cumin1001> START - Cookbook sre.hosts.decommission for hosts registry1001.eqiad.wmnet [production]
22:26 <brennen> train status: 1.36.0-wmf.34 (T274938): T277094 believed resolved, promoting to group1. [production]
22:25 <brennen@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670535|Fix client error logging (T277094)]] (duration: 01m 09s) [production]
21:53 <mutante> ferm/iptables docker NAT rules applied by puppet on releases servers after breaking out fules into their own profile class (T276869) [production]
21:51 <dwisehaupt> upgraded mariadb and keeping replication stopped on frdb1002 to start the utf8mb4 table alters under a root screen session [production]
21:43 <brennen> train status: 1.36.0-wmf.34 (T274938): client errors may still be missing for group0; continuing to hold for T277094 until we know what's broken. [production]
21:40 <brennen@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670533|Revert "Error in shouldLog logic drops most errors" (T277094)]] (duration: 01m 08s) [production]
21:38 <dwisehaupt> stopping mysql replication on frdev1001 and starting utf8mb4 table alters under a root screen session [production]
21:38 <dwisehaupt> stopping mysql replication on frdb1003 and starting utf8mb4 table alters under a root screen session [production]
21:30 <brennen> train status: 1.36.0-wmf.34 (T274938): logstash client error board was set up incorrectly; reverting earlier patch for T277094 and will proceed to group1. [production]
21:19 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: cdc47f3e35e815081f787def2d51f3fd337ecf6c: jawiki: Growth features: Add help panel links (T276830) (duration: 01m 08s) [production]
21:16 <eileen> civicrm revision changed from b13e70d968 to 550be50105, config revision is 970b10b0b3 [production]
21:13 <cdanis@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
21:00 <cdanis@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:57 <cdanis@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
20:56 <Urbanecm> Fixing wrong sync message: urbanecm@deploy1002 Synchronized dblists/growthexperiments.dblist f72c3d6c4fcbda692c5bf8c37a38667c3ba12d80: jawiki: Enable Growth features in stealth mode (T276830) (duration: 01m 08s) [production]
20:56 <Urbanecm> Fixing wrong sync message: urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: f72c3d6c4fcbda692c5bf8c37a38667c3ba12d80: jawiki: Enable Growth features in stealth mode (T276830) (duration: 01m 07s) [production]
20:54 <urbanecm@deploy1002> Synchronized dblists/growthexperiments.dblist: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 08s) [production]
20:53 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 07s) [production]
20:50 <cdanis@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:48 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 08s) [production]
20:41 <brennen@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670529|Error in shouldLog logic drops most errors (T277094)]] (duration: 01m 14s) [production]
20:36 <cdanis@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
19:58 <brennen> train status: 1.36.0-wmf.34 (T274938): currently blocked at group0 as client error logging is broken (UBN ticket incoming), will hold for patch. [production]
19:37 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: a130e9f2eab6dec12aec4380efdfd6bde1767aeb: Enable Growth features on eowiki in stealth mode (T276123) (duration: 01m 08s) [production]
19:35 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: REIMAGE [production]
19:33 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: REIMAGE [production]
19:32 <ryankemper> T266470 `ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-all' 'sudo enable-puppet "revoking old cert and generating new one with new alt_names - T266470 - root"'` && `ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-all' 'sudo run-puppet-agent'` [production]
19:29 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 84271f616081e28e48676a2dd498bd904d5c0b76: Enable DiscussionTools beta features on frwiktionary (T276189) (duration: 01m 09s) [production]
19:28 <ryankemper> T266470 `ryankemper@wdqs1004:~$ sudo enable-puppet "revoking old cert and generating new one with new alt_names - T266470 - root"` && `sudo run-puppet-agent` [production]
19:27 <ryankemper> T266470 `/srv/private` commit SHA for this change is `45852086679616bccb5bba3dd6396082b0f25a3d` [production]
19:26 <ryankemper> T266470 `sudo chown -Rv gitpuppet:gitpuppet /srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/` && `sudo chown -v gitpuppet:gitpuppet /srv/private/modules/secret/secrets/ssl/wdqs.discovery.wmnet.key` [production]
19:25 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 5093618d5069dd287a4f33c1d49b5e5c8a05a13c: Enable DiscussionTools beta feature for newtopictool on most wikis (T275827) (duration: 01m 08s) [production]
19:23 <ryankemper> T266470 Deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/670562 (copies over new pubkey) [production]
19:23 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 4824679d79d462459eba6b77a5af787817f186d2: Disable DiscussionTools Reply Tool A/B test (T276967) (duration: 01m 07s) [production]
19:22 <urbanecm@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/DiscussionTools/includes/Hooks/HookUtils.php: 9cb48f08f452a124868e1bf9d700a45c1d7255f4: Allow users to continue using reply tool after disabling A/B test (T276967) (duration: 01m 07s) [production]
19:20 <urbanecm@deploy1002> Synchronized php-1.36.0-wmf.33/extensions/DiscussionTools/includes/Hooks/HookUtils.php: 4193ff71df421f2fe2ed3e1f2fa1c54334e722e2: Allow users to continue using reply tool after disabling A/B test (T276967) (duration: 01m 09s) [production]
19:18 <urbanecm@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: e998086f7cf7839d2c9aa917776509b3198c3142: searchSatisfaction: Allow for async initialisation (T274869) (duration: 01m 08s) [production]