4051-4100 of 10000 results (28ms)
2021-03-11 §
06:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1136', diff saved to https://phabricator.wikimedia.org/P14749 and previous config saved to /var/cache/conftool/dbconfig/20210311-063814-marostegui.json [production]
06:36 <marostegui> Drop testreduce from m5 - T276787 [production]
05:34 <thcipriani> restarted apache2 on gerrit1001 [production]
00:50 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2219.codfw.wmnet [production]
00:49 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2218.codfw.wmnet [production]
00:49 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2217.codfw.wmnet [production]
00:48 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2216.codfw.wmnet [production]
00:48 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2215.codfw.wmnet [production]
2021-03-10 §
23:49 <mholloway-shell@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/EventLogging: EventLogging: Stream always in sample if the user is in debugMode (T276515) (duration: 01m 23s) [production]
23:41 <dwisehaupt> disabled silverpop daily run in process-control until utf8mb4 conversion completes on frdev1001 [production]
23:12 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: REIMAGE [production]
23:10 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: REIMAGE [production]
23:10 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry1002.eqiad.wmnet [production]
23:01 <legoktm@cumin1001> START - Cookbook sre.hosts.decommission for hosts registry1002.eqiad.wmnet [production]
22:55 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry[2001-2002].codfw.wmnet [production]
22:51 <andrewbogott> updating puppet compiler facts to catch up with a new custom fact [production]
22:44 <legoktm@cumin1001> START - Cookbook sre.hosts.decommission for hosts registry[2001-2002].codfw.wmnet [production]
22:40 <legoktm@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts registry1001.eqiad.wmnet [production]
22:32 <brennen@deploy1002> Synchronized php: group1 wikis to 1.36.0-wmf.34 (duration: 01m 30s) [production]
22:30 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.34 [production]
22:27 <legoktm@cumin1001> START - Cookbook sre.hosts.decommission for hosts registry1001.eqiad.wmnet [production]
22:26 <brennen> train status: 1.36.0-wmf.34 (T274938): T277094 believed resolved, promoting to group1. [production]
22:25 <brennen@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670535|Fix client error logging (T277094)]] (duration: 01m 09s) [production]
21:53 <mutante> ferm/iptables docker NAT rules applied by puppet on releases servers after breaking out fules into their own profile class (T276869) [production]
21:51 <dwisehaupt> upgraded mariadb and keeping replication stopped on frdb1002 to start the utf8mb4 table alters under a root screen session [production]
21:43 <brennen> train status: 1.36.0-wmf.34 (T274938): client errors may still be missing for group0; continuing to hold for T277094 until we know what's broken. [production]
21:40 <brennen@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670533|Revert "Error in shouldLog logic drops most errors" (T277094)]] (duration: 01m 08s) [production]
21:38 <dwisehaupt> stopping mysql replication on frdev1001 and starting utf8mb4 table alters under a root screen session [production]
21:38 <dwisehaupt> stopping mysql replication on frdb1003 and starting utf8mb4 table alters under a root screen session [production]
21:30 <brennen> train status: 1.36.0-wmf.34 (T274938): logstash client error board was set up incorrectly; reverting earlier patch for T277094 and will proceed to group1. [production]
21:19 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: cdc47f3e35e815081f787def2d51f3fd337ecf6c: jawiki: Growth features: Add help panel links (T276830) (duration: 01m 08s) [production]
21:16 <eileen> civicrm revision changed from b13e70d968 to 550be50105, config revision is 970b10b0b3 [production]
21:13 <cdanis@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
21:00 <cdanis@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:57 <cdanis@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
20:56 <Urbanecm> Fixing wrong sync message: urbanecm@deploy1002 Synchronized dblists/growthexperiments.dblist f72c3d6c4fcbda692c5bf8c37a38667c3ba12d80: jawiki: Enable Growth features in stealth mode (T276830) (duration: 01m 08s) [production]
20:56 <Urbanecm> Fixing wrong sync message: urbanecm@deploy1002 Synchronized wmf-config/InitialiseSettings.php: f72c3d6c4fcbda692c5bf8c37a38667c3ba12d80: jawiki: Enable Growth features in stealth mode (T276830) (duration: 01m 07s) [production]
20:54 <urbanecm@deploy1002> Synchronized dblists/growthexperiments.dblist: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 08s) [production]
20:53 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 07s) [production]
20:50 <cdanis@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'canary' . [production]
20:48 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 92ae985df5411de7ff983a778aebde0e10f6253e: thwiki: Make Growth features available to newcomers (T274646) (duration: 01m 08s) [production]
20:41 <brennen@deploy1002> Synchronized php-1.36.0-wmf.34/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: Backport: [[gerrit:670529|Error in shouldLog logic drops most errors (T277094)]] (duration: 01m 14s) [production]
20:36 <cdanis@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
19:58 <brennen> train status: 1.36.0-wmf.34 (T274938): currently blocked at group0 as client error logging is broken (UBN ticket incoming), will hold for patch. [production]
19:37 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: a130e9f2eab6dec12aec4380efdfd6bde1767aeb: Enable Growth features on eowiki in stealth mode (T276123) (duration: 01m 08s) [production]
19:35 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: REIMAGE [production]
19:33 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-logging1003.eqiad.wmnet with reason: REIMAGE [production]
19:32 <ryankemper> T266470 `ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-all' 'sudo enable-puppet "revoking old cert and generating new one with new alt_names - T266470 - root"'` && `ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-all' 'sudo run-puppet-agent'` [production]
19:29 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 84271f616081e28e48676a2dd498bd904d5c0b76: Enable DiscussionTools beta features on frwiktionary (T276189) (duration: 01m 09s) [production]
19:28 <ryankemper> T266470 `ryankemper@wdqs1004:~$ sudo enable-puppet "revoking old cert and generating new one with new alt_names - T266470 - root"` && `sudo run-puppet-agent` [production]