901-950 of 10000 results (34ms)
2021-03-11 ยง
21:28 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2222.codfw.wmnet [production]
21:27 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2223.codfw.wmnet [production]
21:27 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2221.codfw.wmnet [production]
21:27 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw2220.codfw.wmnet [production]
21:21 <brennen@deploy1002> rebuilt and synchronized wikiversions files: Revert "all wikis to 1.36.0-wmf.34" [production]
21:20 <brennen> train status: 1.36.0-wmf.34 (T274938): rolling back to group1 and marking T277229 a train blocker [production]
21:17 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1003.eqiad.wmnet with reason: REIMAGE [production]
21:15 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on backup1003.eqiad.wmnet with reason: REIMAGE [production]
21:14 <tgr@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:670858|Enable GrowthExperiments link recommendations on testwiki (T277173)] (duration: 00m 59s) [production]
21:13 <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@3810277]: T273847 export queries to relforge dag deployment - correct start date (duration: 01m 53s) [production]
21:12 <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@3810277]: T273847 export queries to relforge dag deployment - correct start date [production]
21:05 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw2216.codfw.wmnet [production]
21:04 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts mw2215.codfw.wmnet [production]
21:03 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
21:03 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
21:03 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission for hosts mw2215.codfw.wmnet [production]
21:00 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2216.codfw.wmnet with reason: decom [production]
21:00 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2216.codfw.wmnet with reason: decom [production]
21:00 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on mw2215.codfw.wmnet with reason: decom [production]
21:00 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on mw2215.codfw.wmnet with reason: decom [production]
21:00 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
21:00 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
20:58 <mutante> deactivating codfw API canaries on old hardware (T277119) [production]
20:57 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw2216.codfw.wmnet [production]
20:57 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw2215.codfw.wmnet [production]
20:50 <otto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
20:46 <zpapierski@deploy1002> Finished deploy [wikimedia/discovery/analytics@cc478d4]: T273847 export queries to relforge dag deployment (duration: 02m 09s) [production]
20:44 <zpapierski@deploy1002> Started deploy [wikimedia/discovery/analytics@cc478d4]: T273847 export queries to relforge dag deployment [production]
20:35 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' . [production]
20:33 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' . [production]
20:28 <otto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventstreams-internal' for release 'main' . [production]
20:20 <mutante> phab1001 - systemctl start phabricator_clean_tmp_files - now Succeeded [production]
20:17 <razzi@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host matomo1002.eqiad.wmnet [production]
20:13 <razzi@cumin1001> START - Cookbook sre.hosts.reboot-single for host matomo1002.eqiad.wmnet [production]
20:04 <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.34 [production]
19:59 <mutante> phab1001 - sudo systemctl start phabricator_clean_tmp_files (manually run after conversion from cron to timer, and it fails with permission issues) [production]
19:55 <tgr_> T277173 running mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=testwiki GrowthExperiments [production]
19:54 <tgr@deploy1002> Synchronized wmf-config/: Config: [[gerrit:670857|Configure GrowthExperiments Add Link settings, step 2 (T277173)]] (duration: 01m 08s) [production]
19:43 <robh@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:30 <tgr@deploy1002> Synchronized wmf-config/: Config: [[gerrit:670887|Configure GrowthExperiments Add Link settings, step 1 (T277173)]] (duration: 01m 08s) [production]
19:18 <tgr@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:668196|wikitech: enable BetaFeatures (T125941)]] (duration: 01m 08s) [production]
19:13 <hnowlan@deploy1002> Finished deploy [restbase/deploy@6f0fe23]: Remove internal ratelimits that were causing service proxy issues (duration: 16m 25s) [production]
18:56 <hnowlan@deploy1002> Started deploy [restbase/deploy@6f0fe23]: Remove internal ratelimits that were causing service proxy issues [production]
18:47 <tgr_> running mwscript extensions/GrowthExperiments/maintenance/importOresTopics.php testwiki --count 1000 --verbose --wikiId enwiki --apiUrl 'https://en.wikipedia.org/w/api.php' [production]
17:31 <effie> install mecached 1.6.6-1 on mwdebug1001 [production]
16:26 <effie> upgrade memcached on mc1021, mc2021 [production]
16:20 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:14 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db1110 (re)pooling @ 100%: Repool db1110 after schema change', diff saved to https://phabricator.wikimedia.org/P14802 and previous config saved to /var/cache/conftool/dbconfig/20210311-161138-root.json [production]
15:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db1110 (re)pooling @ 60%: Repool db1110 after schema change', diff saved to https://phabricator.wikimedia.org/P14801 and previous config saved to /var/cache/conftool/dbconfig/20210311-155635-root.json [production]