2701-2750 of 10000 results (35ms)
2021-09-28 ยง
21:56 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2415.codfw.wmnet with reason: REIMAGE [production]
21:52 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2414.codfw.wmnet with reason: REIMAGE [production]
21:49 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2414.codfw.wmnet with reason: REIMAGE [production]
21:22 <ryankemper> T280247 Puppet run complete on all of `cp-text`, trafficserver backend work is done [production]
21:22 <pt1979@cumin1001> END (PASS) - Cookbook sre.experimental.reimage (exit_code=0) for host thumbor2005.codfw.wmnet [production]
21:19 <bd808> bd808@mwmaint1002 echo "https://toolhub.wikimedia.org/static/js/chunk-vendors.js" | mwscript purgeList.php [production]
21:17 <topranks> Configure cr2-esams for NaWas BGP peering to gateway-1 IPv6 and gateway-2 (T288505) [production]
21:11 <topranks> Configure cr2-esams for NaWas BGP peering to gateway-1 IPv4 (T288505) [production]
21:10 <ryankemper> T280247 `ryankemper@cumin1001:~$ sudo cumin -b 5 'A:cp-text' 'sudo run-puppet-agent --force'` [production]
21:09 <ryankemper> T280247 `ryankemper@cp1075:~$ sudo grep commons-query /etc/trafficserver/remap.config` shows `map http://commons-query.wikimedia.org https://wcqs.discovery.wmnet`; proceeding to rest of fleet in batches of 5 [production]
21:08 <pt1979@cumin1001> START - Cookbook sre.experimental.reimage for host thumbor2005.codfw.wmnet [production]
21:07 <ryankemper> T280247 Running on single cp-text host: `ryankemper@cp1075:~$ sudo run-puppet-agent --force` [production]
21:05 <ryankemper> T280247 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/720078 [production]
21:03 <ryankemper> T280247 `ryankemper@cumin1001:~$ sudo cumin 'A:cp-text' 'sudo disable-puppet "Add trafficserver backend mapping for commons-query.wikimedia.org - T280247"'` [production]
21:02 <legoktm> legoktm@deploy1002:~$ echo "https://toolhub.wikimedia.org/" | mwscript purgeList.php [production]
20:52 <otto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'canary' . [production]
20:52 <otto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' . [production]
20:51 <ryankemper> T280247 Puppet successfully ran on all `w*qs*` hosts; GUI working as before for WDQS, and WCQS seems fine as well. Deploy succeeded without any hitches [production]
20:49 <legoktm> re-enabling and running puppet on A:cp-text: sudo cumin -b 5 A:cp-text 'enable-puppet --force && run-puppet-agent' [production]
20:49 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
20:49 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
20:41 <legoktm> disabling puppet on A:cp-text in preparation for adding toolhub [production]
20:38 <ryankemper> T280247 `ryankemper@cumin1001:~$ sudo cumin -b 5 'P{w*qs*}' 'sudo run-puppet-agent --force'`; 25 hosts total so will take 5 iterations [production]
20:37 <ryankemper> T280247 Test queries on `wdqs1003` passed (tunneled into `wdqs1003`), proceeding to rest of fleet [production]
20:37 <ryankemper> T280247 Ran on wdqs canary `wdqs1003`: `ryankemper@wdqs1003:~$ sudo run-puppet-agent --force` [production]
20:33 <ryankemper> T280247 Running on single wcqs hosts: `ryankemper@wcqs1001:~$ sudo run-puppet-agent --force` [production]
20:33 <ryankemper> T280247 `ryankemper@cumin1001` -> `sudo cumin 'P{w*qs*}' 'sudo disable-puppet "Make query_service nginx proxy to GUI microsite - T280247"'` [production]
20:33 <topranks> Adding IPv6 address to NaWas sub-interfaceon cr2-esams (AMS-IX) - T288505 [production]
19:48 <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.2 refs T281166 [production]
19:35 <legoktm@deploy1002> Synchronized private/PrivateSettings.php: Use IPUtils instead of removed IP class (T292010) (duration: 01m 09s) [production]
19:27 <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: Revert "group0 wikis to 1.38.0-wmf.1" [production]
19:08 <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.2 refs T281166 [production]
19:05 <legoktm@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=toolhub [production]
19:04 <legoktm> adding toolhub to discovery DNS (T280881) [production]
19:00 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@0a38bc5]: tegola: use eqiad discovery endpoin (duration: 00m 20s) [production]
19:00 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@0a38bc5]: tegola: use eqiad discovery endpoin [production]
18:54 <ryankemper> T280001 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/721600 (add wcqs scap dsh groups), running puppet on scap::dsh hosts: `ryankemper@cumin1001:~$ sudo cumin 'P:scap::dsh' 'sudo run-puppet-agent'` [production]
18:45 <jhuneidi@deploy1002> Finished scap: testwikis wikis to 1.38.0-wmf.2 refs T281166 (duration: 49m 27s) [production]
18:21 <pt1979@cumin1001> END (FAIL) - Cookbook sre.experimental.reimage (exit_code=99) for host thumbor2005.codfw.wmnet [production]
18:18 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on puppetmaster1005.eqiad.wmnet with reason: REIMAGE [production]
18:18 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@0a38bc5]: tegola: use eqiad discovery endpoin (duration: 00m 08s) [production]
18:18 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@0a38bc5]: tegola: use eqiad discovery endpoin [production]
18:16 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on puppetmaster1005.eqiad.wmnet with reason: REIMAGE [production]
18:14 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on puppetmaster1004.eqiad.wmnet with reason: REIMAGE [production]
18:12 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on puppetmaster1004.eqiad.wmnet with reason: REIMAGE [production]
18:02 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
18:02 <otto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
18:01 <pt1979@cumin1001> START - Cookbook sre.experimental.reimage for host thumbor2005.codfw.wmnet [production]
18:00 <otto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
17:57 <otto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]