4301-4350 of 10000 results (32ms)
2020-08-17 §
09:21 <jayme@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) [production]
09:18 <_joe_> running a full apt-get upgrade on mw1379-1380 [production]
09:18 <_joe_> re-upgrading imagemagick on mw1378 [production]
09:16 <_joe_> upgrading packages on mw1377 [production]
09:14 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
09:06 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
09:06 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
09:05 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
08:25 <jayme> forcing a puppet run on all mw-api servers in eqiad - T260329 [production]
07:52 <_joe_> repooling mw1382 [production]
07:37 <_joe_> running the same test on mw1382 T260329 [production]
07:34 <_joe_> repooling mw1381 [production]
07:15 <_joe_> running the same test on mw1381 T260329 [production]
07:15 <_joe_> repooled mw1281 [production]
06:26 <_joe_> stop testing on mw1281, T260329 [production]
05:45 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:43 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
05:28 <marostegui> Stop mysql on db1099:3311, db1099:3318 for reimage [production]
05:28 <_joe_> depooling mw1281 for testing for T260329 [production]
05:25 <marostegui> Deploy schema change on db1139:3311 [production]
05:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1099:3311, db1099:3318 for reimage and MCR change', diff saved to https://phabricator.wikimedia.org/P12263 and previous config saved to /var/cache/conftool/dbconfig/20200817-052147-marostegui.json [production]
2020-08-16 §
11:12 <gehel> repooling wdqs1004 - catched up on lag [production]
2020-08-15 §
21:18 <gehel> depooling wdqs1004 and restarting services, will wait to catch up on lag before repooling [production]
2020-08-14 §
19:41 <effie> restart mwdebug1002 [production]
16:58 <cdanis> done deploying 'allow nameservers of Jio AS55836 to skip RPKI validation I9fcff8' to all routers T260449 [production]
16:44 <cdanis> ❌cdanis@cumin1001.eqiad.wmnet ~ 🕧☕ homer 'cr2-esams*' commit 'allow nameservers of Jio AS55836 to skip RPKI validation I9fcff8' [production]
16:39 <cdanis> ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕧☕ homer 'cr1-codfw*' commit 'allow nameservers of Jio AS55836 to skip RPKI validation I9fcff8' [production]
16:36 <cdanis> ❌cdanis@cumin1001.eqiad.wmnet ~ 🕧☕ homer 'cr2-codfw*' commit 'allow nameservers of Jio AS55836 to skip RPKI validation I9fcff8' [production]
02:41 <eileen> tools revision changed from 9a89f45974 to b4ebd1e564 [production]
2020-08-13 §
23:39 <tzatziki> removing 3 files for legal compliance [production]
22:03 <mutante> switching xhgui from tungsten to xhgui1001 - ran puppet on webperf*001 - T180761 T158837 [production]
21:54 <andrew@deploy1001> Finished deploy [horizon/deploy@f3dcb29]: fix proxy in project-local domain --bug T260388 (duration: 03m 53s) [production]
21:50 <andrew@deploy1001> Started deploy [horizon/deploy@f3dcb29]: fix proxy in project-local domain --bug T260388 [production]
21:11 <mutante> rsyncing /var/lib/jenkins from releases1001 to releases1002 and then all other releases* servers. 57GB, overwriting existing data from manual config (T247652) [production]
20:53 <kormat> dropping xhgui.xhgui on m2 [production]
19:35 <thcipriani@deploy1001> Synchronized php-1.36.0-wmf.4/extensions/DiscussionTools: [[gerrit:620030|Revert new reply API (again)]] T259855 (duration: 00m 57s) [production]
18:06 <herron> restarted ES on logstash1010 [production]
18:05 <dpifke@deploy1001> Synchronized wmf-config/ProductionServices.php: Enabling new XHGui backend (T180761) (duration: 00m 56s) [production]
17:16 <hnowlan> deployed ATS and varnish rules to route api.wikimedia.org [production]
16:26 <hnowlan> created api.wikimedia.org [production]
15:49 <hnowlan> moving api-gateway service to state production. critical set to false [production]
15:41 <herron> restart ES on logstash1012 [production]
14:56 <fdans@deploy1001> Finished deploy [analytics/refinery@ba1a439]: Regular analytics weekly train (duration: 11m 34s) [production]
14:45 <ema> repool mw1382 with kernel memory accounting disabled T260281 [production]
14:45 <fdans@deploy1001> Started deploy [analytics/refinery@ba1a439]: Regular analytics weekly train [production]
14:41 <oblivian@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
14:40 <oblivian@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
14:38 <ema> reboot mw1382 with kernel memory accounting disabled T260281 [production]
14:34 <oblivian@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
14:34 <_joe_> rebooting mw1381 with a newer kernel, mw1383 as control with the old kernel T260329 [production]