2020-08-17
§
|
09:21 |
<jayme@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) |
[production] |
09:18 |
<_joe_> |
running a full apt-get upgrade on mw1379-1380 |
[production] |
09:18 |
<_joe_> |
re-upgrading imagemagick on mw1378 |
[production] |
09:16 |
<_joe_> |
upgrading packages on mw1377 |
[production] |
09:14 |
<jayme@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
09:06 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
09:06 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
09:05 |
<jayme@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
08:25 |
<jayme> |
forcing a puppet run on all mw-api servers in eqiad - T260329 |
[production] |
07:52 |
<_joe_> |
repooling mw1382 |
[production] |
07:37 |
<_joe_> |
running the same test on mw1382 T260329 |
[production] |
07:34 |
<_joe_> |
repooling mw1381 |
[production] |
07:15 |
<_joe_> |
running the same test on mw1381 T260329 |
[production] |
07:15 |
<_joe_> |
repooled mw1281 |
[production] |
06:26 |
<_joe_> |
stop testing on mw1281, T260329 |
[production] |
05:45 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
05:43 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
05:28 |
<marostegui> |
Stop mysql on db1099:3311, db1099:3318 for reimage |
[production] |
05:28 |
<_joe_> |
depooling mw1281 for testing for T260329 |
[production] |
05:25 |
<marostegui> |
Deploy schema change on db1139:3311 |
[production] |
05:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1099:3311, db1099:3318 for reimage and MCR change', diff saved to https://phabricator.wikimedia.org/P12263 and previous config saved to /var/cache/conftool/dbconfig/20200817-052147-marostegui.json |
[production] |
2020-08-13
§
|
23:39 |
<tzatziki> |
removing 3 files for legal compliance |
[production] |
22:03 |
<mutante> |
switching xhgui from tungsten to xhgui1001 - ran puppet on webperf*001 - T180761 T158837 |
[production] |
21:54 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@f3dcb29]: fix proxy in project-local domain --bug T260388 (duration: 03m 53s) |
[production] |
21:50 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@f3dcb29]: fix proxy in project-local domain --bug T260388 |
[production] |
21:11 |
<mutante> |
rsyncing /var/lib/jenkins from releases1001 to releases1002 and then all other releases* servers. 57GB, overwriting existing data from manual config (T247652) |
[production] |
20:53 |
<kormat> |
dropping xhgui.xhgui on m2 |
[production] |
19:35 |
<thcipriani@deploy1001> |
Synchronized php-1.36.0-wmf.4/extensions/DiscussionTools: [[gerrit:620030|Revert new reply API (again)]] T259855 (duration: 00m 57s) |
[production] |
18:06 |
<herron> |
restarted ES on logstash1010 |
[production] |
18:05 |
<dpifke@deploy1001> |
Synchronized wmf-config/ProductionServices.php: Enabling new XHGui backend (T180761) (duration: 00m 56s) |
[production] |
17:16 |
<hnowlan> |
deployed ATS and varnish rules to route api.wikimedia.org |
[production] |
16:26 |
<hnowlan> |
created api.wikimedia.org |
[production] |
15:49 |
<hnowlan> |
moving api-gateway service to state production. critical set to false |
[production] |
15:41 |
<herron> |
restart ES on logstash1012 |
[production] |
14:56 |
<fdans@deploy1001> |
Finished deploy [analytics/refinery@ba1a439]: Regular analytics weekly train (duration: 11m 34s) |
[production] |
14:45 |
<ema> |
repool mw1382 with kernel memory accounting disabled T260281 |
[production] |
14:45 |
<fdans@deploy1001> |
Started deploy [analytics/refinery@ba1a439]: Regular analytics weekly train |
[production] |
14:41 |
<oblivian@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
14:40 |
<oblivian@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
14:38 |
<ema> |
reboot mw1382 with kernel memory accounting disabled T260281 |
[production] |
14:34 |
<oblivian@cumin1001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
14:34 |
<_joe_> |
rebooting mw1381 with a newer kernel, mw1383 as control with the old kernel T260329 |
[production] |