|
2020-02-14
§
|
| 17:37 |
<jforrester@deploy1001> |
Unlocked for deployment [ALL REPOSITORIES]: Testing T245062 fix on mwdebug1001 (duration: 03m 05s) |
[production] |
| 17:33 |
<jforrester@deploy1001> |
Locking from deployment [ALL REPOSITORIES]: Testing T245062 fix on mwdebug1001 (planned duration: 60m 00s) |
[production] |
| 16:11 |
<moritzm> |
installing git-lfs updates from Buster 10.3 point update |
[production] |
| 15:55 |
<moritzm> |
uploaded pypuppetdb 0.3.3-2~wmf+deb10u1 to apt.wikimedia.org |
[production] |
| 15:55 |
<bblack> |
(log(n)) |
[production] |
| 15:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2086:3318 T239453', diff saved to https://phabricator.wikimedia.org/P10414 and previous config saved to /var/cache/conftool/dbconfig/20200214-155443-marostegui.json |
[production] |
| 15:52 |
<moritzm> |
uploaded pypuppetdb 0.3.3-2~wmf+deb9u1 to apt.wikimedia.org |
[production] |
| 15:46 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Resync initialisesetting to try and pick up previoiusly deployed cirrus query routing changes (duration: 01m 05s) |
[production] |
| 15:42 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:42 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:32 |
<effie> |
restart mc-gp* for updates |
[production] |
| 15:17 |
<bd808> |
Toil reduction: !log messages now work from the SRE team's Freenode channel. |
[production] |
| 13:50 |
<gehel> |
restart relforge for JVM upgrade - T245120 |
[production] |
| 10:35 |
<vgutierrez> |
revert ats 8.0.6-rc0 experiment on cp40[26,32] |
[production] |
| 10:14 |
<vgutierrez> |
rolling restart of ats-be to enable TLSv1.3 against origin servers - T170567 |
[production] |
| 09:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1107 after 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10409 and previous config saved to /var/cache/conftool/dbconfig/20200214-093456-marostegui.json |
[production] |
| 09:32 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:32 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:25 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:25 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:25 |
<volans> |
manually absented /usr/local/bin/apt2xml on the 5 hosts with puppet disabled |
[production] |
| 09:15 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:15 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:12 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:12 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:05 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:05 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:05 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:05 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:05 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:05 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:46 |
<moritzm> |
installing 4.19.98 kernel update on Buster systems |
[production] |
| 08:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1107 with weight 100 for 10.4 testing - T242702', diff saved to https://phabricator.wikimedia.org/P10408 and previous config saved to /var/cache/conftool/dbconfig/20200214-080600-marostegui.json |
[production] |
| 06:51 |
<vgutierrez> |
updating puppet compiler facts |
[production] |
| 01:27 |
<dpifke@deploy1001> |
Finished deploy [performance/navtiming@2eec00a]: (no justification provided) (duration: 00m 05s) |
[production] |
| 01:27 |
<dpifke@deploy1001> |
Started deploy [performance/navtiming@2eec00a]: (no justification provided) |
[production] |
| 00:53 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T245202 cirrus: Move all move_like traffic to codfw (duration: 01m 02s) |
[production] |
| 00:51 |
<jforrester@deploy1001> |
Synchronized wmf-config/PoolCounterSettings.php: T245202 cirrus: Increase the pool counter limits a bit (duration: 01m 05s) |
[production] |
|
2020-02-13
§
|
| 22:13 |
<jeh> |
running filesystem tests on cloudvirt1024 T241884 |
[production] |
| 21:42 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 21:41 |
<otto@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 21:40 |
<jbond42> |
refresh facts on compilers |
[production] |
| 21:38 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 21:37 |
<otto@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 21:35 |
<ottomata> |
deploying production and canary releases for eventgate-logging-external (and destroying the 'logging-external' release) (safe because eventgate-logging-external is not in use) - T245203 |
[production] |
| 21:29 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . |
[production] |
| 21:28 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . |
[production] |
| 20:33 |
<marxarelli> |
rollback to group1 due to 500 spike (2k/min) (T233867) |
[production] |
| 20:32 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: (no justification provided) |
[production] |
| 20:30 |
<marxarelli> |
varnish 500 spike. rolling back |
[production] |