|
2020-08-25
ยง
|
| 14:10 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@7a3221d]: add hostname checking --bug T207538 (duration: 03m 50s) |
[production] |
| 14:06 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@7a3221d]: add hostname checking --bug T207538 |
[production] |
| 13:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1114 for MCR change', diff saved to https://phabricator.wikimedia.org/P12347 and previous config saved to /var/cache/conftool/dbconfig/20200825-135248-marostegui.json |
[production] |
| 13:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'fully repool db1111 MCR changes', diff saved to https://phabricator.wikimedia.org/P12346 and previous config saved to /var/cache/conftool/dbconfig/20200825-134736-marostegui.json |
[production] |
| 13:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1111 MCR changes', diff saved to https://phabricator.wikimedia.org/P12345 and previous config saved to /var/cache/conftool/dbconfig/20200825-133734-marostegui.json |
[production] |
| 13:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1111 MCR changes', diff saved to https://phabricator.wikimedia.org/P12344 and previous config saved to /var/cache/conftool/dbconfig/20200825-132027-marostegui.json |
[production] |
| 13:17 |
<moritzm> |
installing firejail security updates on remaining mw* servers in eqiad |
[production] |
| 12:56 |
<godog> |
upgrade nagios-nrpe-server on scb2* and mwlog* - T261198 |
[production] |
| 12:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1111 MCR changes', diff saved to https://phabricator.wikimedia.org/P12343 and previous config saved to /var/cache/conftool/dbconfig/20200825-125108-marostegui.json |
[production] |
| 12:45 |
<marostegui> |
Update MySQL on db1111 after MCR change |
[production] |
| 12:39 |
<marostegui> |
alter table sites on s6, directly on the primary master T260476 |
[production] |
| 12:39 |
<godog> |
test nagios-nrpe-server with dh 2048 on scb2001 - T261198 |
[production] |
| 12:35 |
<moritzm> |
imported ceph packages from stretch-backports to component/ceph T256877 |
[production] |
| 12:10 |
<moritzm> |
installing ruby-json security updates |
[production] |
| 12:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1135 MCR change', diff saved to https://phabricator.wikimedia.org/P12341 and previous config saved to /var/cache/conftool/dbconfig/20200825-120708-marostegui.json |
[production] |
| 12:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1118 MCR changes', diff saved to https://phabricator.wikimedia.org/P12340 and previous config saved to /var/cache/conftool/dbconfig/20200825-120211-marostegui.json |
[production] |
| 11:59 |
<jayme@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . |
[production] |
| 11:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1118 MCR changes', diff saved to https://phabricator.wikimedia.org/P12339 and previous config saved to /var/cache/conftool/dbconfig/20200825-114938-marostegui.json |
[production] |
| 11:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1118 MCR changes', diff saved to https://phabricator.wikimedia.org/P12338 and previous config saved to /var/cache/conftool/dbconfig/20200825-113758-marostegui.json |
[production] |
| 11:36 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 11:32 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 11:29 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1118 MCR changes', diff saved to https://phabricator.wikimedia.org/P12337 and previous config saved to /var/cache/conftool/dbconfig/20200825-112859-marostegui.json |
[production] |
| 11:25 |
<marostegui> |
Upgrade mysql on db1118 after MCR change |
[production] |
| 11:16 |
<Urbanecm> |
EU B&C done |
[production] |
| 11:07 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: d869e308492ee72cb3d1998b15409aa44a4af9c7: Enable ContentTranslation as a default tool in Assamese and Burmese WPs (T258503; T258505) (duration: 01m 00s) |
[production] |
| 10:59 |
<moritzm> |
installing remaining libx11 security updates |
[production] |
| 10:37 |
<arturo> |
import all binary packages from tesseract-ocr-lang into stretch-wikimedia/component/tesseract-410-bpo (T247422) |
[production] |
| 10:32 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 10:28 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' . |
[production] |
| 10:28 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' . |
[production] |
| 10:23 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 10:23 |
<moritzm> |
removed fermium.wikimedia.org from debmonitor |
[production] |
| 09:45 |
<marostegui> |
Create missing table cx_notification_log on x1 wikishared T261190 |
[production] |
| 08:50 |
<XioNoX> |
re-activate eqord peering/transit - T259593 |
[production] |
| 08:19 |
<XioNoX> |
reconfigure eqord to be AS65020 - T259593 |
[production] |
| 08:18 |
<XioNoX> |
deactivate eqord peering/transit - T259593 |
[production] |
| 07:22 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=1) |
[production] |
| 07:13 |
<marostegui> |
Upgrade MySQL on dbstore1004 |
[production] |
| 07:09 |
<dcausse> |
depooling wdqs1005 (high lag) |
[production] |
| 07:04 |
<dcausse> |
restartint blazegraph on wdqs1005 (T242453) |
[production] |
| 06:20 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.prepare-upgrade |
[production] |
| 05:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1111, db1118 for MCR change', diff saved to https://phabricator.wikimedia.org/P12336 and previous config saved to /var/cache/conftool/dbconfig/20200825-053856-marostegui.json |
[production] |
| 05:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1084,db1092 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12335 and previous config saved to /var/cache/conftool/dbconfig/20200825-053801-marostegui.json |
[production] |
| 05:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1084,db1092 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12334 and previous config saved to /var/cache/conftool/dbconfig/20200825-052602-marostegui.json |
[production] |
| 05:21 |
<moritzm> |
installing Java security updates on relforge* |
[production] |
| 05:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1084,db1092 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12333 and previous config saved to /var/cache/conftool/dbconfig/20200825-051327-marostegui.json |
[production] |
| 05:11 |
<marostegui> |
Remove revisions triggers from db2094:3311 T238966 |
[production] |
| 05:10 |
<marostegui> |
Deploy MCR schema change on s1 codfw, this will create lag on s1 codfw - T238966 |
[production] |
| 05:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1084,db1092 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12332 and previous config saved to /var/cache/conftool/dbconfig/20200825-050451-marostegui.json |
[production] |
| 04:02 |
<ejegg> |
updated fundraising python tools from 305f2a4438 to dcad0bfe75 |
[production] |