|
2020-08-25
§
|
| 10:28 |
<oblivian@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' . |
[production] |
| 10:23 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 10:23 |
<moritzm> |
removed fermium.wikimedia.org from debmonitor |
[production] |
| 09:45 |
<marostegui> |
Create missing table cx_notification_log on x1 wikishared T261190 |
[production] |
| 08:50 |
<XioNoX> |
re-activate eqord peering/transit - T259593 |
[production] |
| 08:19 |
<XioNoX> |
reconfigure eqord to be AS65020 - T259593 |
[production] |
| 08:18 |
<XioNoX> |
deactivate eqord peering/transit - T259593 |
[production] |
| 07:22 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.network.prepare-upgrade (exit_code=1) |
[production] |
| 07:13 |
<marostegui> |
Upgrade MySQL on dbstore1004 |
[production] |
| 07:09 |
<dcausse> |
depooling wdqs1005 (high lag) |
[production] |
| 07:04 |
<dcausse> |
restartint blazegraph on wdqs1005 (T242453) |
[production] |
| 06:20 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.prepare-upgrade |
[production] |
| 05:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1111, db1118 for MCR change', diff saved to https://phabricator.wikimedia.org/P12336 and previous config saved to /var/cache/conftool/dbconfig/20200825-053856-marostegui.json |
[production] |
| 05:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1084,db1092 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12335 and previous config saved to /var/cache/conftool/dbconfig/20200825-053801-marostegui.json |
[production] |
| 05:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1084,db1092 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12334 and previous config saved to /var/cache/conftool/dbconfig/20200825-052602-marostegui.json |
[production] |
| 05:21 |
<moritzm> |
installing Java security updates on relforge* |
[production] |
| 05:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1084,db1092 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12333 and previous config saved to /var/cache/conftool/dbconfig/20200825-051327-marostegui.json |
[production] |
| 05:11 |
<marostegui> |
Remove revisions triggers from db2094:3311 T238966 |
[production] |
| 05:10 |
<marostegui> |
Deploy MCR schema change on s1 codfw, this will create lag on s1 codfw - T238966 |
[production] |
| 05:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1084,db1092 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12332 and previous config saved to /var/cache/conftool/dbconfig/20200825-050451-marostegui.json |
[production] |
| 04:02 |
<ejegg> |
updated fundraising python tools from 305f2a4438 to dcad0bfe75 |
[production] |
| 01:49 |
<eileen> |
civicrm revision changed from ce28723709 to 0f195c6cca, config revision is 96839009f1 |
[production] |
| 01:39 |
<eileen> |
civicrm revision is ce28723709, config revision is 96839009f1 |
[production] |
| 01:30 |
<eileen> |
civicrm revision is ce28723709, config revision is 54c8c7abf2 |
[production] |
| 01:17 |
<cdanis> |
repool esams |
[production] |
| 01:11 |
<cdanis> |
T259621 wrong junos version was staged on cr2-esams, abandoning this attempt and putting back in service |
[production] |
| 01:07 |
<cdanis> |
cdanis@re0.cr2-esams> request system software add validate re1 /var/tmp/junos-vmhost-install-mx-x86-64-17.3R3-S8.1.tgz |
[production] |
| 00:56 |
<cdanis> |
T259621 ❌cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 homer 'cr*' commit 'drain cr2-esams transport link' |
[production] |
| 00:36 |
<cdanis> |
T259621 cdanis@re1.cr3-esams> request chassis routing-engine master switch |
[production] |
| 00:30 |
<cdanis> |
T259621 cdanis@re1.cr3-esams> request vmhost reboot re0 |
[production] |
| 00:24 |
<cdanis> |
T259621 cdanis@re1.cr3-esams> request vmhost software add /var/tmp/junos-vmhost-install-mx-x86-64-17.3R3-S8.1.tgz re0 |
[production] |
| 00:18 |
<cdanis> |
T259621 cdanis@re0.cr3-esams> request chassis routing-engine master switch |
[production] |
| 00:14 |
<cdanis> |
T259621 cdanis@re0.cr3-esams> request vmhost reboot re1 |
[production] |
| 00:08 |
<cdanis> |
T259621 cdanis@re0.cr3-esams> request vmhost software add /var/tmp/junos-vmhost-install-mx-x86-64-17.3R3-S8.1.tgz re1 |
[production] |
|
2020-08-24
§
|
| 23:46 |
<cdanis> |
depool esams T259621 |
[production] |
| 23:16 |
<Urbanecm> |
Evening B&C window done |
[production] |
| 23:06 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: 778f710bbbdb24730f7ce4c75d5ff1ca7a5ce3b3: Alternate configuration mechanism for Parsoid (T241961) (duration: 00m 58s) |
[production] |
| 22:13 |
<rzl@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 22:10 |
<rzl@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 21:29 |
<sbassett@deploy1001> |
Synchronized private/PrivateSettings.php: Deployed additional mitigations for T257687 (duration: 00m 58s) |
[production] |
| 20:29 |
<rzl> |
re-enabled puppet on 'R:File = /etc/nutcracker/nutcracker.yml' T261154 |
[production] |
| 19:25 |
<rzl> |
disabling puppet on 'R:File = /etc/nutcracker/nutcracker.yaml' to swap mc2028 out for mc2037 T261154 |
[production] |
| 18:10 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: cirrus: Increase weight of grants and research namespaces in metawiki search (duration: 00m 58s) |
[production] |
| 15:20 |
<jynus> |
shutdown backup2001 T260764 |
[production] |
| 15:13 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 15:08 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . |
[production] |
| 15:04 |
<vgutierrez> |
rolling restart of ats-tls to disable ECDHE-RSA-AES128-SHA - T258405 |
[production] |
| 14:58 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 14:55 |
<rzl> |
switchover test complete, puppet re-enabled on cumin1001 |
[production] |
| 14:54 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) |
[production] |