|
2018-09-13
§
|
| 08:16 |
<marostegui> |
Stop replication on s4 eqiad master (db1068) and deploy a schema change - this will generate lag on s4 eqiad - T144010 |
[production] |
| 08:08 |
<marostegui> |
Deploy schema change on s5 eqiad master - T187089 |
[production] |
| 08:01 |
<moritzm> |
rebooting dns recursors in eqsin for kernel security update |
[production] |
| 08:01 |
<marostegui> |
Disconnect replication eqiad -> codfw on s1-s8, x1, es2, es3 - T189107 |
[production] |
| 07:47 |
<elukey> |
reboot kafka10[12-23] (old analytics cluster) for kernel upgrades |
[production] |
| 07:46 |
<elukey> |
execute apt-get autoremove on notebook* to remove old nginx packages (not used anymore) |
[production] |
| 07:25 |
<jynus@deploy1001> |
Synchronized wmf-config/db-codfw.php: Tune db weights even more (duration: 00m 50s) |
[production] |
| 06:56 |
<elukey> |
reboot notebook100[3,4] for kernel upgrades - T203165 |
[production] |
| 06:41 |
<elukey> |
reboot stat100[4-6] for kernel upgrades - T203165 |
[production] |
| 05:11 |
<marostegui> |
Stop MySQL on db2054 and dbstore2001:3317 to clone db2054 - T204127 |
[production] |
| 02:46 |
<l10nupdate@deploy1001> |
ResourceLoader cache refresh completed at Thu Sep 13 02:46:16 UTC 2018 (duration 10m 46s) |
[production] |
| 02:35 |
<l10nupdate@deploy1001> |
scap sync-l10n completed (1.32.0-wmf.20) (duration: 14m 03s) |
[production] |
| 02:24 |
<ejegg> |
restarted fundraising CiviCRM jobs |
[production] |
| 01:48 |
<ejegg> |
updated CiviCRM from 2a3583cca1 to b0d7df4e60 |
[production] |
| 01:45 |
<ejegg> |
disabled fundraising CiviCRM jobs |
[production] |
| 00:00 |
<smalyshev@deploy1001> |
Finished deploy [wdqs/wdqs@7e5e537]: Deploy Blazegraph & Updater for T202765 and T203646 handling (duration: 23m 45s) |
[production] |
|
2018-09-12
§
|
| 23:36 |
<smalyshev@deploy1001> |
Started deploy [wdqs/wdqs@7e5e537]: Deploy Blazegraph & Updater for T202765 and T203646 handling |
[production] |
| 22:41 |
<Trey314159> |
reindexing Esperanto wikis on elastic@codfw and elastic@eqiad complete (T203005) |
[production] |
| 19:50 |
<jynus@deploy1001> |
Synchronized wmf-config/db-codfw.php: Decrease db2061 load (duration: 00m 50s) |
[production] |
| 19:13 |
<jynus@deploy1001> |
Synchronized wmf-config/db-codfw.php: Decrease db2065 load (duration: 00m 49s) |
[production] |
| 18:24 |
<Trey314159> |
reindexing Esperanto wikis on elastic@codfw and elastic@eqiad (T203005) |
[production] |
| 17:59 |
<otto@deploy1001> |
Finished deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars for T203804 (duration: 06m 11s) |
[production] |
| 17:53 |
<otto@deploy1001> |
Started deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars for T203804 |
[production] |
| 17:52 |
<otto@deploy1001> |
Finished deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars for T203804 (duration: 01m 41s) |
[production] |
| 17:50 |
<otto@deploy1001> |
Started deploy [analytics/refinery@407da92]: Deploying refinery-source 0.0.74 jars for T203804 |
[production] |
| 17:38 |
<akosiaris> |
restart icinga T196336 |
[production] |
| 16:53 |
<ottomata> |
restarting eventlogging processors to pick up change to mysql whitelist |
[production] |
| 16:52 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Decrease weight for db2065 (duration: 00m 49s) |
[production] |
| 16:48 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=wtp2020.codfw.wmnet |
[production] |
| 16:39 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: cirrussearch: eqiad -> local (codfw) (duration: 00m 50s) |
[production] |
| 16:32 |
<jynus@deploy1001> |
Synchronized wmf-config/db-codfw.php: Reduce db2065, db2061 weights (duration: 00m 48s) |
[production] |
| 16:14 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Decrease weight for db2049 (duration: 00m 49s) |
[production] |
| 16:06 |
<jynus@deploy1001> |
Synchronized wmf-config/db-codfw.php: Reduce db2056 weight (duration: 00m 49s) |
[production] |
| 15:56 |
<akosiaris> |
switch s*-master DNS records to codfw |
[production] |
| 15:51 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Decrease weight for db2049 (duration: 00m 50s) |
[production] |
| 15:38 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: 43b98e414 (duration: 00m 50s) |
[production] |
| 15:32 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Decrease weight for db2070 (duration: 00m 49s) |
[production] |
| 15:24 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-restart-parsoid (exit_code=0) (volans@sarin) |
[production] |
| 15:22 |
<oblivian@sarin> |
conftool action : set/weight=15; selector: dc=codfw,cluster=api_appserver,service=apache2,name=mw21.* |
[production] |
| 15:19 |
<oblivian@sarin> |
conftool action : set/weight=20; selector: dc=codfw,cluster=api_appserver,service=apache2,name=mw22[2-9].* |
[production] |
| 15:17 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Tweak weights in s1 and s2 (duration: 00m 50s) |
[production] |
| 15:15 |
<_joe_> |
repool mw2141 |
[production] |
| 15:14 |
<_joe_> |
depooling mw2141 for investigation |
[production] |
| 15:08 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-restart-parsoid (volans@sarin) |
[production] |
| 15:05 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) (volans@sarin) |
[production] |
| 15:05 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-restore-ttl (volans@sarin) |
[production] |
| 15:05 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) (volans@sarin) |
[production] |
| 15:03 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-start-maintenance (volans@sarin) |
[production] |
| 14:59 |
<END> |
(PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) (volans@sarin) |
[production] |
| 14:58 |
<START> |
- Cookbook sre.switchdc.mediawiki.08-update-tendril (volans@sarin) |
[production] |