|
2019-01-15
ยง
|
| 19:28 |
<SMalyshev> |
depooling wdqs1008 and wdqs2004 for DB copying for T213854 |
[production] |
| 18:52 |
<bblack> |
authdns-update for https://gerrit.wikimedia.org/r/c/operations/dns/+/484546 (make normal git stuff match manual changes already in place) |
[production] |
| 18:44 |
<hashar> |
[2019-01-15 18:44:06,959] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.15.6-5-g4b9c845200 ready |
[production] |
| 18:43 |
<hashar> |
Restarting Gerrit to catch up with a DNS change with the database |
[production] |
| 18:43 |
<volans> |
restarted debmonitor on debmonitor1001 |
[production] |
| 18:40 |
<bblack> |
DNS manually updated for m1-master -> dbproxy1006 and m2-master -> dbproxy1007 |
[production] |
| 17:26 |
<godog> |
roll-restart logstash in eqiad - T213081 |
[production] |
| 17:21 |
<godog> |
depool logstash1007 before restarting logstash - T213081 |
[production] |
| 17:13 |
<godog> |
set partitions to 3 for existing kafka-logging topics - T213081 |
[production] |
| 17:06 |
<XioNoX> |
move back cr1-eqiad:xe-4/1/3 to xe-3/3/1 - T212791 |
[production] |
| 16:57 |
<XioNoX> |
move cr1-eqiad:xe-3/3/1 to xe-4/1/3 - T212791 |
[production] |
| 16:52 |
<jynus> |
stop db1115 for hw maintenance |
[production] |
| 16:50 |
<godog> |
roll-restart kafka-logging in eqiad to apply new topic defaults - T213081 |
[production] |
| 16:00 |
<jynus> |
stop es1019 for hw maintenance T213422 |
[production] |
| 15:53 |
<dcausse> |
T210381: elastic search clusters, catching up updates since first import on new psi&omega clusters in eqiad&codfw (from mwmaint1002) |
[production] |
| 15:10 |
<fdans@deploy1001> |
Finished deploy [analytics/superset/deploy@UNKNOWN]: reverting deploy of 0.26.3-wikimedia1 (duration: 00m 32s) |
[production] |
| 15:10 |
<fdans@deploy1001> |
Started deploy [analytics/superset/deploy@UNKNOWN]: reverting deploy of 0.26.3-wikimedia1 |
[production] |
| 15:02 |
<fdans@deploy1001> |
Finished deploy [analytics/superset/deploy@9d6156a]: reverting deploy of 0.26.3-wikimedia1 (duration: 06m 06s) |
[production] |
| 15:01 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1103 (duration: 00m 48s) |
[production] |
| 14:56 |
<fdans@deploy1001> |
Started deploy [analytics/superset/deploy@9d6156a]: reverting deploy of 0.26.3-wikimedia1 |
[production] |
| 14:41 |
<fdans@deploy1001> |
Finished deploy [analytics/superset/deploy@408a30e]: deploying 0.26.3-wikimedia1 (duration: 00m 36s) |
[production] |
| 14:40 |
<fdans@deploy1001> |
Started deploy [analytics/superset/deploy@408a30e]: deploying 0.26.3-wikimedia1 |
[production] |
| 14:14 |
<moritzm> |
rebooting acamar |
[production] |
| 13:53 |
<marostegui> |
Downtime db1115 and es1019 for 4 hours - T196726 T213422 |
[production] |
| 13:33 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1119 T85757 (duration: 00m 46s) |
[production] |
| 13:15 |
<marostegui> |
Deploy schema change on db1119 - T85757 |
[production] |
| 13:15 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1119 T85757 (duration: 00m 46s) |
[production] |
| 13:00 |
<elukey> |
restart memcached on mc1024 to pick up new settings (-R 200) - T208844 |
[production] |
| 12:47 |
<dcausse> |
EU SWAT done |
[production] |
| 12:36 |
<dcausse@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T210381: [cirrus] Start writing to psi & omega (take 2) (2/2) (duration: 00m 45s) |
[production] |
| 12:33 |
<dcausse@deploy1001> |
Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] Start writing to psi & omega (take 2) (1/2) (duration: 00m 45s) |
[production] |
| 12:15 |
<onimisionipe> |
starting upgrading of prometheus-elasticsearch-exporter for eqiad T210592 |
[production] |
| 12:14 |
<dcausse@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Change links of wgGEHelpPanelLinks for kowiki T209467 (duration: 00m 46s) |
[production] |
| 12:09 |
<dcausse@deploy1001> |
Synchronized wmf-config/CommonSettings.php: [cirrus] Add cirrussearch-big-indices tag T210381 (duration: 00m 46s) |
[production] |
| 12:06 |
<jynus> |
upgrade and restart db1103 |
[production] |
| 12:03 |
<onimisionipe> |
starting upgrading of prometheus-elasticsearch-exporter for codfw T210592 |
[production] |
| 11:50 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1103 (duration: 00m 45s) |
[production] |
| 11:44 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1091 fully (duration: 00m 45s) |
[production] |
| 11:10 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1083 T85757 (duration: 00m 45s) |
[production] |
| 11:02 |
<jynus> |
dropping database test on db1124:s5 with replication |
[production] |
| 11:01 |
<elukey> |
run 'apt-get purge tmpreaper' on mw1297,1298,2150,2151,2244,2245 (all role spare) to avoid daily cronspam |
[production] |
| 10:58 |
<END> |
(PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) (volans@cumin2001) |
[production] |
| 10:57 |
<marostegui> |
Deploy schema change on db1083 - T85757 |
[production] |
| 10:56 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1083 T85757 (duration: 00m 46s) |
[production] |
| 10:53 |
<START> |
- Cookbook sre.hosts.upgrade-and-reboot (volans@cumin2001) |
[production] |
| 10:49 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1091 with low load (duration: 00m 45s) |
[production] |
| 10:40 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1080 T85757 (duration: 00m 45s) |
[production] |
| 10:20 |
<marostegui> |
Deploy schema change on db1080 - T85757 |
[production] |
| 10:20 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1080 T85757 (duration: 00m 45s) |
[production] |
| 10:19 |
<jynus> |
upgrade and restart db1091 |
[production] |