|
2018-04-26
§
|
| 12:53 |
<marostegui> |
Deploy schema change on db1113:3312 - T191519 T188299 T190148 |
[production] |
| 12:53 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1113 for alter table (duration: 01m 33s) |
[production] |
| 12:51 |
<gehel> |
reindexing lost updates on elasticsearch - T193112 |
[production] |
| 12:04 |
<mobrovac@tin> |
Finished deploy [cpjobqueue/deploy@7fbb152]: Support the exclude_topics config stanza (duration: 01m 12s) |
[production] |
| 12:03 |
<mobrovac@tin> |
Started deploy [cpjobqueue/deploy@7fbb152]: Support the exclude_topics config stanza |
[production] |
| 10:35 |
<moritzm> |
reimaging mw1312 mw1317, mw1339 (API servers) to stretch |
[production] |
| 10:29 |
<moritzm> |
reimaging mw1269, mw1323, mw1324 (app servers) to stretch |
[production] |
| 09:57 |
<marostegui> |
Drop prefswitch_survey on s1 - T173439 |
[production] |
| 09:50 |
<godog> |
eqiad-prod: more weight to ms-be104[0-3] for container/account - T190081 |
[production] |
| 09:45 |
<marostegui> |
Drop prefswitch_survey on s3 - T173439 |
[production] |
| 09:41 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1109 with low load (duration: 01m 16s) |
[production] |
| 09:30 |
<marostegui> |
Drop prefswitch_survey on s7 - T173439 |
[production] |
| 09:16 |
<marostegui> |
Drop prefswitch_survey on s2 - T173439 |
[production] |
| 09:15 |
<mark> |
Temp disabling cr1-ulsfo:xe-1/2/0 (Chicago transport) due to stability issues |
[production] |
| 09:13 |
<marostegui> |
Drop prefswitch_survey on s4 - T173439 |
[production] |
| 09:02 |
<marostegui> |
Drop prefswitch_survey on s5 and s6 - T173439 |
[production] |
| 09:01 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 01m 16s) |
[production] |
| 08:51 |
<moritzm> |
reimaging mw1320, mw1321, mw1322 (app servers) to stretch |
[production] |
| 08:32 |
<moritzm> |
re-attempt reimage of mw1246 (failed yesterday with an error on the puppetmaster, testing whether this can be reproduced) |
[production] |
| 08:24 |
<jynus> |
stop and upgrade db1109 |
[production] |
| 07:58 |
<marostegui> |
Deploy schema change on db1090 - T191519 T188299 T190148 |
[production] |
| 07:45 |
<jynus> |
stopping db1090 mariadb instance to move its path, port and socket |
[production] |
| 07:21 |
<gehel> |
restarting redis masters in codfw - T193112 |
[production] |
| 07:20 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1090, pool db1122 with full weight (duration: 01m 23s) |
[production] |
| 07:16 |
<gehel> |
re-enabling puppet on rdb2* - T193112 |
[production] |
| 06:19 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch |
[production] |
| 05:18 |
<marostegui> |
Deploy schema change on dbstore1002:s2 - T191519 T188299 T190148 |
[production] |
| 04:39 |
<ebernhardson> |
unfreeze writes to elasticsearch codfw cluster |
[production] |
| 03:54 |
<_joe_> |
stopping redis replication from eqiad to codfw for the jobqueue cluster, we have an issue ongoing with CirrusSearch jobs and replication is broken |
[production] |
| 03:41 |
<ejegg> |
re-enabled ingenico recurring charge job |
[production] |
| 02:05 |
<mutante> |
mw2163 through mw2166: since the wmf-auto-reimage failed after OS but before puppet run due to "Failed to puppet_generate_certs" i manually logged in with install-console and signed puppet certs (T174431) |
[production] |
|
2018-04-25
§
|
| 22:55 |
<samwilson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Undeploy GlobalPreferences T184121 (duration: 01m 16s) |
[production] |
| 22:21 |
<samwilson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Deploy GlobalPreferences T189806 (duration: 01m 18s) |
[production] |
| 21:03 |
<demon@tin> |
rebuilt and synchronized wikiversions files: group1 to wmf.1 |
[production] |
| 21:01 |
<demon@tin> |
Synchronized php: symlink bump (duration: 01m 16s) |
[production] |
| 20:58 |
<hasharAway> |
on tin: rebased php-1.31.0-wmf.30 for https://gerrit.wikimedia.org/r/#/c/429018/ |
[production] |
| 20:21 |
<XioNoX> |
remove test VIP for eqiad ping offload server - T190090 |
[production] |
| 20:18 |
<bsitzmann@tin> |
Finished deploy [mobileapps/deploy@5a4a282]: Config: Start up to 4 workers in parallel during start-up (duration: 06m 48s) |
[production] |
| 20:11 |
<bsitzmann@tin> |
Started deploy [mobileapps/deploy@5a4a282]: Config: Start up to 4 workers in parallel during start-up |
[production] |
| 19:39 |
<otto@tin> |
Finished deploy [eventlogging/eventbus@f0783bb]: Fix for logging error https://gerrit.wikimedia.org/r/#/c/428982/ (duration: 01m 45s) |
[production] |
| 19:37 |
<otto@tin> |
Started deploy [eventlogging/eventbus@f0783bb]: Fix for logging error https://gerrit.wikimedia.org/r/#/c/428982/ |
[production] |
| 19:12 |
<urandom> |
altering timeline tables for 6 month TTL -- T192689 |
[production] |
| 19:11 |
<otto@tin> |
Finished deploy [eventlogging/eventbus@f0783bb]: Fix for logging error https://gerrit.wikimedia.org/r/#/c/428982/ (duration: 00m 11s) |
[production] |
| 19:11 |
<otto@tin> |
Started deploy [eventlogging/eventbus@f0783bb]: Fix for logging error https://gerrit.wikimedia.org/r/#/c/428982/ |
[production] |
| 19:09 |
<otto@tin> |
Started deploy [eventlogging/eventbus@f562c1b]: Fix for logging error https://gerrit.wikimedia.org/r/#/c/428982/ |
[production] |
| 18:55 |
<imarlier@tin> |
Finished deploy [performance/coal@1e79c79]: deploy fix for coal-web (duration: 00m 06s) |
[production] |
| 18:55 |
<imarlier@tin> |
Started deploy [performance/coal@1e79c79]: deploy fix for coal-web |
[production] |
| 18:16 |
<ejegg> |
updated CiviCRM from 219798b2c5 to 47197006d5 |
[production] |
| 17:35 |
<urandom> |
starting cleanups on row 'a' Cassandra nodes -- T189822 |
[production] |
| 17:33 |
<mepps> |
update civicrm from 6ddeb167ec to 219798b2c5 |
[production] |