|
2019-04-12
§
|
| 13:24 |
<akosiaris> |
re-enable puppet across the fleet. Patch merged, recovery storm coming |
[production] |
| 13:18 |
<akosiaris> |
disable puppet across the fleet to avoid incoming puppet alert storm |
[production] |
| 12:57 |
<marostegui> |
Purge old rows and optimize tables on spare host pc1010 T210725 |
[production] |
| 12:53 |
<urandom> |
decommissioning cassandra-c, restbase2008 -- T208087 |
[production] |
| 12:49 |
<gehel> |
rolling restart of cassandra on maps* for jvm upgrade |
[production] |
| 12:22 |
<arturo> |
T220095 disable icinga checks for labtestcontrol2003 |
[production] |
| 12:16 |
<gilles@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T220807 Reduce cawiki survey sampling rate (duration: 05m 11s) |
[production] |
| 11:56 |
<moritzm> |
upgrading app server canaries to version 1.8.1 of the PHP wikidiff extension (HHVM already deployed) T203069 |
[production] |
| 11:46 |
<moritzm> |
upgrading acmechief hosts to latest buster state |
[production] |
| 11:44 |
<gilles@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T220807 Oversample navtiming on cawiki and commonswiki (duration: 05m 14s) |
[production] |
| 11:37 |
<Trey314159> |
reindexing Greek, Turkish, and Irish wikis on elastic@eqiad and elastic@codfw complete (T217806) |
[production] |
| 11:19 |
<moritzm> |
installed Java security updates on relforge* hosts |
[production] |
| 11:10 |
<moritzm> |
installing Java security updates on remaining maps hosts |
[production] |
| 10:32 |
<arturo> |
T219626 reimaging cloudcontrol2001-dev |
[production] |
| 10:13 |
<elukey> |
matomo updated to 3.9.1 on matomo1001 + deb upload to wikimedia-stretch - T218037 |
[production] |
| 09:53 |
<moritzm> |
updated mwdebug1001 to php-wikidiff 1.8.1 |
[production] |
| 09:37 |
<moritzm> |
updated mwdebug1002 to php-wikidiff 1.8.1 |
[production] |
| 09:30 |
<volans> |
reset mgmt card on labtestcontrol2003 - T220783 |
[production] |
| 09:07 |
<moritzm> |
added the wikimedia repository key to the stretch build chroot on boron, fixes builds using the PHP72/SPICERACK hooks |
[production] |
| 09:05 |
<arturo> |
T218021 disable icinga checks for labtestcontrol2001 |
[production] |
| 08:35 |
<gilles@deploy1001> |
Synchronized php-1.33.0-wmf.25/extensions/NavigationTiming/modules/ext.navigationTiming.js: T220788 Fix veaction === null case (duration: 00m 54s) |
[production] |
| 08:02 |
<moritzm> |
updated ssacli in thirdparty/hwraid component for stretch to 3.30-13.0 T220787 |
[production] |
| 07:12 |
<marostegui> |
Manually install ssacli on db2[097|098|099|100|101|102] T220787 T220572 |
[production] |
| 07:04 |
<moritzm> |
synced ssacli to thirdparty/hwraid components for jessie/stretch T220787 |
[production] |
| 01:00 |
<mutante> |
puppet cert clean, puppet node clean, puppet node deactivate on cloudnet2001-dev.codfw.wmnet (T218025) |
[production] |
| 00:25 |
<tstarling@deploy1001> |
Synchronized wmf-config/profiler.php: increase excimer max depth (duration: 00m 53s) |
[production] |
| 00:02 |
<ejegg> |
updated fundraising CiviCRM from 24b968b1f9 to 1bc1570967 |
[production] |
|
2019-04-11
§
|
| 23:57 |
<urandom> |
decommissioning cassandra-b, restbase2008 -- T208087 |
[production] |
| 22:15 |
<jforrester@deploy1001> |
Synchronized php-1.33.0-wmf.25/extensions/WikibaseMediaInfo/resources/: Hot-deploy fix for WBMI variable cache miss T220665 (duration: 00m 55s) |
[production] |
| 20:46 |
<mutante> |
deleting job of wikibugs-phab-listener in an attempt to restart it |
[production] |
| 12:48 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: dc=codfw,cluster=maps,name=maps2001.codfw.wmnet |
[production] |
| 12:20 |
<kartik@deploy1001> |
scap-helm cxserver finished |
[production] |
| 12:19 |
<kartik@deploy1001> |
scap-helm cxserver cluster eqiad completed |
[production] |
| 12:19 |
<kartik@deploy1001> |
scap-helm cxserver upgrade -f cxserver-eqiad-values.yaml production stable/cxserver [namespace: cxserver, clusters: eqiad] |
[production] |
| 12:16 |
<kartik@deploy1001> |
scap-helm cxserver finished |
[production] |
| 12:15 |
<kartik@deploy1001> |
scap-helm cxserver cluster codfw completed |
[production] |
| 12:15 |
<kartik@deploy1001> |
scap-helm cxserver upgrade -f cxserver-codfw-values.yaml production stable/cxserver [namespace: cxserver, clusters: codfw] |
[production] |
| 11:39 |
<zeljkof> |
EU SWAT finished |
[production] |
| 09:57 |
<elukey> |
roll restart druid-coordinator/overlord on druid100[4-6] to pick up new jvm settings |
[production] |
| 08:20 |
<moritzm> |
upgrading remaining job runners to HHVM 3.18.5+dfsg-1+wmf8+deb9u2 and wikidiff 1.8.1 (T203069) |
[production] |
| 06:33 |
<moritzm> |
uploaded jenkins 2.164.2 to apt.wikimedia.org (stretch-wikimedia / thirdparty/ci) |
[production] |
| 06:32 |
<moritzm> |
uploaded jenkins 2.164.2 to apt.wikimedia.org (jessie-wikimedia / thirdparty) |
[production] |
| 06:24 |
<moritzm> |
upgrading remaining API Servers to HHVM 3.18.5+dfsg-1+wmf8+deb9u2 and wikidiff 1.8.1 (T203069) |
[production] |
| 05:03 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove s3 ready only T219115 (duration: 00m 36s) |
[production] |
| 05:02 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Switchover s3 master eqiad from db1078 to db1075 T219115 (duration: 00m 36s) |
[production] |
| 05:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Set s3 on read-only T219115 (duration: 00m 37s) |
[production] |
| 05:00 |
<marostegui> |
Starting s3 failover from db1078 to db1075 - T219115 |
[production] |
| 04:32 |
<marostegui> |
Disable puppet on db1078 and db1075 T219115 |
[production] |
| 04:18 |
<marostegui> |
Start topology changes to move s3 slaves under db1075 T219115 |
[production] |
| 04:14 |
<marostegui> |
Disable GTID on s3 hosts - https://phabricator.wikimedia.org/T219115 |
[production] |