|
2019-04-04
ยง
|
| 18:13 |
<moritzm> |
restarted apache on people.wikimedia.org to pick up OpenSSL update |
[production] |
| 17:59 |
<bstorm_> |
stopped postgresql on labsdb1006.eqiad.wmnet and moved the database master functionality (and all rsyncs) to clouddb1003.clouddb-services.eqiad.wmflabs |
[production] |
| 17:59 |
<ppchelko@deploy1001> |
Finished deploy [cpjobqueue/deploy@922cbc0]: Switch to new logging infrastructure T211125 (duration: 04m 03s) |
[production] |
| 17:55 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@922cbc0]: Switch to new logging infrastructure T211125 |
[production] |
| 17:47 |
<ppchelko@deploy1001> |
Finished deploy [changeprop/deploy@f69dc9c]: Switch to new logging infrastructure T211125 (duration: 01m 44s) |
[production] |
| 17:45 |
<ppchelko@deploy1001> |
Started deploy [changeprop/deploy@f69dc9c]: Switch to new logging infrastructure T211125 |
[production] |
| 17:33 |
<jynus> |
stopping replication on dbstore2001:s8 for backup testing T206203 |
[production] |
| 17:29 |
<jynus> |
killing ongoing backup at dbprov2002, stuck |
[production] |
| 17:28 |
<gehel@cumin2001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 17:10 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 16:31 |
<herron> |
beginning rolling kafka restarts on kafka200[123] for security updates |
[production] |
| 16:01 |
<herron> |
repooling kafka2003 eventbus |
[production] |
| 15:59 |
<mutante> |
wikivoyage-old.org domain has been retired and deactivated (T219867, T81727) |
[production] |
| 15:56 |
<herron> |
depooling kafka2003 for eventbus security updates |
[production] |
| 15:55 |
<herron> |
repooling kafka2002 eventbus |
[production] |
| 15:52 |
<herron> |
depooling kafka2002 for eventbus security updates |
[production] |
| 15:52 |
<herron> |
pooling kafka2001 eventbus |
[production] |
| 15:42 |
<herron> |
depooling kafka2001 for eventbus security updates |
[production] |
| 15:38 |
<moritzm> |
rolling restart of proton to pick up openssl security update |
[production] |
| 15:03 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=0) |
[production] |
| 14:59 |
<moritzm> |
installing libdatetime-timezone-perl updates |
[production] |
| 14:24 |
<jiji@cumin1001> |
conftool action : set/pooled=no; selector: dc=codfw,service=cxserver,cluster=scb,name=scb.* |
[production] |
| 14:24 |
<jiji@cumin1001> |
conftool action : set/pooled=no; selector: dc=eqiad,service=cxserver,cluster=scb,name=scb.* |
[production] |
| 14:23 |
<jijiki> |
Depooling scb* from service cxserver traffic |
[production] |
| 13:46 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 13:46 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 03m 37s) |
[production] |
| 13:29 |
<jbond42> |
restart of gerrit apache service will occure at 13:40 |
[production] |
| 13:28 |
<volans> |
upgraded spicerack to 0.0.22 on cumin[12]001 |
[production] |
| 13:27 |
<volans> |
uploaded spicerack_0.0.22-1_amd64.deb to apt.wikimedia.org stretch-wikimedia |
[production] |
| 13:23 |
<moritzm> |
upgrading mw1261 to HHVM 3.18.5+dfsg-1+wmf8+deb9u2 / wikidiff 1.8.1 |
[production] |
| 13:20 |
<jijiki> |
Stopped all citoid services from scb* - 494215 |
[production] |
| 13:15 |
<jbond42> |
restart of phabricator apache service will occure at 14:25 |
[production] |
| 12:46 |
<moritzm> |
uploaded HHVM 3.18.5+dfsg-1+wmf8+deb9u2 to apt.wikimedia.org/stretch-wikimedia |
[production] |
| 12:10 |
<arturo> |
T219626 reimaging cloudcontrol2001-dev again |
[production] |
| 11:43 |
<moritzm> |
upgrading HHVM on mwdebug servers in eqiad along with update to hhvm-wikidiff 1.8.1 |
[production] |
| 11:35 |
<moritzm> |
uploaded nodejs 10.15.2~dfsg-1+wmf1 to the component/node10 component of apt.wikimedia.org/stretch-wikimedia (updated to latest 10.x release and a change to ensure zlib binary compat with NodeSource) (T215562) |
[production] |
| 11:34 |
<Amir1> |
EU SWAT is done |
[production] |
| 11:32 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:500976|Add mediawiki.org to the URL shortener whitelist]] (duration: 00m 58s) |
[production] |
| 11:28 |
<jbond42> |
rolling security updates for apache on jessie |
[production] |
| 11:25 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:498371|Enable ReferencePreviews beta feature on de- and ar-wiki (T218766)]] (duration: 01m 00s) |
[production] |
| 11:21 |
<arturo> |
T219626 reimaging cloudcontrol2001-dev again |
[production] |
| 11:08 |
<arturo> |
drop python-psutil from jessie-wikimedia/openstack-mitaka-jessie, related to T219626 |
[production] |
| 10:56 |
<moritzm> |
uploaded hhvm-wikidiff 1.8.1 to apt.wikimedia.org/stretch-wikimedia (source package is named php-wikdiff2 for legacy reasons) (T203069) |
[production] |
| 10:21 |
<arturo> |
T219626 reimaging cloudcontrol2001-dev again |
[production] |
| 10:01 |
<moritzm> |
installing openssl1.0 security updates on stretch-based DB hosts |
[production] |
| 08:36 |
<moritzm> |
rolling restart of parsoid to pick up OpenSSL security update |
[production] |
| 08:06 |
<moritzm> |
uploaded Apache 2.4.10-10+deb8u14+wmf1 to apt.wikimedia.org/jessie-wikimedia (latest jessie security update rebased with our local patches) |
[production] |
| 05:39 |
<marostegui> |
Stop MySQL on db2033 for decommission - T219493 |
[production] |
| 05:32 |
<marostegui> |
Remove db2033 from tendril and zarcillo - T219493 |
[production] |
| 05:19 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2033 for decommission T219493 (duration: 00m 59s) |
[production] |