|
2019-03-21
ยง
|
| 16:29 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 16:03 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2096 for onsite maintenance (duration: 00m 50s) |
[production] |
| 16:01 |
<marostegui> |
Poweroff db2096 for onsite maintenance T218336 |
[production] |
| 15:20 |
<moritzm> |
rebooting flerovium/furud for kernel updates |
[production] |
| 14:35 |
<moritzm> |
restarging jenkins on releases* after Java update |
[production] |
| 14:18 |
<gtirloni> |
downtimed labtestweb2001 (T218881) |
[production] |
| 14:11 |
<vgutierrez> |
re-enabling puppet in acme-chief clients - T218862 |
[production] |
| 14:09 |
<arturo> |
T218024 disabled icinga checks for labtestweb2001 |
[production] |
| 14:07 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.e6-upgrade |
[production] |
| 13:58 |
<vgutierrez> |
update acme-chief to version 0.15 in acmechief1001 - T218862 |
[production] |
| 13:54 |
<vgutierrez> |
disabling puppet in acme-chief clients - T218862 |
[production] |
| 13:48 |
<akosiaris> |
reboot oresrdb2001 |
[production] |
| 13:39 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 51s) |
[production] |
| 13:37 |
<elukey> |
upgrade openjdk-8 on an-worker1080 and restarted hadoop daemons |
[production] |
| 13:28 |
<moritzm> |
installing Java security updates on notebook hosts |
[production] |
| 13:22 |
<zfilipin@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.22 |
[production] |
| 13:18 |
<gtirloni> |
downtimed cloudcontrol*, cloudservices*, labcontrol*, labweb* (T210818) |
[production] |
| 13:06 |
<moritzm> |
installing Java security updates on stat hosts |
[production] |
| 12:40 |
<arturo> |
T216497 remove python-cliff from jessie-wikimedia/openstack-mitaka-jessie |
[production] |
| 12:35 |
<jijiki> |
Pooling mw1339 back |
[production] |
| 12:33 |
<jijiki> |
Pooling mw1290 back |
[production] |
| 12:08 |
<arturo> |
T216497 add python-cliff to jessie-wikimedia/openstack-mitaka-jessie |
[production] |
| 12:02 |
<vgutierrez> |
uploaded acme-chief 0.15 to apt.wikimedia.org (buster) - T218862 |
[production] |
| 11:54 |
<elukey> |
restart yarn node managers on an-worker10[82,89,92] - shutdown after a long yarn failover and only now downtime is expired |
[production] |
| 11:36 |
<mutante> |
gerrit2001 (not the master prod server)- scheduled downtime and rebooting for upgrade |
[production] |
| 11:04 |
<zeljkof> |
EU SWAT finished |
[production] |
| 11:04 |
<zfilipin@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:495494|Add new throttle rule for LMU Edit-a-thon (T217929)]] (duration: 00m 57s) |
[production] |
| 10:57 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=prometheus2004.codfw.wmnet |
[production] |
| 10:52 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=prometheus2003.codfw.wmnet |
[production] |
| 10:46 |
<elukey> |
restart hadoop yarn resource managers on an-master100[1,2] to pick up new settings |
[production] |
| 10:23 |
<moritzm> |
rebooting labtestcontrol2001 for kernel update |
[production] |
| 10:19 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 56s) |
[production] |
| 09:59 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1086 (duration: 00m 58s) |
[production] |
| 09:42 |
<jiji@cumin1001> |
conftool action : set/pooled=no; selector: dc=codfw,service=cxserver,cluster=scb,name=scb.* |
[production] |
| 09:42 |
<jijiki> |
Depool scb* in codfw from serving cxserver, finishing its migration to k8s - T213195 |
[production] |
| 09:29 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1086 after mysql upgrade (duration: 00m 56s) |
[production] |
| 09:27 |
<moritzm> |
rolling reboot of maps servers in codfw for kernel update |
[production] |
| 09:17 |
<marostegui> |
Upgrade and reboot db1086 |
[production] |
| 08:53 |
<marostegui> |
Upgrade db1086 |
[production] |
| 08:53 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1086 for upgrade (duration: 00m 56s) |
[production] |
| 08:43 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1086 (duration: 00m 57s) |
[production] |
| 08:20 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 56s) |
[production] |
| 08:09 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 56s) |
[production] |
| 08:01 |
<vgutierrez> |
deploying directory based certificates in acme-chief clients - T207295 |
[production] |
| 07:35 |
<_joe_> |
rolling restart of php-fpm to pick up some changes |
[production] |
| 07:34 |
<marostegui> |
Deploy schema change on db1079, this will generate lag on labsdb:s8 |
[production] |
| 07:33 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1079 (duration: 00m 57s) |
[production] |
| 07:03 |
<elukey> |
restart pdfrender on scb1002 |
[production] |
| 06:55 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 (duration: 00m 56s) |
[production] |
| 06:24 |
<marostegui> |
Run wmcs-wikireplica-dns on cloudcontrol1003 to get dbproxy1011 back |
[production] |