|
2019-03-21
§
|
| 13:18 |
<gtirloni> |
downtimed cloudcontrol*, cloudservices*, labcontrol*, labweb* (T210818) |
[production] |
| 13:06 |
<moritzm> |
installing Java security updates on stat hosts |
[production] |
| 12:40 |
<arturo> |
T216497 remove python-cliff from jessie-wikimedia/openstack-mitaka-jessie |
[production] |
| 12:35 |
<jijiki> |
Pooling mw1339 back |
[production] |
| 12:33 |
<jijiki> |
Pooling mw1290 back |
[production] |
| 12:08 |
<arturo> |
T216497 add python-cliff to jessie-wikimedia/openstack-mitaka-jessie |
[production] |
| 12:02 |
<vgutierrez> |
uploaded acme-chief 0.15 to apt.wikimedia.org (buster) - T218862 |
[production] |
| 11:54 |
<elukey> |
restart yarn node managers on an-worker10[82,89,92] - shutdown after a long yarn failover and only now downtime is expired |
[production] |
| 11:36 |
<mutante> |
gerrit2001 (not the master prod server)- scheduled downtime and rebooting for upgrade |
[production] |
| 11:04 |
<zeljkof> |
EU SWAT finished |
[production] |
| 11:04 |
<zfilipin@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:495494|Add new throttle rule for LMU Edit-a-thon (T217929)]] (duration: 00m 57s) |
[production] |
| 10:57 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=prometheus2004.codfw.wmnet |
[production] |
| 10:52 |
<filippo@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=prometheus2003.codfw.wmnet |
[production] |
| 10:46 |
<elukey> |
restart hadoop yarn resource managers on an-master100[1,2] to pick up new settings |
[production] |
| 10:23 |
<moritzm> |
rebooting labtestcontrol2001 for kernel update |
[production] |
| 10:19 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 56s) |
[production] |
| 09:59 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1086 (duration: 00m 58s) |
[production] |
| 09:42 |
<jiji@cumin1001> |
conftool action : set/pooled=no; selector: dc=codfw,service=cxserver,cluster=scb,name=scb.* |
[production] |
| 09:42 |
<jijiki> |
Depool scb* in codfw from serving cxserver, finishing its migration to k8s - T213195 |
[production] |
| 09:29 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1086 after mysql upgrade (duration: 00m 56s) |
[production] |
| 09:27 |
<moritzm> |
rolling reboot of maps servers in codfw for kernel update |
[production] |
| 09:17 |
<marostegui> |
Upgrade and reboot db1086 |
[production] |
| 08:53 |
<marostegui> |
Upgrade db1086 |
[production] |
| 08:53 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1086 for upgrade (duration: 00m 56s) |
[production] |
| 08:43 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1086 (duration: 00m 57s) |
[production] |
| 08:20 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 56s) |
[production] |
| 08:09 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 56s) |
[production] |
| 08:01 |
<vgutierrez> |
deploying directory based certificates in acme-chief clients - T207295 |
[production] |
| 07:35 |
<_joe_> |
rolling restart of php-fpm to pick up some changes |
[production] |
| 07:34 |
<marostegui> |
Deploy schema change on db1079, this will generate lag on labsdb:s8 |
[production] |
| 07:33 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1079 (duration: 00m 57s) |
[production] |
| 07:03 |
<elukey> |
restart pdfrender on scb1002 |
[production] |
| 06:55 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 (duration: 00m 56s) |
[production] |
| 06:24 |
<marostegui> |
Run wmcs-wikireplica-dns on cloudcontrol1003 to get dbproxy1011 back |
[production] |
| 06:13 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 (duration: 01m 10s) |
[production] |
| 06:12 |
<marostegui> |
Upgrade and reboot dbproxy1011 |
[production] |
| 06:04 |
<marostegui> |
Run wmcs-wikireplica-dns on cloudcontrol1003 to drain dbproxy1011 |
[production] |
| 00:09 |
<jforrester@deploy1001> |
Synchronized php-1.33.0-wmf.22/includes/parser/BlockLevelPass.php: SAT T218817 Unbreak parser line counting for long wikitext pages I22eebb70a I55a2c4c0 I41a45266d (duration: 00m 56s) |
[production] |
| 00:08 |
<twentyafterfour> |
deploying phabricator upgrade |
[production] |
| 00:01 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT Move FundraisingTranslateWorkflow load to after Translate I73452ae8 (duration: 00m 56s) |
[production] |
|
2019-03-20
§
|
| 23:49 |
<jforrester@deploy1001> |
Synchronized php-1.33.0-wmf.22/resources/lib/ooui/oojs-ui-core.js: SWAT T218722 T218830 Bring forward UBN OOUI fix (duration: 00m 57s) |
[production] |
| 23:28 |
<maxsem@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/497948/ (duration: 00m 56s) |
[production] |
| 23:10 |
<maxsem@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/490648/ (duration: 00m 56s) |
[production] |
| 22:29 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T214075 Enable federation of Wikidata items and properties on Test Commons (duration: 00m 57s) |
[production] |
| 21:37 |
<XioNoX> |
apply transit-in4 term offload-ping4 with test IP to cr1/2-codfw - T190090 |
[production] |
| 21:34 |
<XioNoX> |
apply transit-in4 term offload-ping4 with test IP to cr2-codfw |
[production] |
| 21:00 |
<XioNoX> |
apply icmp redirect on cr1-codfw:xe-5/0/2 (to cr4-ulsfo) for test IP 208.80.154.225 - T190090 |
[production] |
| 20:24 |
<zfilipin@deploy1001> |
Synchronized php: group1 wikis to 1.33.0-wmf.22 (duration: 01m 46s) |
[production] |
| 20:23 |
<zfilipin@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.22 |
[production] |
| 20:13 |
<otto@deploy1001> |
scap-helm eventgate-analytics cluster codfw completed |
[production] |