|
2019-05-16
§
|
| 14:07 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 14:07 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:57 |
<marostegui> |
and recreate the following hosts in tendril: db2103,db2104,db2105,db2106,db2107,db2108,db2109,db2110,db2111,db2112,db2113,db2115,db2116,db2117,db2119 T222772 |
[production] |
| 13:50 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:50 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:39 |
<cmjohnson1> |
replacing pdu in rack B5 eqiad |
[production] |
| 13:04 |
<hashar@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.34.0-wmf.5 |
[production] |
| 13:00 |
<arturo> |
labweb1001 depooled |
[production] |
| 12:59 |
<mobrovac> |
bootstrap restbase1020-c - T219404 |
[production] |
| 12:58 |
<aborrero@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=labweb1001.wikimedia.org,service=labweb |
[production] |
| 12:21 |
<godog> |
stop swift and rsync on ms-be10[16,17,18,32,33] for eqiad B5 pdu replacement - T223126 |
[production] |
| 12:02 |
<jynus> |
stop and shutdown db1098,db1131,db1139 T223126 |
[production] |
| 11:56 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:55 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:54 |
<moritzm> |
rebooting mw app servers in codfw for kernel update |
[production] |
| 11:32 |
<hoo@deploy1001> |
Synchronized wmf-config/extension-list: Add EntitySchema to extension-list (T221650) (duration: 00m 56s) |
[production] |
| 11:22 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1098 & db1131 for maintenance (duration: 00m 57s) |
[production] |
| 11:00 |
<arturo> |
T223148 downtime cloudvirt[1014,1028].eqiad.wmnet and labweb1001.wikimedia.org for 8 hours |
[production] |
| 11:00 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:00 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:00 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:00 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:00 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:00 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:50 |
<godog> |
bootstrap restbase1020-b - T219404 |
[production] |
| 10:27 |
<jiji@deploy1001> |
Finished deploy [cpjobqueue/deploy@4d55dff]: Migrating updateBetaFeaturesUserCounts to PHP7 - T219148 (duration: 01m 07s) |
[production] |
| 10:26 |
<jiji@deploy1001> |
Started deploy [cpjobqueue/deploy@4d55dff]: Migrating updateBetaFeaturesUserCounts to PHP7 - T219148 |
[production] |
| 08:52 |
<akosiaris> |
upgrade mathoid to statsd_exporter 0.9 T220709 |
[production] |
| 08:48 |
<akosiaris@deploy1001> |
scap-helm mathoid finished |
[production] |
| 08:48 |
<akosiaris@deploy1001> |
scap-helm mathoid cluster codfw completed |
[production] |
| 08:48 |
<akosiaris@deploy1001> |
scap-helm mathoid cluster eqiad completed |
[production] |
| 08:48 |
<akosiaris@deploy1001> |
scap-helm mathoid upgrade -f mathoid-values.yaml production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw] |
[production] |
| 08:47 |
<akosiaris@deploy1001> |
scap-helm mathoid upgrade -f mathoid-values.yaml [namespace: mathoid, clusters: eqiad,codfw] |
[production] |
| 08:37 |
<godog> |
bootstrap restbase1020-a - T219404 |
[production] |
| 08:32 |
<elukey> |
depool/restart-nutcracker-pool mw1293/1313 - T214275 |
[production] |
| 08:22 |
<elukey> |
depool/restart-nutcracker-pool mw1238 - T214275 |
[production] |
| 08:03 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1104 (duration: 00m 56s) |
[production] |
| 07:57 |
<moritzm> |
installing linux 4.9.168-1+deb9u2~deb8u1 kernel on jessie hosts (no reboots, just installing the new package) |
[production] |
| 07:45 |
<moritzm> |
removed intel-microcode 3.20180807a from jessie-wikimedia (superceded by newer version in security.debian.org, which doesn't get picked up by apt due to the higher apr priority of jessie-wikimedia) |
[production] |
| 07:44 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1104 into API (duration: 00m 56s) |
[production] |
| 07:25 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1104 (duration: 00m 57s) |
[production] |
| 06:59 |
<moritzm> |
installing intel-microcode updates |
[production] |
| 05:34 |
<elukey> |
roll restart of nutcracker on mw2* to pick up new config changes (no more memcached config) - T214275 |
[production] |
| 05:33 |
<marostegui> |
Stop MySQL on db1104 to clone db1126 |
[production] |
| 05:29 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1104 (duration: 00m 56s) |
[production] |
| 05:18 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Pool db2106, db2110, db2119 into s4 - T222772 (duration: 00m 56s) |
[production] |
| 05:17 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Pool db2106, db2110, db2119 into s4 - T222772 (duration: 00m 58s) |
[production] |
| 02:27 |
<onimisionipe> |
pooling elastic2038 after unbanning - T217398 |
[production] |