|
2019-05-16
§
|
| 12:58 |
<aborrero@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=labweb1001.wikimedia.org,service=labweb |
[production] |
| 12:21 |
<godog> |
stop swift and rsync on ms-be10[16,17,18,32,33] for eqiad B5 pdu replacement - T223126 |
[production] |
| 12:02 |
<jynus> |
stop and shutdown db1098,db1131,db1139 T223126 |
[production] |
| 11:56 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:55 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:54 |
<moritzm> |
rebooting mw app servers in codfw for kernel update |
[production] |
| 11:32 |
<hoo@deploy1001> |
Synchronized wmf-config/extension-list: Add EntitySchema to extension-list (T221650) (duration: 00m 56s) |
[production] |
| 11:22 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1098 & db1131 for maintenance (duration: 00m 57s) |
[production] |
| 11:00 |
<arturo> |
T223148 downtime cloudvirt[1014,1028].eqiad.wmnet and labweb1001.wikimedia.org for 8 hours |
[production] |
| 11:00 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:00 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:00 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:00 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:00 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:00 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:50 |
<godog> |
bootstrap restbase1020-b - T219404 |
[production] |
| 10:27 |
<jiji@deploy1001> |
Finished deploy [cpjobqueue/deploy@4d55dff]: Migrating updateBetaFeaturesUserCounts to PHP7 - T219148 (duration: 01m 07s) |
[production] |
| 10:26 |
<jiji@deploy1001> |
Started deploy [cpjobqueue/deploy@4d55dff]: Migrating updateBetaFeaturesUserCounts to PHP7 - T219148 |
[production] |
| 08:52 |
<akosiaris> |
upgrade mathoid to statsd_exporter 0.9 T220709 |
[production] |
| 08:48 |
<akosiaris@deploy1001> |
scap-helm mathoid finished |
[production] |
| 08:48 |
<akosiaris@deploy1001> |
scap-helm mathoid cluster codfw completed |
[production] |
| 08:48 |
<akosiaris@deploy1001> |
scap-helm mathoid cluster eqiad completed |
[production] |
| 08:48 |
<akosiaris@deploy1001> |
scap-helm mathoid upgrade -f mathoid-values.yaml production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw] |
[production] |
| 08:47 |
<akosiaris@deploy1001> |
scap-helm mathoid upgrade -f mathoid-values.yaml [namespace: mathoid, clusters: eqiad,codfw] |
[production] |
| 08:37 |
<godog> |
bootstrap restbase1020-a - T219404 |
[production] |
| 08:32 |
<elukey> |
depool/restart-nutcracker-pool mw1293/1313 - T214275 |
[production] |
| 08:22 |
<elukey> |
depool/restart-nutcracker-pool mw1238 - T214275 |
[production] |
| 08:03 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1104 (duration: 00m 56s) |
[production] |
| 07:57 |
<moritzm> |
installing linux 4.9.168-1+deb9u2~deb8u1 kernel on jessie hosts (no reboots, just installing the new package) |
[production] |
| 07:45 |
<moritzm> |
removed intel-microcode 3.20180807a from jessie-wikimedia (superceded by newer version in security.debian.org, which doesn't get picked up by apt due to the higher apr priority of jessie-wikimedia) |
[production] |
| 07:44 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1104 into API (duration: 00m 56s) |
[production] |
| 07:25 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1104 (duration: 00m 57s) |
[production] |
| 06:59 |
<moritzm> |
installing intel-microcode updates |
[production] |
| 05:34 |
<elukey> |
roll restart of nutcracker on mw2* to pick up new config changes (no more memcached config) - T214275 |
[production] |
| 05:33 |
<marostegui> |
Stop MySQL on db1104 to clone db1126 |
[production] |
| 05:29 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1104 (duration: 00m 56s) |
[production] |
| 05:18 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Pool db2106, db2110, db2119 into s4 - T222772 (duration: 00m 56s) |
[production] |
| 05:17 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Pool db2106, db2110, db2119 into s4 - T222772 (duration: 00m 58s) |
[production] |
| 02:27 |
<onimisionipe> |
pooling elastic2038 after unbanning - T217398 |
[production] |
|
2019-05-15
§
|
| 22:16 |
<mutante> |
phab1003 - start ssh-phab service after adding service IPs |
[production] |
| 22:01 |
<eileen> |
civicrm update - lost the commit versions but 5.13.4 release |
[production] |
| 21:47 |
<mutante> |
phab1003 - ip -6 addr del 2620:0:861:ed1a::3:16/128 dev lo - remove extra service IP for phab's separate sshd, duplicated with phab1001 (T190568) |
[production] |
| 21:24 |
<jforrester@deploy1001> |
Synchronized wmf-config/MetaContactPages.php: Add movecomsignup contact page on meta T218363 (duration: 00m 56s) |
[production] |
| 21:23 |
<eileen> |
civicrm revision changed from 7d3ef1f2ae to c69c6e2e6a, config revision is a099f13a55 |
[production] |
| 21:00 |
<fdans@deploy1001> |
Finished deploy [analytics/refinery@ffa4931]: deploying analytics refinery (duration: 15m 31s) |
[production] |
| 20:45 |
<tgr@deploy1001> |
Finished deploy [proton/deploy@9373c42]: Add gistcdn.githack.com to host blacklist (T213362) (duration: 02m 41s) |
[production] |
| 20:45 |
<fdans@deploy1001> |
Started deploy [analytics/refinery@ffa4931]: deploying analytics refinery |
[production] |
| 20:42 |
<tgr@deploy1001> |
Started deploy [proton/deploy@9373c42]: Add gistcdn.githack.com to host blacklist (T213362) |
[production] |
| 20:20 |
<robh> |
rebooting cloudvirt1015 into dell hardware tests per T220853 |
[production] |
| 20:18 |
<arlolra@deploy1001> |
Finished deploy [parsoid/deploy@8f28977]: Updating Parsoid to 6658cad (duration: 06m 23s) |
[production] |