|
2020-10-30
§
|
| 23:01 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 23:01 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:43 |
<GarbageMonster> |
restarting codesearch to pick up new config (T266909) |
[codesearch] |
| 21:02 |
<jiji@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:00 |
<jiji@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:59 |
<mutante> |
mw1267,mw1268 - scap pull and repool - back to prod - T266164 |
[production] |
| 20:57 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1267.eqiad.wmnet |
[production] |
| 20:57 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1268.eqiad.wmnet |
[production] |
| 20:56 |
<mutante> |
mw1267,mw1268 - scap pull |
[production] |
| 20:33 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:07 |
<longma> |
reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/631864 |
[releng] |
| 20:06 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:04 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:48 |
<cdanis> |
the above scap began (and mostly finished) several minutes ago but is hanging on a couple hosts down for maintenance |
[production] |
| 18:48 |
<cdanis@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: lower frwiki featured feeds limit 1a41ef634 T266865 (duration: 05m 14s) |
[production] |
| 18:47 |
<cdanis> |
✔️ cdanis@deploy1001.eqiad.wmnet /srv/mediawiki-staging 🕝☕ scap sync-file wmf-config/InitialiseSettings.php 'lower frwiki featured feeds limit 1a41ef634 T266865' |
[production] |
| 18:37 |
<andrewbogott> |
re-enabling puppet on mailman-mailman01; it's been disabled for 128176 minutes |
[mailman] |
| 18:33 |
<andrewbogott> |
truncating /var/lib/docker/containers/00a5169f547721a9e3dd2efa59141be77c923921de3274d15fbd55f40e504f90/00a5169f547721a9e3dd2efa59141be77c923921de3274d15fbd55f40e504f90-json.log on ffedprops-opennext; that file was as big as the whole hard drive |
[wikidata-dev] |
| 18:27 |
<hashar@deploy1001> |
Finished deploy [integration/docroot@c35e5e9]: Add ECS to doc.wikimedia.org index (duration: 00m 06s) |
[production] |
| 18:27 |
<hashar@deploy1001> |
Started deploy [integration/docroot@c35e5e9]: Add ECS to doc.wikimedia.org index |
[production] |
| 18:16 |
<hashar> |
Successfully tagged docker-registry.discovery.wmnet/releng/ecs:0.0.2-1 # T234565 |
[releng] |
| 17:48 |
<hashar> |
Successfully tagged docker-registry.discovery.wmnet/releng/ecs:0.0.1-1 # T234565 |
[releng] |
| 17:38 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:36 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:22 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:22 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:19 |
<effie> |
disable puppet on mc1036 and mc2036 - T252391 |
[production] |
| 17:18 |
<effie> |
enable puppet on all mediawiki and mc* hosts |
[production] |
| 17:01 |
<elukey> |
kafka preferred-replica-election on jumbo1001 |
[analytics] |
| 16:19 |
<elukey> |
kafka-jumbo1006 still running with 1g nick |
[production] |
| 15:36 |
<effie> |
stopping puppet on mediawiki and mc* hosts |
[production] |
| 15:11 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:11 |
<rzl@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:09 |
<rzl> |
downtiming mc2036 for buster reimage |
[production] |
| 14:42 |
<elukey> |
stop kafka-jumbo1006 to swap NICs (1g -> 10g, d1 -> d4 rack) |
[production] |