|
2020-10-30
§
|
| 21:00 |
<jiji@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:59 |
<mutante> |
mw1267,mw1268 - scap pull and repool - back to prod - T266164 |
[production] |
| 20:57 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1267.eqiad.wmnet |
[production] |
| 20:57 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1268.eqiad.wmnet |
[production] |
| 20:56 |
<mutante> |
mw1267,mw1268 - scap pull |
[production] |
| 20:33 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 20:32 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:06 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:04 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:48 |
<cdanis> |
the above scap began (and mostly finished) several minutes ago but is hanging on a couple hosts down for maintenance |
[production] |
| 18:48 |
<cdanis@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: lower frwiki featured feeds limit 1a41ef634 T266865 (duration: 05m 14s) |
[production] |
| 18:47 |
<cdanis> |
✔️ cdanis@deploy1001.eqiad.wmnet /srv/mediawiki-staging 🕝☕ scap sync-file wmf-config/InitialiseSettings.php 'lower frwiki featured feeds limit 1a41ef634 T266865' |
[production] |
| 18:27 |
<hashar@deploy1001> |
Finished deploy [integration/docroot@c35e5e9]: Add ECS to doc.wikimedia.org index (duration: 00m 06s) |
[production] |
| 18:27 |
<hashar@deploy1001> |
Started deploy [integration/docroot@c35e5e9]: Add ECS to doc.wikimedia.org index |
[production] |
| 17:38 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:36 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:22 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:22 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:19 |
<effie> |
disable puppet on mc1036 and mc2036 - T252391 |
[production] |
| 17:18 |
<effie> |
enable puppet on all mediawiki and mc* hosts |
[production] |
| 16:19 |
<elukey> |
kafka-jumbo1006 still running with 1g nick |
[production] |
| 15:36 |
<effie> |
stopping puppet on mediawiki and mc* hosts |
[production] |
| 15:11 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:11 |
<rzl@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:09 |
<rzl> |
downtiming mc2036 for buster reimage |
[production] |
| 14:42 |
<elukey> |
stop kafka-jumbo1006 to swap NICs (1g -> 10g, d1 -> d4 rack) |
[production] |
| 14:14 |
<cmjohnson1> |
moving mw1267 and mw168 to rack A8 eqiad T266164 |
[production] |
| 12:29 |
<XioNoX> |
set normal VRRP balancing on cr2-eqiad |
[production] |
| 10:08 |
<klausman@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:08 |
<klausman@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:02 |
<ladsgroup@deploy1001> |
Synchronized static/images/project-logos: Revert: Changing logo of Wikidata for the brithday (duration: 01m 12s) |
[production] |
| 09:13 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 09:07 |
<elukey@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 08:58 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 08:54 |
<elukey> |
decom an-tool1006 (old analytics test vm) - T255139 |
[production] |
| 08:53 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |