|
2020-10-30
§
|
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:31 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:06 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:04 |
<robh@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:48 |
<cdanis> |
the above scap began (and mostly finished) several minutes ago but is hanging on a couple hosts down for maintenance |
[production] |
| 18:48 |
<cdanis@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: lower frwiki featured feeds limit 1a41ef634 T266865 (duration: 05m 14s) |
[production] |
| 18:47 |
<cdanis> |
✔️ cdanis@deploy1001.eqiad.wmnet /srv/mediawiki-staging 🕝☕ scap sync-file wmf-config/InitialiseSettings.php 'lower frwiki featured feeds limit 1a41ef634 T266865' |
[production] |
| 18:27 |
<hashar@deploy1001> |
Finished deploy [integration/docroot@c35e5e9]: Add ECS to doc.wikimedia.org index (duration: 00m 06s) |
[production] |
| 18:27 |
<hashar@deploy1001> |
Started deploy [integration/docroot@c35e5e9]: Add ECS to doc.wikimedia.org index |
[production] |
| 17:38 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:36 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:22 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:22 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:19 |
<effie> |
disable puppet on mc1036 and mc2036 - T252391 |
[production] |
| 17:18 |
<effie> |
enable puppet on all mediawiki and mc* hosts |
[production] |
| 16:19 |
<elukey> |
kafka-jumbo1006 still running with 1g nick |
[production] |
| 15:36 |
<effie> |
stopping puppet on mediawiki and mc* hosts |
[production] |
| 15:11 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:11 |
<rzl@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:09 |
<rzl> |
downtiming mc2036 for buster reimage |
[production] |
| 14:42 |
<elukey> |
stop kafka-jumbo1006 to swap NICs (1g -> 10g, d1 -> d4 rack) |
[production] |
| 14:14 |
<cmjohnson1> |
moving mw1267 and mw168 to rack A8 eqiad T266164 |
[production] |
| 12:29 |
<XioNoX> |
set normal VRRP balancing on cr2-eqiad |
[production] |
| 10:08 |
<klausman@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:08 |
<klausman@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:02 |
<ladsgroup@deploy1001> |
Synchronized static/images/project-logos: Revert: Changing logo of Wikidata for the brithday (duration: 01m 12s) |
[production] |
| 09:13 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 09:07 |
<elukey@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 08:58 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 08:54 |
<elukey> |
decom an-tool1006 (old analytics test vm) - T255139 |
[production] |
| 08:53 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
|
2020-10-29
§
|
| 23:59 |
<eileen> |
process-control config revision is 6891d35bce |
[production] |
| 23:39 |
<Urbanecm> |
Evening B&C window done |
[production] |
| 23:38 |
<Urbanecm> |
urbanecm@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=trwikiquote --add-prefix=BROKEN --fix # T266605 # P13112 |
[production] |
| 23:37 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: ddb7e08e9c1d07f704c9f7585d8b6089f1895b5c: Add namespace aliases to Turkish Wikiquote (T266605) (duration: 00m 57s) |
[production] |
| 23:36 |
<eileen> |
process-control config revision is 1114512f90 |
[production] |
| 23:29 |
<Urbanecm> |
urbanecm@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=trwikisource --add-prefix=BROKEN --fix # T266606 # P13111 |
[production] |
| 23:27 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: c3a8555154673c4c5a65f6ec2a1219d0832f48e0: Add namespace aliases to Turkish Wikisource (T266606) (duration: 00m 56s) |
[production] |
| 23:23 |
<Urbanecm> |
urbanecm@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=trwikibooks --fix # T266608 |
[production] |
| 23:22 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 1800d11ec8c07ff6ccffe0fd03ce11e6786f8a6e: Add namespace aliases to Turkish Wikibooks (T266608) (duration: 00m 57s) |
[production] |
| 23:22 |
<eileen> |
civicrm revision changed from e1d65b0f3a to 3317d30356, config revision is d70fe02cb9 |
[production] |
| 23:18 |
<Urbanecm> |
urbanecm@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=trwiktionary --fix # T266609 |
[production] |
| 23:17 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 090f75730727e7a3ca5a85af0ff9071213dd047f: Add namespace aliases to Turkish Wiktionary (T266609) (duration: 00m 58s) |
[production] |
| 22:35 |
<mutante> |
mw1268 - depooled for T266164 |
[production] |
| 22:34 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1268.eqiad.wmnet |
[production] |
| 22:33 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 22:33 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:32 |
<mutante> |
mw1269 rsyncd/ferm for scap proxy was enabled - mw1268 rsyncd/ferm for scan proxy was removed - deploy1001 scap-proxies dsh group was adjusted |
[production] |
| 22:21 |
<mutante> |
replacing scap proxy for rack A7 eqiad because mw1268 needs to move physically (T266164) |
[production] |