|
2020-12-10
§
|
| 09:17 |
<dcaro> |
removing leaked dns record discordwiki.eqiad.wmflabs (clinic duty) |
[admin] |
| 09:17 |
<dcaro> |
removing leaked dns record discordwiki.srwiki-dev.eqiad.wmflabs (clinic duty) |
[srwiki-dev] |
| 09:11 |
<effie> |
disable puppet on all hosts running redis - T265643 |
[production] |
| 08:58 |
<dcaro> |
starting a new etcd instance completely from ansible playbook (etcd-8) (T267412) |
[toolsbeta] |
| 08:22 |
<godog> |
swift codfw-prod: more weight to ms-be20[58-61] - T269337 |
[production] |
| 06:38 |
<kart_> |
Upgraded Apertium to 2020-12-09-115733-production |
[production] |
| 06:35 |
<kartik@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 06:35 |
<kartik@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'apertium' for release 'plain' . |
[production] |
| 06:30 |
<kartik@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'plain' . |
[production] |
| 06:30 |
<kartik@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'apertium' for release 'production' . |
[production] |
| 06:27 |
<kartik@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'apertium' for release 'staging' . |
[production] |
| 00:42 |
<ejegg> |
updated payments-wiki from 756c2f7ce0 to df80a99b40 |
[production] |
| 00:26 |
<robh> |
cr2-eqsin bad fan being swapped via T267544 |
[production] |
| 00:05 |
<mutante> |
re-enabling puppet (disabled without reason) on wikistats-wild-tiger |
[wikistats] |
|
2020-12-09
§
|
| 23:21 |
<mutante> |
repooling parse2001 after buster reimage - T268524 |
[production] |
| 23:19 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2243.codfw.wmnet |
[production] |
| 23:16 |
<mutante> |
repooling parse2001 after buster reimage - T245757 |
[production] |
| 23:15 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=parse2001.codfw.wmnet |
[production] |
| 23:04 |
<mutante> |
zero.wikimedia.beta.wmflabs.org removed from beta_sites (deployment-prep) T187716 |
[production] |
| 22:32 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 22:30 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:24 |
<mutante> |
shut down instance traffic-dnsbox |
[traffic] |
| 21:40 |
<bstorm> |
shutting down labstore1006 for maintenance T268285 |
[production] |
| 21:11 |
<mutante> |
traffic-dnsbox deleted /var/log/daemon.log.1 to free disk space - 100% -> 68% , ran puppet but it has puppet unrelated puppet failure, instance is most likely unused and can be deleted also per -traffic |
[traffic] |
| 20:54 |
<mutante> |
traffic-dnsbox had broken puppet and reason was out of disk space, gzipping 6.8G /var/log/daemon.log.1 |
[traffic] |
| 20:34 |
<elukey> |
execute on mysql:an-coord1002 "set GLOBAL replicate_wild_ignore_table='superset_staging.%'" to avoid replication for superset_staging from an-coord1002 |
[analytics] |
| 20:27 |
<mutante> |
mw1281,mw1282,mw1283 - scap pull |
[production] |
| 20:26 |
<mutante> |
repooling mw1281,mw1282,mw1283 - now in rack A8 |
[production] |
| 20:26 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw128[1-3].eqiad.wmnet |
[production] |
| 20:22 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1283.eqiad.wmnet |
[production] |
| 20:22 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1282.eqiad.wmnet |
[production] |
| 20:21 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1281.eqiad.wmnet |
[production] |
| 20:18 |
<twentyafterfour> |
wmf.21 looks good on group1 wikis. Still seeing T269603 but not at an increased rate. (refs T264801) |
[production] |
| 20:13 |
<twentyafterfour@deploy1001> |
Synchronized php: group1 wikis to 1.36.0-wmf.21 (duration: 01m 02s) |
[production] |
| 20:12 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.21 |
[production] |
| 19:38 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: ce01bbe7b05eda8065fc57c865a69370e8aae797: Enable ArticlePlaceholder at papwiki (T223693) (duration: 01m 02s) |
[production] |
| 19:17 |
<twentyafterfour@deploy1001> |
Synchronized php-1.36.0-wmf.21/includes/page/Article.php: deploy 0d99fe6d54 Article::view - remove the old subtitle from doOutputFromParserCache. Bug: T269727 (duration: 01m 04s) |
[production] |
| 18:59 |
<mutante> |
testreduce1001 - installed make |
[production] |
| 18:48 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:43 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:42 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:41 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:17 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw2243.codfw.wmnet |
[production] |
| 18:16 |
<mutante> |
depooling mw2243 (jobrunner) for reimaging (T245757) |
[production] |
| 18:15 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2243.codfw.wmnet |
[production] |
| 18:12 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:11 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:05 |
<mutante> |
mw1281,mw1282,mw1283 shut down for T266164 |
[production] |
| 18:02 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:02 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |