|
2020-04-27
§
|
| 09:11 |
<marostegui> |
Deploy schema change on s1 codfw, lag will show up - T250055 |
[production] |
| 08:52 |
<moritzm> |
restarting cas on idp1001 to pick up Java 11 security update (will void active SSO sessions) |
[production] |
| 08:26 |
<marostegui> |
Deploy schema change on s5 codfw, lag will show up - T250055 |
[production] |
| 08:24 |
<kormat> |
Truncating and optimizing parsercache for pc1010 and pc2010 T247787 |
[production] |
| 08:18 |
<mutante> |
running puppet on all cp-ats |
[production] |
| 08:15 |
<godog> |
add 80G to prometheus global LV |
[production] |
| 07:25 |
<elukey> |
roll restart elastic-chi on cloudelastic100[1-4] to pick up the last JVM GC settings - T231517 |
[production] |
| 07:15 |
<marostegui> |
Kill updateSpecialPages.php wikidatawiki --override --only=Fewestrevisions as it is causing lag - T238199 |
[production] |
| 07:14 |
<elukey> |
powercycle an-worker1089 - unreachable via ssh, mgmt serial available, soft cpu lock events registered in dmesg |
[production] |
| 06:59 |
<elukey> |
force ifdown/ifup eno1 on analytics1052 - interface negotiated speed flapping |
[production] |
| 06:42 |
<moritzm> |
installing Java security updates on IDP hosts, will void current SSO sessions |
[production] |
| 06:30 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: name=mw1280.eqiad.wmnet |
[production] |
| 06:22 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:19 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:00 |
<marostegui> |
Stop MySQL on labsdb1011 for reimage - T249188 |
[production] |
| 05:58 |
<moritzm> |
installing git security updates on jessie |
[production] |
| 05:56 |
<marostegui> |
Compress tables on db1104 - T232446 |
[production] |
| 05:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1104 for defragmentation - T232446', diff saved to https://phabricator.wikimedia.org/P11039 and previous config saved to /var/cache/conftool/dbconfig/20200427-055320-marostegui.json |
[production] |
| 05:47 |
<vgutierrez> |
rolling restart ats-tls in cp[1085,1089] and text@esams - T249335 |
[production] |
| 05:33 |
<marostegui> |
Depool labsdb1011 T249188 |
[production] |
|
2020-04-25
§
|
| 22:31 |
<bd808> |
Bumped Service quota from 3 to 5 (T250922, T246592) |
[tools.moedata] |
| 20:58 |
<wm-bot> |
<lucaswerkmeister> deployed 0dadbb4d4e (toolforge.org) |
[tools.lexeme-forms] |
| 17:45 |
<wm-bot> |
<lucaswerkmeister> deployed 03e5c7016a (fix image.js) |
[tools.wd-image-positions] |
| 17:18 |
<wm-bot> |
<lucaswerkmeister> deployed 5fca9d7d94 (toolforge.org) |
[tools.wd-image-positions] |
| 17:09 |
<wm-bot> |
<lucaswerkmeister> deployed 90342028c5 (toolforge.org) |
[tools.quickcategories] |
| 16:45 |
<bd808> |
Bumped Service quota to 3 (T250922, T246592) |
[tools.moedata] |
| 16:43 |
<wm-bot> |
<lucaswerkmeister> deployed e124d91062 (toolforge.org) |
[tools.wd-shex-infer] |
| 16:37 |
<wm-bot> |
<lucaswerkmeister> deployed 225aea85f1 (python3.7) |
[tools.wd-image-positions] |
| 15:52 |
<wm-bot> |
<lucaswerkmeister> deployed 86a2f065e6 (update pygments, use service.template) |
[tools.wd-shex-infer] |
| 15:21 |
<wm-bot> |
<lucaswerkmeister> deployed 6f1bcdf321 (migrate to toolforge.org) |
[tools.wb2rdf] |
| 10:23 |
<addshore> |
going to restart and probably depool for a short time wdqs1005 as it is in a deadlock T242453 |
[production] |
| 08:36 |
<wm-bot> |
<taavi> git pull to 3639cc4 |
[tools.majavah-bot] |
| 05:52 |
<_joe_> |
depooling mw1407 again, should not be serving traffic |
[production] |
| 05:27 |
<shdubsh> |
restart elasticsearch on logstash2022 |
[production] |
|
2020-04-24
§
|
| 22:22 |
<wm-bot> |
<jeanfred> Deploy latest from Git master: 6963040, 35a3ea7, 22e27b5, 9dd44ff, 0e45815, 6f36ffd (T248788) |
[tools.integraality] |
| 21:25 |
<cdanis@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=wdqs,name=eqiad |
[production] |
| 19:41 |
<Amir1> |
applying T114117 on labswiki (wikitech) |
[production] |
| 19:02 |
<wm-bot> |
<jeanfred> Deploy latest from Git master: 9fea19b, f38047e |
[tools.integraality] |
| 18:58 |
<shdubsh> |
restart elasticsearch on logstash2021 |
[production] |
| 18:50 |
<shdubsh> |
restart elasticsearch on logstash2020 |
[production] |
| 15:12 |
<cdanis@cumin1001> |
conftool action : set/pooled=false; selector: dnsdisc=wdqs,name=eqiad |
[production] |
| 15:08 |
<addshore> |
depool and restart wdqs1006 to catch up with lag after deadlock T242453 |
[production] |
| 11:13 |
<Amir1> |
apply T250071 on s10 (labswiki) |
[production] |