2301-2350 of 10000 results (45ms)
2019-04-14 §
05:36 <ebernhardson> unbanning elastic1027 after about half the shards left and load dropped [production]
05:31 <ebernhardson> ban elastic1027 from elasticsearch-psi in eqiad [production]
04:59 <ebernhardson> restart elasticsearch_6@production-searhc-psi-eqiad on elastic1027 due to 100% cpu for last 30+ minutes [production]
2019-04-13 §
18:46 <godog> 3h downtime for cloudvirt1015 [production]
15:58 <ebernhardson> restart elasticsearch on elastic1027 [production]
15:34 <shdubsh> restart recommendation_api on scb1001 [production]
15:33 <shdubsh> restart recommendation_api on scb2001 [production]
10:46 <onimisionipe> depooling maps2001 for postgres init [production]
08:05 <gehel> repooling wdqs1008 - data transfer completed - T220830 [production]
00:32 <krinkle@deploy1001> Synchronized php-1.33.0-wmf.25/includes/: Idc19cc29764a / T220854 - hot fix (duration: 05m 37s) [production]
2019-04-12 §
21:16 <Krinkle> scap was unable to sync to 1 apache (connect to host cloudweb2001-dev.wikimedia.org port 22: Connection timed out) [production]
21:10 <krinkle@deploy1001> Synchronized php-1.33.0-wmf.25/extensions/ImageMap/includes/ImageMap.php: I0ee84f059da / T217087 (duration: 05m 12s) [production]
19:27 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) [production]
19:27 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
19:24 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) [production]
19:24 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
18:59 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
18:59 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
17:17 <onimisionipe> depooling maps2002 for postgres init [production]
17:16 <onimisionipe> repooling maps2001 - postgres init is complete [production]
16:14 <elukey> install ifstat on all the mc1* hosts for network bandwidth investigation [production]
15:56 <gehel> starting data trasnfer from wdqs1008 to wdqs1009 - T220830 [production]
15:32 <thcipriani> gerrit back [production]
15:29 <thcipriani> gerrit restart incoming [production]
14:29 <onimisionipe> depool maps2001 for postgres initialization [production]
13:24 <akosiaris> re-enable puppet across the fleet. Patch merged, recovery storm coming [production]
13:18 <akosiaris> disable puppet across the fleet to avoid incoming puppet alert storm [production]
12:57 <marostegui> Purge old rows and optimize tables on spare host pc1010 T210725 [production]
12:53 <urandom> decommissioning cassandra-c, restbase2008 -- T208087 [production]
12:49 <gehel> rolling restart of cassandra on maps* for jvm upgrade [production]
12:22 <arturo> T220095 disable icinga checks for labtestcontrol2003 [production]
12:16 <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T220807 Reduce cawiki survey sampling rate (duration: 05m 11s) [production]
11:56 <moritzm> upgrading app server canaries to version 1.8.1 of the PHP wikidiff extension (HHVM already deployed) T203069 [production]
11:46 <moritzm> upgrading acmechief hosts to latest buster state [production]
11:44 <gilles@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T220807 Oversample navtiming on cawiki and commonswiki (duration: 05m 14s) [production]
11:37 <Trey314159> reindexing Greek, Turkish, and Irish wikis on elastic@eqiad and elastic@codfw complete (T217806) [production]
11:19 <moritzm> installed Java security updates on relforge* hosts [production]
11:10 <moritzm> installing Java security updates on remaining maps hosts [production]
10:32 <arturo> T219626 reimaging cloudcontrol2001-dev [production]
10:13 <elukey> matomo updated to 3.9.1 on matomo1001 + deb upload to wikimedia-stretch - T218037 [production]
09:53 <moritzm> updated mwdebug1001 to php-wikidiff 1.8.1 [production]
09:37 <moritzm> updated mwdebug1002 to php-wikidiff 1.8.1 [production]
09:30 <volans> reset mgmt card on labtestcontrol2003 - T220783 [production]
09:07 <moritzm> added the wikimedia repository key to the stretch build chroot on boron, fixes builds using the PHP72/SPICERACK hooks [production]
09:05 <arturo> T218021 disable icinga checks for labtestcontrol2001 [production]
08:35 <gilles@deploy1001> Synchronized php-1.33.0-wmf.25/extensions/NavigationTiming/modules/ext.navigationTiming.js: T220788 Fix veaction === null case (duration: 00m 54s) [production]
08:02 <moritzm> updated ssacli in thirdparty/hwraid component for stretch to 3.30-13.0 T220787 [production]
07:12 <marostegui> Manually install ssacli on db2[097|098|099|100|101|102] T220787 T220572 [production]
07:04 <moritzm> synced ssacli to thirdparty/hwraid components for jessie/stretch T220787 [production]
01:00 <mutante> puppet cert clean, puppet node clean, puppet node deactivate on cloudnet2001-dev.codfw.wmnet (T218025) [production]