3051-3100 of 10000 results (33ms)
2020-11-26 §
03:07 <krinkle@deploy1001> Synchronized wmf-config/mc.php: I805699ecfa (duration: 00m 58s) [production]
2020-11-25 §
23:28 <akosiaris@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
22:55 <mutante> mwdebug1003 - scap pull - which rsyncs from deploy1001 and runs php-fpm restart check script (T245757) [production]
22:47 <ejegg> increased Ingenico API call timeout [production]
22:34 <shdubsh> beginning rolling restart of logstash cluster - eqiad [production]
22:23 <akosiaris@cumin1001> START - Cookbook sre.ganeti.makevm [production]
22:14 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
22:14 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
22:12 <akosiaris@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
21:19 <akosiaris@cumin1001> START - Cookbook sre.ganeti.makevm [production]
20:49 <krinkle@deploy1001> Synchronized php-1.36.0-wmf.18/includes/libs/CSSMin.php: I26ed3e5e9a - fix T268308 (duration: 00m 59s) [production]
20:43 <mutante> LDAP added user duminasi to group wmf (T266791) [production]
20:06 <ryankemper@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-restart (exit_code=0) [production]
18:44 <elukey> upload new hive* packages 2.2.3-2 to stretch-wikimedia - thirdparty/bigtop14 component [production]
18:42 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-restart [production]
18:38 <mutante> LDAP adding swagoel to NDA T267314#6625628 [production]
18:31 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-restart (exit_code=99) [production]
18:05 <ryankemper> T268770 [cloudelastic] Thawed writes to cloudelastic cluster following restarts: `/usr/local/bin/mwscript extensions/CirrusSearch/maintenance/FreezeWritesToCluster.php --wiki=enwiki --cluster=cloudelastic --thaw` on `mwmaint1002` [production]
18:01 <ryankemper> [cloudelastic] (forgot to mention this) Thawed writes to cloudelastic cluster following restarts: `/usr/local/bin/mwscript extensions/CirrusSearch/maintenance/FreezeWritesToCluster.php --wiki=enwiki --cluster=cloudelastic --thaw` on `mwmaint1002` [production]
17:58 <ryankemper> T268770 [cloudelastic] restarts complete, service is healthy. This is done. [production]
17:55 <ryankemper> T268770 [cloudelastic] restarts on `cloudelastic1006` complete and all 3 elasticsearch clusters are green, all cloudelastic instances are now complete [production]
17:49 <ryankemper> T268770 [cloudelastic] restarts on `cloudelastic1005` complete and all 3 elasticsearch clusters are green, proceeding to next instance [production]
17:44 <shdubsh> beginning rolling restart of logstash cluster - codfw [production]
17:44 <ryankemper> T268770 [cloudelastic] restarts on `cloudelastic1004` complete and all 3 elasticsearch clusters are green, proceeding to next instance [production]
17:39 <ryankemper> T268770 [cloudelastic] restarts on `cloudelastic1003` complete and all 3 elasticsearch clusters are green, proceeding to next instance [production]
17:39 <ryankemper> T268770 [cloudelastic] restarts on `cloudelastic1002` complete and all 3 elasticsearch clusters are green, proceeding to next instance [production]
17:28 <ryankemper> T268770 [cloudelastic] restarts on `cloudelastic1001` complete and all 3 elasticsearch clusters are green, proceeding to next instance [production]
17:22 <ryankemper> T268770 Freezing writes to cloudelastic in preparation for restarts: `/usr/local/bin/mwscript extensions/CirrusSearch/maintenance/FreezeWritesToCluster.php --wiki=enwiki --cluster=cloudelastic` on `mwmaint1002` [production]
17:09 <ryankemper> T268770 [cloudelastic] Downtimed `cloudelastic100[1-6]` in icinga in preparation for cloudelastic search elasticsearch cluster restart [production]
17:05 <ryankemper> T268770 Begin rolling restart of eqiad cirrus elasticsearch, 3 nodes at a time [production]
17:04 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-restart [production]
17:00 <godog> fail sdk on ms-be2031 [production]
16:49 <godog> clean up sdk1 on / on ms-be2031 [production]
16:46 <elukey> move analytics1066 to C3 - T267065 [production]
16:44 <akosiaris@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
16:21 <mutante> puppetmaster - revoking old and signing new cert for mwdebug1003 [production]
16:11 <elukey> move analytics1065 to C3 - T267065 [production]
16:10 <mutante> shutting down mwdebug1003 - reimaging for T245757 [production]
16:08 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:08 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:02 <moritzm> installing golang-1.7 updates for stretch [production]
15:57 <akosiaris@cumin1001> START - Cookbook sre.ganeti.makevm [production]
15:57 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
15:57 <akosiaris@cumin1001> START - Cookbook sre.ganeti.makevm [production]
15:38 <elukey> move stat1004 to A5 - T267065 [production]
15:37 <akosiaris@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
15:34 <moritzm> removing maps2002 from debmonitor [production]
15:10 <akosiaris@cumin1001> START - Cookbook sre.ganeti.makevm [production]
15:04 <akosiaris@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
15:04 <akosiaris@cumin1001> START - Cookbook sre.ganeti.makevm [production]