901-950 of 10000 results (58ms)
2019-08-19 §
05:50 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2067 from config T230705 (duration: 00m 50s) [production]
05:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2067, will be moved to m1 T230705', diff saved to https://phabricator.wikimedia.org/P8930 and previous config saved to /var/cache/conftool/dbconfig/20190819-054606-marostegui.json [production]
05:29 <elukey> reboot cp2004 due to bnx2x crash (kern.log saved into my home on the host if needed) [production]
2019-08-18 §
08:28 <onimisionipe> running `_cluster/reroute?pretty&explain=true&retry_failed` on eqiad production-search cluster to force allocation of shards [production]
2019-08-16 §
19:48 <sbassett> Deployed security patch for T230576 (ex:MobileFrontend) [production]
18:57 <@> helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . [production]
16:38 <XioNoX> add BGP sessions to Scaleway (AS12876) in esams [production]
16:12 <elukey> upload prometheus-druid-exporter 0.7-1 to stretch/buster-wikimedia [production]
15:42 <elukey> roll restart of druid broker/historicals to pick up new logging/metrics settings [production]
14:39 <onimisionipe> run `bmc-device --cold-reset; echo $?` in elastic2050 hoping it resets mgmt interface -T230597 [production]
14:24 <gehel> rolling reboot of cloudelastic [production]
13:52 <mholloway-shell@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: MachineVision (beta): Request labels targeting Beta Wikidata (duration: 00m 50s) [production]
08:18 <_joe_> stopping php on phab1003, to restart it with systemd [production]
06:50 <_joe_> upgrading envoyproxy across production (http2 CVEs) [production]
02:51 <vgutierrez> repooling cp5002, running compress.so experiment [production]
2019-08-15 §
23:35 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@b4da6e4]: Rollback blazegraph due to T230588 (duration: 09m 48s) [production]
23:25 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@b4da6e4]: Rollback blazegraph due to T230588 [production]
21:54 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@fce8177]: Weekly deploy (duration: 25m 28s) [production]
21:28 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@fce8177]: Weekly deploy [production]
21:27 <ebernhardson> finish restarting cloudelastic-chi-eqiad with -XX:NewRatio=3 [production]
21:18 <ebernhardson> increase cloudelastic indices.recovery.max_bytes_per_sec from 40mbit to 512mbit as these have 10G networking [production]
21:07 <ebernhardson> restart cloudelastic1002 with -XX:NewRatio=3 to match cloudelastic1001 [production]
20:22 <gehel@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
19:37 <ema> depool cp5002 during the EU night, running compress.so experiment [production]
19:28 <gehel@cumin1001> END (PASS) - Cookbook sre.wdqs.reboot-wdqs (exit_code=0) [production]
19:19 <sbassett> Deployed security patch for T230402 (1.34.0-wmf.17) [production]
19:18 <gehel@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
19:18 <sbassett> Deployed security patch for T229541 (1.34.0-wmf.17) [production]
19:17 <gehel@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) [production]
19:17 <gehel@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
19:01 <ebernhardson> restart elasticsearch on cloudelastic1001 with -XX:NewRatio=3 [production]
18:51 <gehel@cumin1001> START - Cookbook sre.wdqs.reboot-wdqs [production]
17:58 <mbsantos@deploy1001> Finished deploy [proton/deploy@fb0b2a5]: Update chromium-renderer to 3f1cc72 (T218220) (duration: 00m 43s) [production]
17:58 <mbsantos@deploy1001> Started deploy [proton/deploy@fb0b2a5]: Update chromium-renderer to 3f1cc72 (T218220) [production]
17:47 <mbsantos@deploy1001> Finished deploy [mobileapps/deploy@1bd2bea]: Update service-mobileapp-node to 5c1da03 (T230067 T229984) (duration: 05m 53s) [production]
17:41 <mbsantos@deploy1001> Started deploy [mobileapps/deploy@1bd2bea]: Update service-mobileapp-node to 5c1da03 (T230067 T229984) [production]
17:11 <ejegg> updated payments-wiki from 44eae2d65f to 00eb090dcc [production]
17:02 <cstone> civicrm revision changed from 3caf54a0d2 to 9c7b2ffbc9 [production]
16:53 <reedy@deploy1001> Synchronized docroot/noc/db.php: Use WmfClusters from seperate file (duration: 00m 47s) [production]
16:52 <reedy@deploy1001> Synchronized src/WmfClusters.php: Move WmfClusters.php (duration: 00m 47s) [production]
16:27 <XioNoX> advertise core v4 range (208.80.152.0/22) from eqord - T167841 [production]
16:09 <ori> Finished messing around with mwdebug1002 [production]
16:06 <reedy@deploy1001> Synchronized docroot/: phpcs fixes (duration: 00m 47s) [production]
16:05 <reedy@deploy1001> Synchronized wmf-config/arclamp.php: phpcs (duration: 00m 47s) [production]
16:04 <reedy@deploy1001> Synchronized tests/: phpunit (duration: 00m 47s) [production]
16:03 <reedy@deploy1001> Synchronized phpcs.xml: more exclusions! (duration: 00m 47s) [production]
15:40 <ebernhardson> unfreeze writes to cloudelastic cluster [production]
15:37 <ema> cp5002: re-pool with compress.so cache:false [production]
15:34 <herron> performing rolling restarts of eqiad kafka-main brokers for security updates [production]
15:34 <gehel@cumin1001> END (PASS) - Cookbook sre.wdqs.reboot-wdqs (exit_code=0) [production]