101-150 of 809 results (4ms)
2021-10-15 §
13:32 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
13:32 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
13:24 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
13:19 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
11:49 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
10:46 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
09:15 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
09:05 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
07:41 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
02:14 <ryankemper> T288231 `wdqs2006` data transfer complete and all tests passing on the host. All of `codfw wdqs-internal` is on the new streaming updater [production]
00:09 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
2021-10-14 §
22:35 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
22:33 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) [production]
22:33 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
22:32 <ryankemper> T288231 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/730795; proceeding to data-transfer on `wdqs2006`: `sudo rm -fv /srv/wdqs/data_loaded` on `wdqs2006` followed by `ryankemper@cumin1001:~$ sudo cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs2006.codfw.wmnet --reason "streaming updater cutover for wdqs2005" --blazegraph_instance blazegraph --task-id T288231` [production]
22:28 <ryankemper> T288231 `ryankemper@wdqs2005:~$ sudo pool`: transfer completed successfully; tests passing on host (used `ssh -L 9999:localhost:80 wdqs2005.codfw.wmnet` to establish tunnel) [production]
18:17 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
16:44 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
16:44 <ryankemper> T288231 Manually killed dangling `pigz` / `nc` processes on `wdqs2008` (and `wdqs2005` implicitly). Should be in the right state to re-start the `data-transfer` cookbook from again [production]
16:41 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) [production]
16:36 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
16:34 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
16:08 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
16:07 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) [production]
16:07 <ryankemper> T288231 About to ctrl+c out of ongoing data transfer because puppet run following merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/730794 restarted blazegraph; we'll manually disable updater and kick off the transfer again [production]
15:56 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
2021-10-08 §
04:18 <gehel@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
04:17 <gehel@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
2021-10-01 §
13:12 <gehel@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
13:11 <gehel@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) [production]
13:11 <gehel@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
13:10 <gehel@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
2021-08-12 §
06:46 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
05:15 <ryankemper> [WDQS] `sudo -i cookbook sre.wdqs.data-transfer --source wdqs2005.codfw.wmnet --dest wdqs2004.codfw.wmnet --reason "transferring fresh wikidata journal after nuking wdqs2004's" --blazegraph_instance blazegraph` [production]
05:15 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
2021-08-10 §
18:45 <ryankemper> T288501 `data-transfer` of `wikidata.jnl` completed successfully. Host needs to catch up on ~22 hours of WDQS lag before being re-pooled [production]
18:42 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
17:19 <ryankemper> T288501 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs2005.codfw.wmnet --dest wdqs2003.codfw.wmnet --reason "transferring fresh wikidata journal to resolve disk issue" --blazegraph_instance blazegraph` on `cumin2001` tmux session `wdqs_data_xfer` [production]
17:19 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
2021-07-22 §
06:20 <ryankemper> [WDQS] Pooled `wdqs1006` (was still depooled following data-transfer cookbook runs from several hours ago) [production]
2021-07-21 §
22:44 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
22:41 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1009.eqiad.wmnet --reason "transferring fresh categories journal to resolve categories update lag unknown alert status" --blazegraph_instance categories --without-lvs` on `ryankemper@cumin1001` tmux session `wdqs` [production]
22:41 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
22:40 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
22:37 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1010.eqiad.wmnet --reason "transferring fresh categories journal to resolve categories update lag unknown alert status" --blazegraph_instance categories --without-lvs` on `ryankemper@cumin1001` tmux session `wdqs` [production]
22:37 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
22:36 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]
22:36 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
2021-06-09 §
04:27 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
02:58 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]