1-50 of 10000 results (35ms)
2021-06-02 ยง
23:57 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs2007.codfw.wmnet --dest wdqs2003.codfw.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph` on `ryankemper@cumin2002` tmux session `wdqs_reimage` [production]
23:57 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer [production]
23:56 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1004.eqiad.wmnet --dest wdqs1003.eqiad.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories` on `ryankemper@cumin1001` tmux session `wdqs_reimage` [production]
23:56 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
23:53 <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
23:47 <ryankemper> T280382 `wdqs1004.eqiad.wmnet` has been re-imaged and had the appropriate wikidata/categories journal files transferred. `df -h` shows disk space is no longer an issue following the switch to `raid0`: `/dev/md2 2.9T 998G 1.8T 36% /srv` [production]
23:41 <ladsgroup@deploy1002> scap failed: average error rate on 4/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) [production]
23:38 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
23:28 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs2007.codfw.wmnet --dest wdqs2003.codfw.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories` on `ryankemper@cumin2002` tmux session `wdqs_reimage` [production]
23:28 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer [production]
23:26 <ryankemper> T280382 `wdqs2007.codfw.wmnet` has been re-imaged and had the appropriate wikidata/categories journal files transferred. `df -h` shows disk space is no longer an issue following the switch to `raid0`: `/dev/mapper/vg0-srv 2.7T 998G 1.6T 39% /srv` [production]
23:24 <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
23:18 <ladsgroup@deploy1002> Synchronized php-1.37.0-wmf.7/includes: Backport: [[gerrit:697817|Allow html form field option 'options-messages' to get parsed (T58633)]] (duration: 01m 01s) [production]
22:56 <ryankemper@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2003.codfw.wmnet with reason: REIMAGE [production]
22:54 <ryankemper@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2003.codfw.wmnet with reason: REIMAGE [production]
22:48 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:697855|Enable wgVectorConsolidateUserLinks on the beta cluster (T266536)]] (duration: 00m 57s) [production]
22:39 <ryankemper> T280382 `sudo -i wmf-auto-reimage-host -p T280382 --new wdqs2003.codfw.wmnet` on `ryankemper@cumin2002` tmux session `wdqs_reimage` [production]
22:34 <ryankemper> T280382 Cleaned up no-longer-needed files removed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/697832 => `ryankemper@cumin1001:~$ sudo -E cumin -b 2 'P{apt*}' 'sudo rm -rfv /srv/tftpboot/buster-raid0-installer/pxelinux.cfg'` [production]
22:30 <ryankemper> T280382 Cleaned up no-longer-needed files removed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/697832 => `ryankemper@cumin1001:~$ sudo -E cumin -b 6 'P{install*}' 'sudo rm -fv /srv/tftpboot/buster-raid0-installer/pxelinux.cfg'` [production]
22:27 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1003.eqiad.wmnet with reason: REIMAGE [production]
22:25 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1003.eqiad.wmnet with reason: REIMAGE [production]
22:19 <Amir1> setting charset of all tables in wikitech to binary (T284108 T269348) [production]
22:11 <ryankemper> T280382 `sudo -i wmf-auto-reimage-host -p T280382 --new wdqs1003.eqiad.wmnet` on `ryankemper@cumin1001` tmux session `wdqs_reimage_2` [production]
22:11 <ryankemper> T280382 `sudo -i wmf-auto-reimage-host -p T280382 wdqs1003.eqiad.wmnet` on `ryankemper@cumin1001` tmux session `wdqs_reimage_2` [production]
22:08 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
22:07 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer [production]
22:07 <ryankemper@puppetmaster1001> conftool action : set/pooled=yes; selector: name=wdqs1004.eqiad.wmnet [production]
22:07 <ryankemper@puppetmaster1001> conftool action : set/pooled=yes; selector: name=wdqs2007.codfw.wmnet [production]
22:05 <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
22:01 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
21:59 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest wdqs2007.codfw.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories` on `ryankemper@cumin2002` tmux session `wdqs_reimage` [production]
21:59 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer [production]
21:56 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1004.eqiad.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories` on `ryankemper@cumin1001` tmux session `wdqs_reimage` [production]
21:55 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
21:39 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1004.eqiad.wmnet with reason: REIMAGE [production]
21:38 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh3002.wikimedia.org [production]
21:37 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1004.eqiad.wmnet with reason: REIMAGE [production]
21:32 <ryankemper@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2007.codfw.wmnet with reason: REIMAGE [production]
21:30 <ryankemper@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2007.codfw.wmnet with reason: REIMAGE [production]
21:28 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host doh3002.wikimedia.org [production]
21:21 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh3001.wikimedia.org [production]
21:19 <ryankemper@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=wdqs2007.codfw.wmnet [production]
21:17 <ryankemper> `ryankemper@wdqs1013:~$ sudo depool` (catching up on 17.9h lag) [production]
21:12 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host doh3001.wikimedia.org [production]
21:10 <ryankemper> T280382 T281437 `sudo -i wmf-auto-reimage-host -p T280382 --new wdqs2007.codfw.wmnet` on `ryankemper@cumin2002` tmux session `wdqs_reimage` [production]
21:10 <ryankemper> T280382 `sudo -i wmf-auto-reimage-host -p T280382 --new wdqs1004.eqiad.wmnet` on `ryankemper@cumin1001` tmux session `reimage` [production]
20:58 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts doh3001.wikimedia.org [production]
20:49 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission for hosts doh3001.wikimedia.org [production]
20:27 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host doh3002.wikimedia.org [production]
20:21 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host doh3002.wikimedia.org [production]