1351-1400 of 10000 results (42ms)
2022-07-04 §
08:47 <wm-bot2> Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus [admin]
08:32 <wm-bot2> Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus [admin]
08:32 <wm-bot2> Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
08:25 <wm-bot2> Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
08:25 <wm-bot2> Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
08:24 <marostegui@cumin2002> dbctl commit (dc=all): 'Add db2157 to s5 T311493', diff saved to https://phabricator.wikimedia.org/P30758 and previous config saved to /var/cache/conftool/dbconfig/20220704-082406-marostegui.json [production]
08:19 <wm-bot2> Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
08:19 <wm-bot2> Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus [admin]
08:07 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging MewOphaswongse out of all services on: 634 hosts [production]
08:07 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging MewOphaswongse out of all services on: 634 hosts [production]
08:07 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging MewOphaswongse out of all services on: 1299 hosts [production]
08:06 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging MewOphaswongse out of all services on: 1299 hosts [production]
08:04 <elukey> kill leftover processes of user `mewoph` on stat100x to allow puppet runs [analytics]
08:04 <elukey> kill leftover processes of user `mewoph` on stat100x to allow puppet runs [production]
07:55 <wm-bot2> Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus [admin]
07:51 <wm-bot2> Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus [admin]
07:44 <wm-bot2> Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus [admin]
07:44 <wm-bot2> Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
07:40 <wm-bot2> Rebooting cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
07:40 <wm-bot2> Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
07:39 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cumin1001.eqiad.wmnet [production]
07:36 <wm-bot2> Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
07:36 <wm-bot2> Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus [admin]
07:28 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host cumin1001.eqiad.wmnet [production]
07:09 <wm-bot2> Rebooted cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
07:05 <wm-bot2> Rebooting cloudnet host cloudnet2006-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
07:05 <wm-bot2> Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus [admin]
07:04 <wm-bot2> Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus [admin]
06:49 <marostegui@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2092.codfw.wmnet [production]
06:47 <marostegui@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:43 <marostegui@cumin2002> START - Cookbook sre.dns.netbox [production]
06:39 <marostegui@cumin2002> START - Cookbook sre.hosts.decommission for hosts db2092.codfw.wmnet [production]
06:34 <marostegui@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2091.codfw.wmnet [production]
06:32 <marostegui@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:28 <marostegui@cumin2002> START - Cookbook sre.dns.netbox [production]
06:24 <marostegui@cumin2002> START - Cookbook sre.hosts.decommission for hosts db2091.codfw.wmnet [production]
05:51 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 14 hosts with reason: codfw s4 sanitarium master switch [production]
05:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 14 hosts with reason: codfw s4 sanitarium master switch [production]
2022-07-03 §
21:27 <andrewbogott> rebuilding rabbit cluster in codfw1dev to get rid of some queues so unresponsive that they can't otherwise be deleted [admin]
15:58 <wm-bot> <lucaswerkmeister> deployed e5143b3012 (add nfs-mounted selector, contributed by taavi) using `kubectl edit` [tools.notwikilambda]
11:36 <_joe_> temporarily raised replicas for shellbox to 24 [production]
11:35 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/shellbox: apply [production]
11:35 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/shellbox: apply [production]
2022-07-02 §
19:21 <wm-bot> <lucaswerkmeister> deployed d3e2185bbc (l10n updates) [tools.lexeme-forms]
11:05 <wm-bot2> Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus [admin]
05:56 <taavi> toolsdb: add s54518__mw to list of replication ignored databases, data mismatch between primary and replica [clouddb-services]
05:36 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s) [production]
05:36 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
05:24 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s) [production]
05:23 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]