5101-5150 of 10000 results (70ms)
2022-07-04 §
10:18 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
10:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
10:17 <_joe_> upgraded etcdmirror to 0.0.7 on conf2006, now going with the rest of codfw [production]
10:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
08:24 <marostegui@cumin2002> dbctl commit (dc=all): 'Add db2157 to s5 T311493', diff saved to https://phabricator.wikimedia.org/P30758 and previous config saved to /var/cache/conftool/dbconfig/20220704-082406-marostegui.json [production]
08:07 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging MewOphaswongse out of all services on: 634 hosts [production]
08:07 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging MewOphaswongse out of all services on: 634 hosts [production]
08:07 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging MewOphaswongse out of all services on: 1299 hosts [production]
08:06 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging MewOphaswongse out of all services on: 1299 hosts [production]
08:04 <elukey> kill leftover processes of user `mewoph` on stat100x to allow puppet runs [production]
07:39 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cumin1001.eqiad.wmnet [production]
07:28 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host cumin1001.eqiad.wmnet [production]
06:49 <marostegui@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2092.codfw.wmnet [production]
06:47 <marostegui@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:43 <marostegui@cumin2002> START - Cookbook sre.dns.netbox [production]
06:39 <marostegui@cumin2002> START - Cookbook sre.hosts.decommission for hosts db2092.codfw.wmnet [production]
06:34 <marostegui@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2091.codfw.wmnet [production]
06:32 <marostegui@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
06:28 <marostegui@cumin2002> START - Cookbook sre.dns.netbox [production]
06:24 <marostegui@cumin2002> START - Cookbook sre.hosts.decommission for hosts db2091.codfw.wmnet [production]
05:51 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 14 hosts with reason: codfw s4 sanitarium master switch [production]
05:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 14 hosts with reason: codfw s4 sanitarium master switch [production]
2022-07-03 §
11:36 <_joe_> temporarily raised replicas for shellbox to 24 [production]
11:35 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/shellbox: apply [production]
11:35 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/shellbox: apply [production]
2022-07-02 §
05:36 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s) [production]
05:36 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
05:24 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s) [production]
05:23 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
05:21 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s) [production]
05:20 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
05:11 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s) [production]
05:11 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
04:49 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s) [production]
04:49 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
04:48 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s) [production]
04:48 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
03:59 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s) [production]
03:59 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
03:57 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s) [production]
03:57 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
03:56 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 08s) [production]
03:56 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
02:49 <bmansurov@deploy1002> Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s) [production]
02:49 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
01:33 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
01:32 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
00:45 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance [production]
00:45 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance [production]
2022-07-01 §
23:55 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance [production]