1401-1450 of 10000 results (72ms)
2022-07-01 ยง
10:44 <bmansurov@deploy1002> Started deploy [airflow-dags/research@b3fe77c]: (no justification provided) [production]
10:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P30711 and previous config saved to /var/cache/conftool/dbconfig/20220701-103107-ladsgroup.json [production]
10:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1135 (T309311)', diff saved to https://phabricator.wikimedia.org/P30710 and previous config saved to /var/cache/conftool/dbconfig/20220701-102810-ladsgroup.json [production]
10:28 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance [production]
10:27 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance [production]
10:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179 (T309311)', diff saved to https://phabricator.wikimedia.org/P30709 and previous config saved to /var/cache/conftool/dbconfig/20220701-101602-ladsgroup.json [production]
09:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1179 (T309311)', diff saved to https://phabricator.wikimedia.org/P30708 and previous config saved to /var/cache/conftool/dbconfig/20220701-094927-ladsgroup.json [production]
09:49 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance [production]
09:49 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance [production]
09:39 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 13 hosts with reason: Maintenance [production]
09:39 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on 13 hosts with reason: Maintenance [production]
09:39 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance [production]
09:39 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2103.codfw.wmnet with reason: Maintenance [production]
08:35 <marostegui> Stop mysql on db2073 for cloning db2155 [production]
07:47 <mmandere> kubemaster2001, restart rsyslog [production]
07:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db2154 to s8 T311493', diff saved to https://phabricator.wikimedia.org/P30705 and previous config saved to /var/cache/conftool/dbconfig/20220701-074607-marostegui.json [production]
07:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db2153 to s1 T311493', diff saved to https://phabricator.wikimedia.org/P30704 and previous config saved to /var/cache/conftool/dbconfig/20220701-073512-marostegui.json [production]
06:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db2091 from dbctl T311803', diff saved to https://phabricator.wikimedia.org/P30703 and previous config saved to /var/cache/conftool/dbconfig/20220701-060000-marostegui.json [production]
05:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db2092 from dbctl T311802', diff saved to https://phabricator.wikimedia.org/P30701 and previous config saved to /var/cache/conftool/dbconfig/20220701-054102-marostegui.json [production]
02:31 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2165.codfw.wmnet with OS bullseye [production]
02:16 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2165.codfw.wmnet with reason: host reimage [production]
02:13 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2165.codfw.wmnet with reason: host reimage [production]
02:06 <krinkle@deploy1002> Synchronized wmf-config/: I60edfb0f60 (3/3) (duration: 03m 31s) [production]
02:01 <krinkle@deploy1002> Synchronized multiversion/: I60edfb0f60 (2/3) (duration: 03m 34s) [production]
01:54 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host db2165.codfw.wmnet with OS bullseye [production]
01:49 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2163.codfw.wmnet with OS bullseye [production]
01:39 <krinkle@deploy1002> Synchronized tests/: I60edfb0f60 (1/3) (duration: 03m 32s) [production]
01:35 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2163.codfw.wmnet with reason: host reimage [production]
01:33 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
01:32 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
01:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
01:31 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2163.codfw.wmnet with reason: host reimage [production]
01:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
01:30 <krinkle@deploy1002> Synchronized src/: I796f38d0f04600c (3/3) (duration: 03m 24s) [production]
01:26 <krinkle@deploy1002> Synchronized multiversion/: I796f38d0f04600c (2/3) (duration: 03m 32s) [production]
01:23 <krinkle@deploy1002> Synchronized tests/: I796f38d0f04600c (1/3) (duration: 03m 41s) [production]
01:21 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
01:20 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
01:20 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
01:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
01:17 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2162.codfw.wmnet with OS bullseye [production]
01:12 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host db2163.codfw.wmnet with OS bullseye [production]
01:02 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2162.codfw.wmnet with reason: host reimage [production]
01:00 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2161.codfw.wmnet with OS bullseye [production]
00:57 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2162.codfw.wmnet with reason: host reimage [production]
00:53 <ejegg> updated payments-wiki from ef53c82e to 78dee85e [production]
00:51 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2168.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:51 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2167.mgmt.codfw.wmnet with reboot policy FORCED [production]
00:46 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2161.codfw.wmnet with reason: host reimage [production]
00:42 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2161.codfw.wmnet with reason: host reimage [production]