1601-1650 of 10000 results (77ms)
2022-07-29 §
07:05 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 5%: After maintenance', diff saved to https://phabricator.wikimedia.org/P32105 and previous config saved to /var/cache/conftool/dbconfig/20220729-070509-root.json [production]
06:50 <marostegui@cumin1001> dbctl commit (dc=all): 'db1132 (re)pooling @ 1%: After maintenance', diff saved to https://phabricator.wikimedia.org/P32104 and previous config saved to /var/cache/conftool/dbconfig/20220729-065004-root.json [production]
05:00 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: codfw s8 sanitarium master switch [production]
05:00 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 16 hosts with reason: codfw s8 sanitarium master switch [production]
00:48 <TimStarling> slowly restarting (with batch 1 sleep 5) trafficserver on text caches to fully deploy g 817086 T313578 [production]
2022-07-28 §
22:22 <mforns@deploy1002> Finished deploy [airflow-dags/analytics@9ea9cd1]: (no justification provided) (duration: 00m 09s) [production]
22:21 <mforns@deploy1002> Started deploy [airflow-dags/analytics@9ea9cd1]: (no justification provided) [production]
21:51 <mforns@deploy1002> Finished deploy [airflow-dags/analytics@e8d4704]: (no justification provided) (duration: 00m 09s) [production]
21:51 <mforns@deploy1002> Started deploy [airflow-dags/analytics@e8d4704]: (no justification provided) [production]
21:42 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:41 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:41 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:40 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135 (T312990)', diff saved to https://phabricator.wikimedia.org/P32102 and previous config saved to /var/cache/conftool/dbconfig/20220728-212227-marostegui.json [production]
21:18 <mforns@deploy1002> Finished deploy [airflow-dags/analytics@5ec2435]: (no justification provided) (duration: 00m 09s) [production]
21:18 <mforns@deploy1002> Started deploy [airflow-dags/analytics@5ec2435]: (no justification provided) [production]
21:07 <brennen@deploy1002> Finished deploy [phabricator/deployment@a0f0699]: test deploy to phab2001 (take 2) (duration: 00m 27s) [production]
21:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P32100 and previous config saved to /var/cache/conftool/dbconfig/20220728-210721-marostegui.json [production]
21:06 <brennen@deploy1002> Started deploy [phabricator/deployment@a0f0699]: test deploy to phab2001 (take 2) [production]
21:04 <brennen@deploy1002> Finished deploy [phabricator/deployment@a21dea9]: test deploy to phab2001 (duration: 00m 27s) [production]
21:03 <brennen@deploy1002> Started deploy [phabricator/deployment@a21dea9]: test deploy to phab2001 [production]
20:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P32099 and previous config saved to /var/cache/conftool/dbconfig/20220728-205215-marostegui.json [production]
20:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135 (T312990)', diff saved to https://phabricator.wikimedia.org/P32098 and previous config saved to /var/cache/conftool/dbconfig/20220728-203709-marostegui.json [production]
20:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1135 (T312990)', diff saved to https://phabricator.wikimedia.org/P32097 and previous config saved to /var/cache/conftool/dbconfig/20220728-203446-marostegui.json [production]
20:34 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1135.eqiad.wmnet with reason: Maintenance [production]
20:34 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db1135.eqiad.wmnet with reason: Maintenance [production]
20:34 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
20:33 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
20:33 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 16 hosts with reason: Maintenance [production]
20:33 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 16 hosts with reason: Maintenance [production]
20:33 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2103.codfw.wmnet with reason: Maintenance [production]
20:32 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db2103.codfw.wmnet with reason: Maintenance [production]
20:32 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1140.eqiad.wmnet with reason: Maintenance [production]
20:32 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db1140.eqiad.wmnet with reason: Maintenance [production]
20:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118 (T312990)', diff saved to https://phabricator.wikimedia.org/P32096 and previous config saved to /var/cache/conftool/dbconfig/20220728-203212-marostegui.json [production]
20:18 <thcipriani@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:817263|Register Wikistories streams (T313633)]] (duration: 03m 24s) [production]
20:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P32095 and previous config saved to /var/cache/conftool/dbconfig/20220728-201706-marostegui.json [production]
20:14 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:13 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118', diff saved to https://phabricator.wikimedia.org/P32094 and previous config saved to /var/cache/conftool/dbconfig/20220728-200200-marostegui.json [production]
19:47 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
19:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1118 (T312990)', diff saved to https://phabricator.wikimedia.org/P32093 and previous config saved to /var/cache/conftool/dbconfig/20220728-194654-marostegui.json [production]
19:46 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
19:46 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
19:45 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
19:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1118 (T312990)', diff saved to https://phabricator.wikimedia.org/P32092 and previous config saved to /var/cache/conftool/dbconfig/20220728-194426-marostegui.json [production]
19:44 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1118.eqiad.wmnet with reason: Maintenance [production]
19:44 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db1118.eqiad.wmnet with reason: Maintenance [production]