4601-4650 of 10000 results (82ms)
2022-11-04 §
06:26 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es4 T322389 [production]
06:11 <marostegui@cumin1001> dbctl commit (dc=all): 'db2121 (re)pooling @ 5%: After schema change', diff saved to https://phabricator.wikimedia.org/P38106 and previous config saved to /var/cache/conftool/dbconfig/20221104-061157-root.json [production]
05:56 <marostegui@cumin1001> dbctl commit (dc=all): 'db2121 (re)pooling @ 3%: After schema change', diff saved to https://phabricator.wikimedia.org/P38105 and previous config saved to /var/cache/conftool/dbconfig/20221104-055652-root.json [production]
05:41 <marostegui@cumin1001> dbctl commit (dc=all): 'db2121 (re)pooling @ 1%: After schema change', diff saved to https://phabricator.wikimedia.org/P38104 and previous config saved to /var/cache/conftool/dbconfig/20221104-054147-root.json [production]
03:24 <ejegg> payments-wiki upgraded from 1c8f522f to 8baa6bb5 [production]
01:29 <andrew@deploy1002> Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 46s) [production]
01:28 <andrew@deploy1002> Started deploy [horizon/deploy@9d02cd6]: (no justification provided) [production]
2022-11-03 §
22:45 <andrew@deploy1002> Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 01m 00s) [production]
22:44 <andrew@deploy1002> Started deploy [horizon/deploy@9d02cd6]: (no justification provided) [production]
22:43 <krinkle@deploy1002> Finished deploy [integration/docroot@44f1640]: (no justification provided) (duration: 00m 29s) [production]
22:42 <krinkle@deploy1002> Started deploy [integration/docroot@44f1640]: (no justification provided) [production]
22:40 <cwhite> logstash eqiad - opensearch 2.2.0 upgrade complete T304440 [production]
22:13 <andrew@deploy1002> Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 02m 07s) [production]
22:13 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
22:13 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
22:13 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200 (T318605)', diff saved to https://phabricator.wikimedia.org/P38102 and previous config saved to /var/cache/conftool/dbconfig/20221103-221329-ladsgroup.json [production]
22:11 <andrew@deploy1002> Started deploy [horizon/deploy@9d02cd6]: (no justification provided) [production]
22:11 <andrew@deploy1002> Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 08s) [production]
22:11 <andrew@deploy1002> Started deploy [horizon/deploy@9d02cd6]: (no justification provided) [production]
21:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P38101 and previous config saved to /var/cache/conftool/dbconfig/20221103-215823-ladsgroup.json [production]
21:51 <andrew@deploy1002> Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 16s) [production]
21:51 <andrew@deploy1002> Started deploy [horizon/deploy@9d02cd6]: (no justification provided) [production]
21:48 <andrew@deploy1002> Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 50s) [production]
21:47 <andrew@deploy1002> Started deploy [horizon/deploy@9d02cd6]: (no justification provided) [production]
21:47 <andrew@deploy1002> Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 05s) [production]
21:47 <andrew@deploy1002> Started deploy [horizon/deploy@9d02cd6]: (no justification provided) [production]
21:46 <andrew@deploy1002> Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 00m 50s) [production]
21:45 <andrew@deploy1002> Started deploy [horizon/deploy@9d02cd6]: (no justification provided) [production]
21:43 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P38100 and previous config saved to /var/cache/conftool/dbconfig/20221103-214317-ladsgroup.json [production]
21:28 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1200 (T318605)', diff saved to https://phabricator.wikimedia.org/P38099 and previous config saved to /var/cache/conftool/dbconfig/20221103-212810-ladsgroup.json [production]
21:26 <andrew@deploy1002> Finished deploy [horizon/deploy@9d02cd6]: (no justification provided) (duration: 02m 31s) [production]
21:24 <andrew@deploy1002> Started deploy [horizon/deploy@9d02cd6]: (no justification provided) [production]
21:22 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. [production]
21:22 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. [production]
21:22 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. [production]
21:22 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. [production]
21:19 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. [production]
21:19 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. [production]
21:19 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. [production]
21:19 <jhathaway@deploy1002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. [production]
21:08 <ryankemper> [WCQS] Pooled `wcqs100[1,2]` [production]
21:04 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
21:03 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
21:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
21:02 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
20:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1200 (T318605)', diff saved to https://phabricator.wikimedia.org/P38098 and previous config saved to /var/cache/conftool/dbconfig/20221103-205855-ladsgroup.json [production]
20:58 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance [production]
20:58 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance [production]
20:58 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1185 (T318605)', diff saved to https://phabricator.wikimedia.org/P38097 and previous config saved to /var/cache/conftool/dbconfig/20221103-205823-ladsgroup.json [production]
20:57 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]