5001-5050 of 10000 results (105ms)
2024-05-17 §
08:26 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1201 (T352010)', diff saved to https://phabricator.wikimedia.org/P62574 and previous config saved to /var/cache/conftool/dbconfig/20240517-082636-ladsgroup.json [production]
08:26 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance [production]
08:26 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance [production]
08:26 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1187 (T352010)', diff saved to https://phabricator.wikimedia.org/P62573 and previous config saved to /var/cache/conftool/dbconfig/20240517-082613-ladsgroup.json [production]
08:17 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/services/zotero: apply [production]
08:17 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/services/zotero: apply [production]
08:16 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/services/zotero: apply [production]
08:16 <jayme@deploy1002> helmfile [codfw] START helmfile.d/services/zotero: apply [production]
08:15 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/zotero: apply [production]
08:14 <jayme@deploy1002> helmfile [staging] START helmfile.d/services/zotero: apply [production]
08:11 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P62572 and previous config saved to /var/cache/conftool/dbconfig/20240517-081105-ladsgroup.json [production]
07:55 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P62571 and previous config saved to /var/cache/conftool/dbconfig/20240517-075558-ladsgroup.json [production]
07:40 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1187 (T352010)', diff saved to https://phabricator.wikimedia.org/P62570 and previous config saved to /var/cache/conftool/dbconfig/20240517-074050-ladsgroup.json [production]
06:59 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2167 (T364299)', diff saved to https://phabricator.wikimedia.org/P62568 and previous config saved to /var/cache/conftool/dbconfig/20240517-065920-marostegui.json [production]
06:59 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance [production]
06:59 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance [production]
06:58 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166 (T364299)', diff saved to https://phabricator.wikimedia.org/P62567 and previous config saved to /var/cache/conftool/dbconfig/20240517-065857-marostegui.json [production]
06:43 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P62566 and previous config saved to /var/cache/conftool/dbconfig/20240517-064350-marostegui.json [production]
06:28 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P62565 and previous config saved to /var/cache/conftool/dbconfig/20240517-062842-marostegui.json [production]
06:18 <ryankemper@cumin2002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 55 hosts [production]
06:17 <ryankemper@cumin2002> START - Cookbook sre.hosts.remove-downtime for 55 hosts [production]
06:13 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2166 (T364299)', diff saved to https://phabricator.wikimedia.org/P62564 and previous config saved to /var/cache/conftool/dbconfig/20240517-061334-marostegui.json [production]
06:09 <ryankemper@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: T363975 eqiad cluster restart - ryankemper@cumin2002 - T363975 [production]
05:52 <ryankemper@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: T363975 eqiad cluster restart - ryankemper@cumin2002 - T363975 [production]
05:52 <ryankemper@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 55 hosts with reason: T363975 [production]
05:50 <ryankemper@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on 55 hosts with reason: T363975 [production]
05:17 <marostegui> Restart wikibugs [production]
05:17 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1187 (T352010)', diff saved to https://phabricator.wikimedia.org/P62563 and previous config saved to /var/cache/conftool/dbconfig/20240517-051721-ladsgroup.json [production]
05:17 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance [production]
05:17 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance [production]
05:16 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T352010)', diff saved to https://phabricator.wikimedia.org/P62562 and previous config saved to /var/cache/conftool/dbconfig/20240517-051658-ladsgroup.json [production]
05:06 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
05:05 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance [production]
05:01 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P62561 and previous config saved to /var/cache/conftool/dbconfig/20240517-050150-ladsgroup.json [production]
04:46 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P62560 and previous config saved to /var/cache/conftool/dbconfig/20240517-044642-ladsgroup.json [production]
04:31 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1180 (T352010)', diff saved to https://phabricator.wikimedia.org/P62559 and previous config saved to /var/cache/conftool/dbconfig/20240517-043134-ladsgroup.json [production]
02:12 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1180 (T352010)', diff saved to https://phabricator.wikimedia.org/P62558 and previous config saved to /var/cache/conftool/dbconfig/20240517-021211-ladsgroup.json [production]
02:12 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
02:11 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
02:11 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T352010)', diff saved to https://phabricator.wikimedia.org/P62557 and previous config saved to /var/cache/conftool/dbconfig/20240517-021148-ladsgroup.json [production]
01:56 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P62556 and previous config saved to /var/cache/conftool/dbconfig/20240517-015640-ladsgroup.json [production]
01:41 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P62555 and previous config saved to /var/cache/conftool/dbconfig/20240517-014132-ladsgroup.json [production]
01:26 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1173 (T352010)', diff saved to https://phabricator.wikimedia.org/P62554 and previous config saved to /var/cache/conftool/dbconfig/20240517-012622-ladsgroup.json [production]
2024-05-16 §
23:43 <cwhite> restart apache on gerrit1003 [production]
23:17 <zabe@deploy1002> Synchronized private/PrivateSettings.php: Add secret for encrypting user password hashes - T150647 (duration: 16m 42s) [production]
23:09 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1173 (T352010)', diff saved to https://phabricator.wikimedia.org/P62553 and previous config saved to /var/cache/conftool/dbconfig/20240516-230951-ladsgroup.json [production]
23:09 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]
23:09 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1173.eqiad.wmnet with reason: Maintenance [production]
23:09 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1168 (T352010)', diff saved to https://phabricator.wikimedia.org/P62552 and previous config saved to /var/cache/conftool/dbconfig/20240516-230939-ladsgroup.json [production]
23:05 <ebernhardson@deploy1002> Finished deploy [airflow-dags/search@312e2be]: Correct new range partition sensor granularity (duration: 00m 21s) [production]