4151-4200 of 10000 results (128ms)
2022-11-14 ยง
15:27 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1180.eqiad.wmnet with reason: Maintenance [production]
15:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321126)', diff saved to https://phabricator.wikimedia.org/P39520 and previous config saved to /var/cache/conftool/dbconfig/20221114-152728-marostegui.json [production]
15:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39519 and previous config saved to /var/cache/conftool/dbconfig/20221114-152356-ladsgroup.json [production]
15:21 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P39518 and previous config saved to /var/cache/conftool/dbconfig/20221114-152148-ladsgroup.json [production]
15:15 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
15:15 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
15:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39516 and previous config saved to /var/cache/conftool/dbconfig/20221114-151428-marostegui.json [production]
15:13 <urandom> initiating Cassandra bootstrap, aqs1019-a -- T307802 [production]
15:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P39515 and previous config saved to /var/cache/conftool/dbconfig/20221114-151222-marostegui.json [production]
15:11 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
15:10 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
15:09 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti1021.eqiad.wmnet with reason: host reimage [production]
15:08 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P39514 and previous config saved to /var/cache/conftool/dbconfig/20221114-150850-ladsgroup.json [production]
15:06 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39513 and previous config saved to /var/cache/conftool/dbconfig/20221114-150642-ladsgroup.json [production]
15:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39512 and previous config saved to /var/cache/conftool/dbconfig/20221114-150531-ladsgroup.json [production]
15:05 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti1021.eqiad.wmnet with reason: host reimage [production]
15:05 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
15:05 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance [production]
15:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T322618)', diff saved to https://phabricator.wikimedia.org/P39511 and previous config saved to /var/cache/conftool/dbconfig/20221114-150509-ladsgroup.json [production]
14:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P39510 and previous config saved to /var/cache/conftool/dbconfig/20221114-145921-marostegui.json [production]
14:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P39509 and previous config saved to /var/cache/conftool/dbconfig/20221114-145715-marostegui.json [production]
14:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2175 (T322618)', diff saved to https://phabricator.wikimedia.org/P39508 and previous config saved to /var/cache/conftool/dbconfig/20221114-145343-ladsgroup.json [production]
14:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2175 (T322618)', diff saved to https://phabricator.wikimedia.org/P39507 and previous config saved to /var/cache/conftool/dbconfig/20221114-145122-ladsgroup.json [production]
14:51 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance [production]
14:51 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti1021.eqiad.wmnet with OS bullseye [production]
14:51 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance [production]
14:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39506 and previous config saved to /var/cache/conftool/dbconfig/20221114-145101-ladsgroup.json [production]
14:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39505 and previous config saved to /var/cache/conftool/dbconfig/20221114-145003-ladsgroup.json [production]
14:49 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ganeti1021.eqiad.wmnet with reason: Remove from cluster for eventual reimage [production]
14:49 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on ganeti1021.eqiad.wmnet with reason: Remove from cluster for eventual reimage [production]
14:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1148 (T321130)', diff saved to https://phabricator.wikimedia.org/P39504 and previous config saved to /var/cache/conftool/dbconfig/20221114-144415-marostegui.json [production]
14:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1168 (T321126)', diff saved to https://phabricator.wikimedia.org/P39503 and previous config saved to /var/cache/conftool/dbconfig/20221114-144209-marostegui.json [production]
14:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1168 (T321126)', diff saved to https://phabricator.wikimedia.org/P39502 and previous config saved to /var/cache/conftool/dbconfig/20221114-143957-marostegui.json [production]
14:39 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance [production]
14:39 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on db1168.eqiad.wmnet with reason: Maintenance [production]
14:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T321126)', diff saved to https://phabricator.wikimedia.org/P39501 and previous config saved to /var/cache/conftool/dbconfig/20221114-143936-marostegui.json [production]
14:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39500 and previous config saved to /var/cache/conftool/dbconfig/20221114-143554-ladsgroup.json [production]
14:34 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P39499 and previous config saved to /var/cache/conftool/dbconfig/20221114-143456-ladsgroup.json [production]
14:33 <taavi> ^ correction, starting it on mwmaint1002, not deploy1002 [production]
14:32 <taavi> START taavi@deploy1002:~$ foreachwikiindblist group1 extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all | tee T315510.log # T315510 [production]
14:30 <taavi@deploy1002> Finished scap: Backport for [[gerrit:856566|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353)]] (duration: 07m 05s) [production]
14:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P39498 and previous config saved to /var/cache/conftool/dbconfig/20221114-142429-marostegui.json [production]
14:23 <taavi@deploy1002> taavi and matmarex: Backport for [[gerrit:856566|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet [production]
14:23 <taavi@deploy1002> Started scap: Backport for [[gerrit:856566|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353)]] [production]
14:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2170:3312', diff saved to https://phabricator.wikimedia.org/P39497 and previous config saved to /var/cache/conftool/dbconfig/20221114-142048-ladsgroup.json [production]
14:20 <taavi@deploy1002> Finished scap: Backport for [[gerrit:856550|Use legacy DiscussionTools heading markup except on beta cluster (T314714)]], [[gerrit:855744|ThreadItemStore: Handle race conditions when finding/inserting outside of transaction (T322701)]], [[gerrit:855745|persistRevisionThreadItems: Print time taken]] (duration: 06m 14s) [production]
14:19 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1162 (T322618)', diff saved to https://phabricator.wikimedia.org/P39496 and previous config saved to /var/cache/conftool/dbconfig/20221114-141950-ladsgroup.json [production]
14:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1162 (T322618)', diff saved to https://phabricator.wikimedia.org/P39495 and previous config saved to /var/cache/conftool/dbconfig/20221114-141738-ladsgroup.json [production]
14:17 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance [production]
14:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1148 (T321130)', diff saved to https://phabricator.wikimedia.org/P39494 and previous config saved to /var/cache/conftool/dbconfig/20221114-141731-marostegui.json [production]