2401-2450 of 10000 results (90ms)
2024-02-20 ยง
20:01 <ryankemper@cumin2002> START - Cookbook sre.hosts.decommission for hosts cloudelastic[1001-1004].wikimedia.org [production]
19:53 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1232 (T355609)', diff saved to https://phabricator.wikimedia.org/P57417 and previous config saved to /var/cache/conftool/dbconfig/20240220-195303-marostegui.json [production]
19:52 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1232.eqiad.wmnet with reason: Maintenance [production]
19:52 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1232.eqiad.wmnet with reason: Maintenance [production]
19:52 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1228 (T355609)', diff saved to https://phabricator.wikimedia.org/P57416 and previous config saved to /var/cache/conftool/dbconfig/20240220-195242-marostegui.json [production]
19:48 <ryankemper@cumin2002> END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T347624, testing 961878 patch) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet w/ force delete existing files, repooling source-only afterwards [production]
19:48 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T347624, testing 961878 patch) xfer categories from wdqs2024.codfw.wmnet -> wdqs2025.codfw.wmnet w/ force delete existing files, repooling source-only afterwards [production]
19:43 <ryankemper@cumin2002> START - Cookbook sre.wdqs.restart [production]
19:38 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2192 (T357189)', diff saved to https://phabricator.wikimedia.org/P57415 and previous config saved to /var/cache/conftool/dbconfig/20240220-193842-arnaudb.json [production]
19:37 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P57414 and previous config saved to /var/cache/conftool/dbconfig/20240220-193735-marostegui.json [production]
19:36 <cdanis@deploy2002> helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
19:35 <cdanis@deploy2002> helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
19:23 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P57413 and previous config saved to /var/cache/conftool/dbconfig/20240220-192335-arnaudb.json [production]
19:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P57412 and previous config saved to /var/cache/conftool/dbconfig/20240220-192229-marostegui.json [production]
19:12 <jhuneidi@deploy2002> rebuilt and synchronized wikiversions files: group0 wikis to 1.42.0-wmf.19 refs T354437 [production]
19:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P57411 and previous config saved to /var/cache/conftool/dbconfig/20240220-190829-arnaudb.json [production]
19:07 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1228 (T355609)', diff saved to https://phabricator.wikimedia.org/P57410 and previous config saved to /var/cache/conftool/dbconfig/20240220-190722-marostegui.json [production]
18:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2192 (T357189)', diff saved to https://phabricator.wikimedia.org/P57409 and previous config saved to /var/cache/conftool/dbconfig/20240220-185322-arnaudb.json [production]
18:49 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2192 (T357189)', diff saved to https://phabricator.wikimedia.org/P57408 and previous config saved to /var/cache/conftool/dbconfig/20240220-184925-arnaudb.json [production]
18:49 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance [production]
18:49 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance [production]
18:49 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2178 (T357189)', diff saved to https://phabricator.wikimedia.org/P57407 and previous config saved to /var/cache/conftool/dbconfig/20240220-184903-arnaudb.json [production]
18:42 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1228 (T355609)', diff saved to https://phabricator.wikimedia.org/P57406 and previous config saved to /var/cache/conftool/dbconfig/20240220-184157-marostegui.json [production]
18:41 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1228.eqiad.wmnet with reason: Maintenance [production]
18:41 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1228.eqiad.wmnet with reason: Maintenance [production]
18:41 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1219 (T355609)', diff saved to https://phabricator.wikimedia.org/P57405 and previous config saved to /var/cache/conftool/dbconfig/20240220-184124-marostegui.json [production]
18:33 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P57404 and previous config saved to /var/cache/conftool/dbconfig/20240220-183356-arnaudb.json [production]
18:31 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be) [production]
18:31 <sukhe> pool cp4052: bookworm cp host with haproxy 2.6 built against OpenSSL 1.1.1: T352744 [production]
18:26 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P57403 and previous config saved to /var/cache/conftool/dbconfig/20240220-182617-marostegui.json [production]
18:22 <sukhe> reprepro -C component/haproxy26 include bookworm-wikimedia haproxy_2.6.16-1~bpo12+1_amd64.changes: T352744 [production]
18:18 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P57402 and previous config saved to /var/cache/conftool/dbconfig/20240220-181850-arnaudb.json [production]
18:11 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P57401 and previous config saved to /var/cache/conftool/dbconfig/20240220-181111-marostegui.json [production]
18:03 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2178 (T357189)', diff saved to https://phabricator.wikimedia.org/P57400 and previous config saved to /var/cache/conftool/dbconfig/20240220-180342-arnaudb.json [production]
17:59 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2178 (T357189)', diff saved to https://phabricator.wikimedia.org/P57399 and previous config saved to /var/cache/conftool/dbconfig/20240220-175938-arnaudb.json [production]
17:59 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance [production]
17:59 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2178.codfw.wmnet with reason: Maintenance [production]
17:59 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2171 (T357189)', diff saved to https://phabricator.wikimedia.org/P57398 and previous config saved to /var/cache/conftool/dbconfig/20240220-175917-arnaudb.json [production]
17:56 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1219 (T355609)', diff saved to https://phabricator.wikimedia.org/P57397 and previous config saved to /var/cache/conftool/dbconfig/20240220-175605-marostegui.json [production]
17:44 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P57396 and previous config saved to /var/cache/conftool/dbconfig/20240220-174411-arnaudb.json [production]
17:29 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P57395 and previous config saved to /var/cache/conftool/dbconfig/20240220-172904-arnaudb.json [production]
17:27 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1219 (T355609)', diff saved to https://phabricator.wikimedia.org/P57394 and previous config saved to /var/cache/conftool/dbconfig/20240220-172716-marostegui.json [production]
17:27 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1219.eqiad.wmnet with reason: Maintenance [production]
17:27 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1219.eqiad.wmnet with reason: Maintenance [production]
17:26 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1218 (T355609)', diff saved to https://phabricator.wikimedia.org/P57393 and previous config saved to /var/cache/conftool/dbconfig/20240220-172653-marostegui.json [production]
17:18 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bookworm [production]
17:13 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2171 (T357189)', diff saved to https://phabricator.wikimedia.org/P57392 and previous config saved to /var/cache/conftool/dbconfig/20240220-171358-arnaudb.json [production]
17:11 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P57391 and previous config saved to /var/cache/conftool/dbconfig/20240220-171147-marostegui.json [production]
17:10 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2171 (T357189)', diff saved to https://phabricator.wikimedia.org/P57390 and previous config saved to /var/cache/conftool/dbconfig/20240220-170949-arnaudb.json [production]
17:09 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: Maintenance [production]