151-200 of 10000 results (70ms)
2024-04-04 ยง
16:35 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance [production]
16:35 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1191 (T360332)', diff saved to https://phabricator.wikimedia.org/P59536 and previous config saved to /var/cache/conftool/dbconfig/20240404-163526-arnaudb.json [production]
16:32 <fabfur@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3068.esams.wmnet with reason: host reimage [production]
16:31 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2213 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59535 and previous config saved to /var/cache/conftool/dbconfig/20240404-163107-arnaudb.json [production]
16:30 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59534 and previous config saved to /var/cache/conftool/dbconfig/20240404-163053-arnaudb.json [production]
16:28 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1203 (T355609)', diff saved to https://phabricator.wikimedia.org/P59533 and previous config saved to /var/cache/conftool/dbconfig/20240404-162801-marostegui.json [production]
16:27 <fabfur@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp3068.esams.wmnet with reason: host reimage [production]
16:20 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P59532 and previous config saved to /var/cache/conftool/dbconfig/20240404-162019-arnaudb.json [production]
16:16 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2213 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59531 and previous config saved to /var/cache/conftool/dbconfig/20240404-161601-arnaudb.json [production]
16:15 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59530 and previous config saved to /var/cache/conftool/dbconfig/20240404-161547-arnaudb.json [production]
16:10 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
16:09 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
16:05 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P59529 and previous config saved to /var/cache/conftool/dbconfig/20240404-160508-arnaudb.json [production]
16:04 <jforrester@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
16:04 <fabfur@cumin1002> START - Cookbook sre.hosts.reimage for host cp3068.esams.wmnet with OS bullseye [production]
16:03 <jforrester@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
16:03 <jforrester@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
16:02 <jforrester@deploy1002> helmfile [codfw] START helmfile.d/services/wikifunctions: apply [production]
16:01 <jforrester@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [production]
16:01 <jforrester@deploy1002> helmfile [staging] START helmfile.d/services/wikifunctions: apply [production]
16:00 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2213 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59528 and previous config saved to /var/cache/conftool/dbconfig/20240404-160055-arnaudb.json [production]
16:00 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59527 and previous config saved to /var/cache/conftool/dbconfig/20240404-160041-arnaudb.json [production]
15:55 <fabfur@cumin1002> conftool action : set/pooled=no; selector: name=cp3068.esams.wmnet [production]
15:54 <fabfur> depooling cp3068 for reimage (T360430) [production]
15:54 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1207 (T356166)', diff saved to https://phabricator.wikimedia.org/P59526 and previous config saved to /var/cache/conftool/dbconfig/20240404-155420-marostegui.json [production]
15:54 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1207.eqiad.wmnet with reason: Maintenance [production]
15:54 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1207.eqiad.wmnet with reason: Maintenance [production]
15:53 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1206 (T356166)', diff saved to https://phabricator.wikimedia.org/P59525 and previous config saved to /var/cache/conftool/dbconfig/20240404-155357-marostegui.json [production]
15:50 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1191 (T360332)', diff saved to https://phabricator.wikimedia.org/P59524 and previous config saved to /var/cache/conftool/dbconfig/20240404-155000-arnaudb.json [production]
15:47 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1191 (T360332)', diff saved to https://phabricator.wikimedia.org/P59523 and previous config saved to /var/cache/conftool/dbconfig/20240404-154730-arnaudb.json [production]
15:47 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1191.eqiad.wmnet with reason: Maintenance [production]
15:47 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1191.eqiad.wmnet with reason: Maintenance [production]
15:47 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1174 (T360332)', diff saved to https://phabricator.wikimedia.org/P59522 and previous config saved to /var/cache/conftool/dbconfig/20240404-154707-arnaudb.json [production]
15:45 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2213 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59521 and previous config saved to /var/cache/conftool/dbconfig/20240404-154549-arnaudb.json [production]
15:45 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59520 and previous config saved to /var/cache/conftool/dbconfig/20240404-154535-arnaudb.json [production]
15:45 <ayounsi@cumin1002> END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 108 [production]
15:45 <ayounsi@cumin1002> START - Cookbook sre.network.debug for Netbox circuit ID 108 [production]
15:38 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P59519 and previous config saved to /var/cache/conftool/dbconfig/20240404-153850-marostegui.json [production]
15:36 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1203 (T355609)', diff saved to https://phabricator.wikimedia.org/P59518 and previous config saved to /var/cache/conftool/dbconfig/20240404-153626-marostegui.json [production]
15:36 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2214.codfw.wmnet with reason: depooled, see T361851 [production]
15:36 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance [production]
15:36 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db2214.codfw.wmnet with reason: depooled, see T361851 [production]
15:36 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance [production]
15:36 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1193 (T355609)', diff saved to https://phabricator.wikimedia.org/P59517 and previous config saved to /var/cache/conftool/dbconfig/20240404-153603-marostegui.json [production]
15:32 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P59516 and previous config saved to /var/cache/conftool/dbconfig/20240404-153200-arnaudb.json [production]
15:30 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2213 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59515 and previous config saved to /var/cache/conftool/dbconfig/20240404-153043-arnaudb.json [production]
15:30 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2207 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59514 and previous config saved to /var/cache/conftool/dbconfig/20240404-153030-arnaudb.json [production]
15:30 <sukhe@cumin2002> dbctl commit (dc=all): 'depool db2214', diff saved to https://phabricator.wikimedia.org/P59513 and previous config saved to /var/cache/conftool/dbconfig/20240404-153023-sukhe.json [production]
15:23 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P59512 and previous config saved to /var/cache/conftool/dbconfig/20240404-152341-marostegui.json [production]
15:23 <jgiannelos@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [production]