6301-6350 of 10000 results (41ms)
2022-01-27 ยง
16:12 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance [production]
16:12 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance [production]
16:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135 (T298559)', diff saved to https://phabricator.wikimedia.org/P19493 and previous config saved to /var/cache/conftool/dbconfig/20220127-161231-marostegui.json [production]
16:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T285149)', diff saved to https://phabricator.wikimedia.org/P19492 and previous config saved to /var/cache/conftool/dbconfig/20220127-160749-marostegui.json [production]
16:03 <dcausse> restarting blazegraph on wdqs1005 (jvm stuck for 2hours) [production]
16:01 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
16:00 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
16:00 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
15:59 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
15:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P19491 and previous config saved to /var/cache/conftool/dbconfig/20220127-155726-marostegui.json [production]
15:57 <brennen@deploy1002> Synchronized php: group1 wikis to 1.38.0-wmf.19 refs T293960 (duration: 00m 51s) [production]
15:56 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.19 refs T293960 [production]
15:54 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: sync on canary [production]
15:53 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: sync on production [production]
15:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P19490 and previous config saved to /var/cache/conftool/dbconfig/20220127-155244-marostegui.json [production]
15:52 <otto@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply on production [production]
15:52 <otto@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply on canary [production]
15:52 <brennen> train 1.38.0-wmf.19 (T293960): no current blockers; rolling train forward to group1 before log triage meeting [production]
15:45 <otto@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-analytics: sync on production [production]
15:45 <otto@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-analytics: sync on canary [production]
15:45 <otto@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-analytics: apply on canary [production]
15:45 <otto@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-analytics: apply on production [production]
15:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P19489 and previous config saved to /var/cache/conftool/dbconfig/20220127-154222-marostegui.json [production]
15:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P19488 and previous config saved to /var/cache/conftool/dbconfig/20220127-153739-marostegui.json [production]
15:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135 (T298559)', diff saved to https://phabricator.wikimedia.org/P19487 and previous config saved to /var/cache/conftool/dbconfig/20220127-152717-marostegui.json [production]
15:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T285149)', diff saved to https://phabricator.wikimedia.org/P19486 and previous config saved to /var/cache/conftool/dbconfig/20220127-152235-marostegui.json [production]
15:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1135 (T298559)', diff saved to https://phabricator.wikimedia.org/P19485 and previous config saved to /var/cache/conftool/dbconfig/20220127-151709-marostegui.json [production]
15:17 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance [production]
15:17 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance [production]
15:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134 (T298559)', diff saved to https://phabricator.wikimedia.org/P19484 and previous config saved to /var/cache/conftool/dbconfig/20220127-151701-marostegui.json [production]
15:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1023 (T300006)', diff saved to https://phabricator.wikimedia.org/P19483 and previous config saved to /var/cache/conftool/dbconfig/20220127-151032-ladsgroup.json [production]
15:09 <dcausse@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync on production [production]
15:08 <dcausse@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync on canary [production]
15:07 <dcausse@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-main: apply on production [production]
15:07 <dcausse@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-main: apply on canary [production]
15:04 <dcausse@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync on production [production]
15:04 <dcausse@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync on canary [production]
15:03 <dcausse@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-main: apply on canary [production]
15:03 <dcausse@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-main: apply on production [production]
15:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P19482 and previous config saved to /var/cache/conftool/dbconfig/20220127-150156-marostegui.json [production]
14:59 <mmandere@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh6002.wikimedia.org [production]
14:58 <dcausse@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-main: sync on production [production]
14:57 <dcausse@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-main: apply on canary [production]
14:57 <dcausse@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-main: apply on production [production]
14:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1023', diff saved to https://phabricator.wikimedia.org/P19481 and previous config saved to /var/cache/conftool/dbconfig/20220127-145527-ladsgroup.json [production]
14:54 <ottomata> continuing deployments of eventgate-main and eventgate-analytics to pick up CA cert changes - T296064 (also deploying eventgate-main for a schema repo bump for search) [production]
14:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P19480 and previous config saved to /var/cache/conftool/dbconfig/20220127-144652-marostegui.json [production]
14:46 <mmandere@cumin1001> START - Cookbook sre.ganeti.makevm for new host doh6002.wikimedia.org [production]
14:40 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.addnode (exit_code=0) for new host ganeti1028.eqiad.wmnet to ganeti01.svc.eqiad.wmnet [production]
14:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1023', diff saved to https://phabricator.wikimedia.org/P19479 and previous config saved to /var/cache/conftool/dbconfig/20220127-144022-ladsgroup.json [production]