3551-3600 of 10000 results (50ms)
2022-01-27 ยง
16:51 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventstreams: sync on production [production]
16:50 <otto@deploy1002> helmfile [codfw] START helmfile.d/services/eventstreams: apply on canary [production]
16:50 <otto@deploy1002> helmfile [codfw] START helmfile.d/services/eventstreams: apply on production [production]
16:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P19497 and previous config saved to /var/cache/conftool/dbconfig/20220127-164354-marostegui.json [production]
16:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P19496 and previous config saved to /var/cache/conftool/dbconfig/20220127-162849-marostegui.json [production]
16:27 <otto@deploy1002> helmfile [staging] DONE helmfile.d/services/eventstreams: sync on production [production]
16:27 <otto@deploy1002> helmfile [staging] DONE helmfile.d/services/eventstreams: apply on canary [production]
16:27 <otto@deploy1002> helmfile [staging] START helmfile.d/services/eventstreams: apply on production [production]
16:24 <otto@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventstreams-internal: sync on main [production]
16:24 <hnowlan@cumin1001> END (FAIL) - Cookbook sre.postgresql.postgres-init (exit_code=99) [production]
16:23 <otto@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventstreams-internal: apply on canary [production]
16:23 <otto@deploy1002> helmfile [eqiad] START helmfile.d/services/eventstreams-internal: apply on main [production]
16:22 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventstreams-internal: sync on main [production]
16:21 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventstreams-internal: apply on canary [production]
16:21 <otto@deploy1002> helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply on main [production]
16:20 <hnowlan@cumin1001> START - Cookbook sre.postgresql.postgres-init [production]
16:20 <otto@deploy1002> helmfile [staging] DONE helmfile.d/services/eventstreams-internal: sync on main [production]
16:19 <hnowlan@cumin1001> END (PASS) - Cookbook sre.postgresql.postgres-init (exit_code=0) [production]
16:19 <otto@deploy1002> helmfile [staging] DONE helmfile.d/services/eventstreams-internal: apply on canary [production]
16:19 <otto@deploy1002> helmfile [staging] START helmfile.d/services/eventstreams-internal: apply on main [production]
16:15 <otto@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: sync on production [production]
16:15 <otto@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: sync on canary [production]
16:14 <otto@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply on production [production]
16:14 <otto@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply on canary [production]
16:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T298559)', diff saved to https://phabricator.wikimedia.org/P19495 and previous config saved to /var/cache/conftool/dbconfig/20220127-161344-marostegui.json [production]
16:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1099:3311 (T298559)', diff saved to https://phabricator.wikimedia.org/P19494 and previous config saved to /var/cache/conftool/dbconfig/20220127-161239-marostegui.json [production]
16:12 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance [production]
16:12 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance [production]
16:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135 (T298559)', diff saved to https://phabricator.wikimedia.org/P19493 and previous config saved to /var/cache/conftool/dbconfig/20220127-161231-marostegui.json [production]
16:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T285149)', diff saved to https://phabricator.wikimedia.org/P19492 and previous config saved to /var/cache/conftool/dbconfig/20220127-160749-marostegui.json [production]
16:03 <dcausse> restarting blazegraph on wdqs1005 (jvm stuck for 2hours) [production]
16:01 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
16:00 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
16:00 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
15:59 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
15:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P19491 and previous config saved to /var/cache/conftool/dbconfig/20220127-155726-marostegui.json [production]
15:57 <brennen@deploy1002> Synchronized php: group1 wikis to 1.38.0-wmf.19 refs T293960 (duration: 00m 51s) [production]
15:56 <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.19 refs T293960 [production]
15:54 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: sync on canary [production]
15:53 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: sync on production [production]
15:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P19490 and previous config saved to /var/cache/conftool/dbconfig/20220127-155244-marostegui.json [production]
15:52 <otto@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply on production [production]
15:52 <otto@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply on canary [production]
15:52 <brennen> train 1.38.0-wmf.19 (T293960): no current blockers; rolling train forward to group1 before log triage meeting [production]
15:45 <otto@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-analytics: sync on production [production]
15:45 <otto@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-analytics: sync on canary [production]
15:45 <otto@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-analytics: apply on canary [production]
15:45 <otto@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-analytics: apply on production [production]
15:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P19489 and previous config saved to /var/cache/conftool/dbconfig/20220127-154222-marostegui.json [production]
15:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P19488 and previous config saved to /var/cache/conftool/dbconfig/20220127-153739-marostegui.json [production]