701-750 of 10000 results (45ms)
2022-03-21 ยง
16:11 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/shellbox-media: apply [production]
16:11 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
16:11 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply [production]
16:11 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/proton: apply [production]
16:11 <pt1979@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1024.eqiad.wmnet with OS bullseye [production]
16:10 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/proton: apply [production]
16:10 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [production]
16:10 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/mobileapps: apply [production]
16:10 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/mathoid: apply [production]
16:09 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/mathoid: apply [production]
16:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
16:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
16:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1129 (T298557)', diff saved to https://phabricator.wikimedia.org/P22901 and previous config saved to /var/cache/conftool/dbconfig/20220321-160557-marostegui.json [production]
16:05 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/eventstreams-internal: apply [production]
16:04 <pt1979@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1024.eqiad.wmnet with OS bullseye [production]
16:02 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/eventstreams: apply [production]
16:02 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply [production]
16:00 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-main: apply [production]
16:00 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply [production]
15:59 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply [production]
15:59 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply [production]
15:58 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply [production]
15:58 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
15:57 <otto@deploy1002> Finished deploy [analytics/refinery@cd7bf7a] (hadoop-test): fix prometheus pushgateway url - T294420 (duration: 07m 18s) [production]
15:57 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply [production]
15:57 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
15:57 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
15:57 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/blubberoid: apply [production]
15:56 <reedy@deploy1002> Synchronized wmf-config/CommonSettings-labs.php: T304111 (duration: 00m 50s) [production]
15:56 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/blubberoid: apply [production]
15:56 <rzl@deploy1002> helmfile [codfw] DONE helmfile.d/services/apertium: apply [production]
15:55 <rzl@deploy1002> helmfile [codfw] START helmfile.d/services/apertium: apply [production]
15:54 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-main: apply [production]
15:54 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply [production]
15:54 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-analytics: apply [production]
15:54 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
15:54 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply [production]
15:54 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
15:54 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
15:54 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/blubberoid: apply [production]
15:54 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/blubberoid: apply [production]
15:54 <rzl@deploy1002> helmfile [staging] DONE helmfile.d/services/apertium: apply [production]
15:54 <rzl@deploy1002> helmfile [staging] START helmfile.d/services/apertium: apply [production]
15:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P22900 and previous config saved to /var/cache/conftool/dbconfig/20220321-155052-marostegui.json [production]
15:50 <otto@deploy1002> Started deploy [analytics/refinery@cd7bf7a] (hadoop-test): fix prometheus pushgateway url - T294420 [production]
15:50 <otto@deploy1002> Finished deploy [analytics/refinery@33f66db] (hadoop-test): fix prometheus pushgateway url - T294420 (duration: 00m 03s) [production]
15:50 <otto@deploy1002> Started deploy [analytics/refinery@33f66db] (hadoop-test): fix prometheus pushgateway url - T294420 [production]
15:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1143 (T300775)', diff saved to https://phabricator.wikimedia.org/P22899 and previous config saved to /var/cache/conftool/dbconfig/20220321-154607-marostegui.json [production]
15:46 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance [production]
15:46 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1143.eqiad.wmnet with reason: Maintenance [production]