901-950 of 10000 results (56ms)
2022-06-29 §
04:39 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
04:37 <tstarling@deploy1002> Synchronized wmf-config/InitialiseSettings.php: wgCentralAuthTokenCacheType -> mcrouter T278392 (duration: 03m 44s) [production]
04:36 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw cluster restart to pickup swift-s3 plugin - ryankemper@cumin1001 - T309648 [production]
00:17 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2003-dev.codfw.wmnet with OS bullseye [production]
2022-06-28 §
23:43 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage [production]
23:39 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage [production]
23:20 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
23:20 <cjming@deploy1002> Synchronized php-1.39.0-wmf.17/extensions/VisualEditor/modules/ve-mw/preinit: Backport: [[gerrit:809308|Do not grey out page title while loading on Vector 2022 (T310839)]] (duration: 03m 28s) [production]
23:20 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host cloudgw2003-dev.codfw.wmnet with OS bullseye [production]
23:19 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
23:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
23:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
23:17 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddb2002-dev.codfw.wmnet with OS bullseye [production]
22:50 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb2002-dev.codfw.wmnet with reason: host reimage [production]
22:47 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on clouddb2002-dev.codfw.wmnet with reason: host reimage [production]
22:27 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host clouddb2002-dev.codfw.wmnet with OS bullseye [production]
22:20 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddb2002-dev.codfw.wmnet with OS bullseye [production]
21:38 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1184 (T298560)', diff saved to https://phabricator.wikimedia.org/P30596 and previous config saved to /var/cache/conftool/dbconfig/20220628-213806-ladsgroup.json [production]
21:38 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance [production]
21:37 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance [production]
21:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132 (T298560)', diff saved to https://phabricator.wikimedia.org/P30595 and previous config saved to /var/cache/conftool/dbconfig/20220628-213735-ladsgroup.json [production]
21:31 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host clouddb2002-dev.codfw.wmnet with OS bullseye [production]
21:30 <pt1979@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host clouddb2002-dev.codfw.wmnet with OS bullseye [production]
21:25 <cjming> end of UTC late backport window [production]
21:22 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P30594 and previous config saved to /var/cache/conftool/dbconfig/20220628-212230-ladsgroup.json [production]
21:20 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:808056|Enable title above tabs on group 1 and group 0 wikis (1/2) (T310054)]] (duration: 03m 34s) [production]
21:19 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:16 <cjming@deploy1002> Synchronized php-1.39.0-wmf.18/extensions/VisualEditor: Backport: [[gerrit:809249|Prevent skinStyles from applying to the Vector 2022 skin. (T310197)]] (duration: 03m 38s) [production]
21:13 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:12 <cjming@deploy1002> Synchronized php-1.39.0-wmf.18/extensions/VisualEditor: Backport: [[gerrit:809249|Prevent skinStyles from applying to the Vector 2022 skin. (T310197)]] (duration: 03m 33s) [production]
21:12 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:08 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host clouddb2002-dev.codfw.wmnet with OS bullseye [production]
21:07 <volans@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts sretest2001.codfw.wmnet [production]
21:07 <volans@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1132', diff saved to https://phabricator.wikimedia.org/P30592 and previous config saved to /var/cache/conftool/dbconfig/20220628-210725-ladsgroup.json [production]
21:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:04 <cjming@deploy1002> Synchronized php-1.39.0-wmf.17/extensions/VisualEditor: Backport: [[gerrit:809245|Prevent skinStyles from applying to the Vector 2022 skin. (T310197)]] (duration: 03m 27s) [production]
21:03 <volans@cumin2002> START - Cookbook sre.dns.netbox [production]
21:02 <volans> deployed spicerack 3.0.0 to cumin1001 [production]
21:00 <cjming@deploy1002> Synchronized php-1.39.0-wmf.17/extensions/VisualEditor/modules: Backport: [[gerrit:808071|Rename `data-ve-target-container` attribute to `data-mw-ve-target-container` (T310197)]] (duration: 03m 33s) [production]
21:00 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:59 <volans@cumin2002> START - Cookbook sre.hosts.decommission for hosts sretest2001.codfw.wmnet [production]
20:57 <volans@cumin2002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host sretest2001.codfw.wmnet [production]
20:56 <cjming@deploy1002> Synchronized php-1.39.0-wmf.17/skins/Vector/includes/templates/skin.mustache: Backport: [[gerrit:808068|Rename `data-ve-target-container` attribute to `data-mw-ve-target-container` (T310197)]] (duration: 03m 33s) [production]
20:56 <mforns@deploy1002> Finished deploy [analytics/refinery@2f5987d]: Regular analytics weekly train [analytics/refinery@2f5987d] (duration: 00m 26s) [production]
20:55 <mforns@deploy1002> Started deploy [analytics/refinery@2f5987d]: Regular analytics weekly train [analytics/refinery@2f5987d] [production]