1251-1300 of 10000 results (53ms)
2022-05-04 ยง
21:18 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1179 (T307525)', diff saved to https://phabricator.wikimedia.org/P27538 and previous config saved to /var/cache/conftool/dbconfig/20220504-211853-ladsgroup.json [production]
21:16 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
21:16 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
21:16 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T307525)', diff saved to https://phabricator.wikimedia.org/P27537 and previous config saved to /var/cache/conftool/dbconfig/20220504-211607-ladsgroup.json [production]
21:14 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:788819|labswiki: Enable extension SubPageList3 (T304181)]] (duration: 00m 51s) [production]
21:11 <tgr> running extensions/GrowthExperiments/maintenance/changeWikiConfig.php for T306792 [production]
21:08 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131', diff saved to https://phabricator.wikimedia.org/P27536 and previous config saved to /var/cache/conftool/dbconfig/20220504-210854-ladsgroup.json [production]
21:04 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:03 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:02 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:01 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P27535 and previous config saved to /var/cache/conftool/dbconfig/20220504-210101-ladsgroup.json [production]
20:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1179 (T307525)', diff saved to https://phabricator.wikimedia.org/P27534 and previous config saved to /var/cache/conftool/dbconfig/20220504-205526-ladsgroup.json [production]
20:55 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance [production]
20:55 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1179.eqiad.wmnet with reason: Maintenance [production]
20:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T307525)', diff saved to https://phabricator.wikimedia.org/P27533 and previous config saved to /var/cache/conftool/dbconfig/20220504-205518-ladsgroup.json [production]
20:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1131 (T307525)', diff saved to https://phabricator.wikimedia.org/P27532 and previous config saved to /var/cache/conftool/dbconfig/20220504-205349-ladsgroup.json [production]
20:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P27531 and previous config saved to /var/cache/conftool/dbconfig/20220504-204556-ladsgroup.json [production]
20:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1131 (T307525)', diff saved to https://phabricator.wikimedia.org/P27530 and previous config saved to /var/cache/conftool/dbconfig/20220504-204427-ladsgroup.json [production]
20:44 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [production]
20:44 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1131.eqiad.wmnet with reason: Maintenance [production]
20:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T307525)', diff saved to https://phabricator.wikimedia.org/P27529 and previous config saved to /var/cache/conftool/dbconfig/20220504-204419-ladsgroup.json [production]
20:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P27528 and previous config saved to /var/cache/conftool/dbconfig/20220504-204013-ladsgroup.json [production]
20:32 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:31 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T307525)', diff saved to https://phabricator.wikimedia.org/P27527 and previous config saved to /var/cache/conftool/dbconfig/20220504-203051-ladsgroup.json [production]
20:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P27526 and previous config saved to /var/cache/conftool/dbconfig/20220504-202914-ladsgroup.json [production]
20:27 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:788776|Duplicate eswiki Growth campaign config to itwiki]] (duration: 00m 51s) [production]
20:26 <razzi@cumin1001> END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka jumbo-eqiad cluster: Reboot kafka nodes [production]
20:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P27525 and previous config saved to /var/cache/conftool/dbconfig/20220504-202508-ladsgroup.json [production]
20:20 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:19 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:15 <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:788872|Enable IPInfo instrumentation on all wikis (T296480)]] (duration: 00m 56s) [production]
20:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P27524 and previous config saved to /var/cache/conftool/dbconfig/20220504-201409-ladsgroup.json [production]
20:13 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1175 (T307525)', diff saved to https://phabricator.wikimedia.org/P27523 and previous config saved to /var/cache/conftool/dbconfig/20220504-201002-ladsgroup.json [production]
20:09 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:09 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:07 <cjming@deploy1002> Synchronized php-1.39.0-wmf.10/extensions/WikimediaEvents/includes/PageSplitter/PageSplitterInstrumentation.php: Backport: [[gerrit:788849|Fix undefined offset error (T307019)]] (duration: 00m 50s) [production]
20:03 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:02 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:02 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:01 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
19:59 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T307525)', diff saved to https://phabricator.wikimedia.org/P27522 and previous config saved to /var/cache/conftool/dbconfig/20220504-195903-ladsgroup.json [production]
19:57 <brennen@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.10 refs T305216 (duration: 00m 52s) [production]