2501-2550 of 10000 results (74ms)
2022-12-08 ยง
14:59 <hashar> Restarting Gerrit replica TWICE on gerrit2002.wikimedia.org to apply `-Dh2.maxCompactTime` and get it to trigger compaction # T323754 [production]
14:54 <jiji@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
14:53 <jiji@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]
14:53 <jiji@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-web: apply [production]
14:53 <jiji@deploy1002> helmfile [codfw] START helmfile.d/services/mw-web: apply [production]
14:52 <jiji@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
14:52 <jiji@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
14:52 <jiji@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
14:52 <jiji@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
14:52 <jiji@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
14:50 <jiji@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
14:50 <jiji@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply [production]
14:47 <jiji@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
14:47 <jiji@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
14:47 <jiji@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
14:47 <jiji@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [production]
14:47 <jiji@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
14:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1202 (T322618)', diff saved to https://phabricator.wikimedia.org/P42649 and previous config saved to /var/cache/conftool/dbconfig/20221208-144602-ladsgroup.json [production]
14:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1202 (T322618)', diff saved to https://phabricator.wikimedia.org/P42648 and previous config saved to /var/cache/conftool/dbconfig/20221208-144152-ladsgroup.json [production]
14:41 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance [production]
14:41 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance [production]
14:41 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P42647 and previous config saved to /var/cache/conftool/dbconfig/20221208-144131-ladsgroup.json [production]
14:40 <jiji@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
14:40 <jiji@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
14:26 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P42646 and previous config saved to /var/cache/conftool/dbconfig/20221208-142625-ladsgroup.json [production]
14:21 <jgiannelos@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [production]
14:20 <jgiannelos@deploy1002> helmfile [eqiad] START helmfile.d/services/mobileapps: apply [production]
14:20 <jgiannelos@deploy1002> helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [production]
14:19 <jgiannelos@deploy1002> helmfile [codfw] START helmfile.d/services/mobileapps: apply [production]
14:19 <jgiannelos@deploy1002> helmfile [staging] DONE helmfile.d/services/mobileapps: apply [production]
14:18 <jgiannelos@deploy1002> helmfile [staging] START helmfile.d/services/mobileapps: apply [production]
14:16 <jgiannelos@deploy1002> helmfile [eqiad] DONE helmfile.d/services/proton: apply [production]
14:13 <jgiannelos@deploy1002> helmfile [eqiad] START helmfile.d/services/proton: apply [production]
14:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P42645 and previous config saved to /var/cache/conftool/dbconfig/20221208-141118-ladsgroup.json [production]
14:10 <jgiannelos@deploy1002> helmfile [codfw] DONE helmfile.d/services/proton: apply [production]
14:09 <jgiannelos@deploy1002> helmfile [codfw] START helmfile.d/services/proton: apply [production]
14:08 <jgiannelos@deploy1002> helmfile [staging] DONE helmfile.d/services/proton: apply [production]
14:07 <jgiannelos@deploy1002> helmfile [staging] START helmfile.d/services/proton: apply [production]
13:56 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P42644 and previous config saved to /var/cache/conftool/dbconfig/20221208-135611-ladsgroup.json [production]
13:54 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1194 (T322618)', diff saved to https://phabricator.wikimedia.org/P42643 and previous config saved to /var/cache/conftool/dbconfig/20221208-135402-ladsgroup.json [production]
13:53 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1194.eqiad.wmnet with reason: Maintenance [production]
13:53 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1194.eqiad.wmnet with reason: Maintenance [production]
13:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1191 (T322618)', diff saved to https://phabricator.wikimedia.org/P42642 and previous config saved to /var/cache/conftool/dbconfig/20221208-135341-ladsgroup.json [production]
13:43 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/thumbor: sync [production]
13:43 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/thumbor: sync [production]
13:38 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P42641 and previous config saved to /var/cache/conftool/dbconfig/20221208-133835-ladsgroup.json [production]
13:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P42640 and previous config saved to /var/cache/conftool/dbconfig/20221208-132329-ladsgroup.json [production]
13:20 <aqu@deploy1002> Finished deploy [airflow-dags/analytics@455d142]: Hotfix on HDFS usage (Remove the specific unicode char in comment) - analytics [airflow-dags@455d142] (duration: 00m 15s) [production]
13:20 <aqu@deploy1002> Started deploy [airflow-dags/analytics@455d142]: Hotfix on HDFS usage (Remove the specific unicode char in comment) - analytics [airflow-dags@455d142] [production]
13:19 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@455d142]: Hotfix on HDFS usage (Unicode in comment) - analytics_test [airflow-dags@455d142] (duration: 00m 09s) [production]