2351-2400 of 10000 results (78ms)
2022-10-31 §
08:33 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance [production]
08:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321123)', diff saved to https://phabricator.wikimedia.org/P37057 and previous config saved to /var/cache/conftool/dbconfig/20221031-083342-marostegui.json [production]
08:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P37056 and previous config saved to /var/cache/conftool/dbconfig/20221031-081836-marostegui.json [production]
08:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P37055 and previous config saved to /var/cache/conftool/dbconfig/20221031-080329-marostegui.json [production]
07:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321123)', diff saved to https://phabricator.wikimedia.org/P37054 and previous config saved to /var/cache/conftool/dbconfig/20221031-074823-marostegui.json [production]
07:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2129 (T321123)', diff saved to https://phabricator.wikimedia.org/P37053 and previous config saved to /var/cache/conftool/dbconfig/20221031-074611-marostegui.json [production]
07:46 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
07:45 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
07:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321123)', diff saved to https://phabricator.wikimedia.org/P37052 and previous config saved to /var/cache/conftool/dbconfig/20221031-074549-marostegui.json [production]
07:37 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
07:33 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
07:33 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
07:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P37051 and previous config saved to /var/cache/conftool/dbconfig/20221031-073042-marostegui.json [production]
07:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
07:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P37050 and previous config saved to /var/cache/conftool/dbconfig/20221031-071536-marostegui.json [production]
07:14 <kartik@deploy1002> Finished scap: Backport for [[gerrit:845573|Enable Section Translation in Hawaiian, Pashto and Xhosa WPs (T317289)]] (duration: 06m 48s) [production]
07:14 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
07:13 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
07:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
07:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
07:08 <kartik@deploy1002> kartik and kartik: Backport for [[gerrit:845573|Enable Section Translation in Hawaiian, Pashto and Xhosa WPs (T317289)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
07:07 <kartik@deploy1002> Started scap: Backport for [[gerrit:845573|Enable Section Translation in Hawaiian, Pashto and Xhosa WPs (T317289)]] [production]
07:02 <ryankemper> [WDQS] `ryankemper@wdqs1007:~$ sudo systemctl restart wdqs-blazegraph.service` [production]
07:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321123)', diff saved to https://phabricator.wikimedia.org/P37049 and previous config saved to /var/cache/conftool/dbconfig/20221031-070029-marostegui.json [production]
06:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2124 (T321123)', diff saved to https://phabricator.wikimedia.org/P37048 and previous config saved to /var/cache/conftool/dbconfig/20221031-065817-marostegui.json [production]
06:58 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2124.codfw.wmnet with reason: Maintenance [production]
06:58 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2124.codfw.wmnet with reason: Maintenance [production]
06:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321123)', diff saved to https://phabricator.wikimedia.org/P37047 and previous config saved to /var/cache/conftool/dbconfig/20221031-065756-marostegui.json [production]
06:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P37046 and previous config saved to /var/cache/conftool/dbconfig/20221031-064249-marostegui.json [production]
06:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321123)', diff saved to https://phabricator.wikimedia.org/P37044 and previous config saved to /var/cache/conftool/dbconfig/20221031-061236-marostegui.json [production]
06:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2117 (T321123)', diff saved to https://phabricator.wikimedia.org/P37043 and previous config saved to /var/cache/conftool/dbconfig/20221031-061026-marostegui.json [production]
06:10 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2117.codfw.wmnet with reason: Maintenance [production]
06:09 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2117.codfw.wmnet with reason: Maintenance [production]
06:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1122.eqiad.wmnet with reason: Maintenance [production]
06:08 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1122.eqiad.wmnet with reason: Maintenance [production]
05:42 <kartik@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
05:41 <kartik@deploy1002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
2022-10-29 §
11:25 <taavi> deploy patch for T321971 [production]
11:22 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
11:21 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
11:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
11:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
2022-10-28 §
20:42 <mutante> clouddumps* - deployed gerrit:848444 - as kind of expected it fails - most likely the project dirs are not automatically created before rsync runs the first time - T57503 [production]
20:37 <mutante> clouddumps1001 - puppet run after merging gerrit:848441 for kiwix, changed ferm status from "stopped" to "running". manually ran 'sudo systemctl start kiwix-mirror-update' T57503 [production]
19:17 <mutante> contint* - changing source for scap repo to gitlab - gerrit:850246 T321847 [production]
18:54 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@2326f9c]: Import cirrus indexes to hdfs (duration: 02m 07s) [production]
18:52 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@2326f9c]: Import cirrus indexes to hdfs [production]
18:44 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
18:11 <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer [production]
18:08 <bking@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) [production]