3901-3950 of 10000 results (98ms)
2022-10-31 §
08:34 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance [production]
08:34 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 16:00:00 on db2095.codfw.wmnet with reason: Maintenance [production]
08:34 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2158.codfw.wmnet with reason: Maintenance [production]
08:34 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2158.codfw.wmnet with reason: Maintenance [production]
08:33 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance [production]
08:33 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2141.codfw.wmnet with reason: Maintenance [production]
08:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321123)', diff saved to https://phabricator.wikimedia.org/P37057 and previous config saved to /var/cache/conftool/dbconfig/20221031-083342-marostegui.json [production]
08:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P37056 and previous config saved to /var/cache/conftool/dbconfig/20221031-081836-marostegui.json [production]
08:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P37055 and previous config saved to /var/cache/conftool/dbconfig/20221031-080329-marostegui.json [production]
07:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2129 (T321123)', diff saved to https://phabricator.wikimedia.org/P37054 and previous config saved to /var/cache/conftool/dbconfig/20221031-074823-marostegui.json [production]
07:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2129 (T321123)', diff saved to https://phabricator.wikimedia.org/P37053 and previous config saved to /var/cache/conftool/dbconfig/20221031-074611-marostegui.json [production]
07:46 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
07:45 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
07:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321123)', diff saved to https://phabricator.wikimedia.org/P37052 and previous config saved to /var/cache/conftool/dbconfig/20221031-074549-marostegui.json [production]
07:37 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
07:33 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
07:33 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
07:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P37051 and previous config saved to /var/cache/conftool/dbconfig/20221031-073042-marostegui.json [production]
07:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
07:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P37050 and previous config saved to /var/cache/conftool/dbconfig/20221031-071536-marostegui.json [production]
07:14 <kartik@deploy1002> Finished scap: Backport for [[gerrit:845573|Enable Section Translation in Hawaiian, Pashto and Xhosa WPs (T317289)]] (duration: 06m 48s) [production]
07:14 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
07:13 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
07:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
07:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
07:08 <kartik@deploy1002> kartik and kartik: Backport for [[gerrit:845573|Enable Section Translation in Hawaiian, Pashto and Xhosa WPs (T317289)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet [production]
07:07 <kartik@deploy1002> Started scap: Backport for [[gerrit:845573|Enable Section Translation in Hawaiian, Pashto and Xhosa WPs (T317289)]] [production]
07:02 <ryankemper> [WDQS] `ryankemper@wdqs1007:~$ sudo systemctl restart wdqs-blazegraph.service` [production]
07:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2124 (T321123)', diff saved to https://phabricator.wikimedia.org/P37049 and previous config saved to /var/cache/conftool/dbconfig/20221031-070029-marostegui.json [production]
06:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2124 (T321123)', diff saved to https://phabricator.wikimedia.org/P37048 and previous config saved to /var/cache/conftool/dbconfig/20221031-065817-marostegui.json [production]
06:58 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2124.codfw.wmnet with reason: Maintenance [production]
06:58 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2124.codfw.wmnet with reason: Maintenance [production]
06:57 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321123)', diff saved to https://phabricator.wikimedia.org/P37047 and previous config saved to /var/cache/conftool/dbconfig/20221031-065756-marostegui.json [production]
06:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P37046 and previous config saved to /var/cache/conftool/dbconfig/20221031-064249-marostegui.json [production]
06:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321123)', diff saved to https://phabricator.wikimedia.org/P37044 and previous config saved to /var/cache/conftool/dbconfig/20221031-061236-marostegui.json [production]
06:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db2117 (T321123)', diff saved to https://phabricator.wikimedia.org/P37043 and previous config saved to /var/cache/conftool/dbconfig/20221031-061026-marostegui.json [production]
06:10 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2117.codfw.wmnet with reason: Maintenance [production]
06:09 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db2117.codfw.wmnet with reason: Maintenance [production]
06:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1122.eqiad.wmnet with reason: Maintenance [production]
06:08 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1122.eqiad.wmnet with reason: Maintenance [production]
05:42 <kartik@deploy1002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
05:41 <kartik@deploy1002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
03:10 <wm-bot> <anticomposite> ./stewardbots/StewardBot/manage.sh restart # Ping timeout [tools.stewardbots]
00:12 <Deus> Removing role::simplelamp2 from ceres-01 [wikisp]
2022-10-30 §
11:00 <wm-bot> <lucaswerkmeister> lowered notwikilambda quotas: count/deployments.apps from 5 to 3, services from 4 to 1 (T321990) [tools.notwikilambda]
10:58 <wm-bot> <lucaswerkmeister> kubectl delete service function-evaluator function-orchestrator pygments-server # T314880 [tools.notwikilambda]
10:34 <wm-bot> <lucaswerkmeister> kubectl delete deployment function-evaluator function-orchestrator pygments-server (T314880) [tools.notwikilambda]
10:34 <wm-bot> <lucaswerkmeister> kubectl delete deployment function-evaluator function-orchestrator pygments-server [tools.notwikilambda]
10:30 <wm-bot> <lucaswerkmeister> Double IRC messages to other bridges [tools.bridgebot]
10:22 <wm-bot> <lucaswerkmeister> start webservice again (PHP 7.4 now) with new lighttpd config to rewrite everything to a very bare index.html explaining the shutdown (T314880); partial SQL dump available in public_html too [tools.notwikilambda]