|
2022-10-31
§
|
| 06:58 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2124.codfw.wmnet with reason: Maintenance |
[production] |
| 06:58 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2124.codfw.wmnet with reason: Maintenance |
[production] |
| 06:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321123)', diff saved to https://phabricator.wikimedia.org/P37047 and previous config saved to /var/cache/conftool/dbconfig/20221031-065756-marostegui.json |
[production] |
| 06:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P37046 and previous config saved to /var/cache/conftool/dbconfig/20221031-064249-marostegui.json |
[production] |
| 06:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2117 (T321123)', diff saved to https://phabricator.wikimedia.org/P37044 and previous config saved to /var/cache/conftool/dbconfig/20221031-061236-marostegui.json |
[production] |
| 06:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2117 (T321123)', diff saved to https://phabricator.wikimedia.org/P37043 and previous config saved to /var/cache/conftool/dbconfig/20221031-061026-marostegui.json |
[production] |
| 06:10 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2117.codfw.wmnet with reason: Maintenance |
[production] |
| 06:09 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2117.codfw.wmnet with reason: Maintenance |
[production] |
| 06:08 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1122.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:08 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1122.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:42 |
<kartik@deploy1002> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 05:41 |
<kartik@deploy1002> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 03:10 |
<wm-bot> |
<anticomposite> ./stewardbots/StewardBot/manage.sh restart # Ping timeout |
[tools.stewardbots] |
| 00:12 |
<Deus> |
Removing role::simplelamp2 from ceres-01 |
[wikisp] |
|
2022-10-30
§
|
| 11:00 |
<wm-bot> |
<lucaswerkmeister> lowered notwikilambda quotas: count/deployments.apps from 5 to 3, services from 4 to 1 (T321990) |
[tools.notwikilambda] |
| 10:58 |
<wm-bot> |
<lucaswerkmeister> kubectl delete service function-evaluator function-orchestrator pygments-server # T314880 |
[tools.notwikilambda] |
| 10:34 |
<wm-bot> |
<lucaswerkmeister> kubectl delete deployment function-evaluator function-orchestrator pygments-server (T314880) |
[tools.notwikilambda] |
| 10:34 |
<wm-bot> |
<lucaswerkmeister> kubectl delete deployment function-evaluator function-orchestrator pygments-server |
[tools.notwikilambda] |
| 10:30 |
<wm-bot> |
<lucaswerkmeister> Double IRC messages to other bridges |
[tools.bridgebot] |
| 10:22 |
<wm-bot> |
<lucaswerkmeister> start webservice again (PHP 7.4 now) with new lighttpd config to rewrite everything to a very bare index.html explaining the shutdown (T314880); partial SQL dump available in public_html too |
[tools.notwikilambda] |
| 10:04 |
<wm-bot> |
<lucaswerkmeister> stopped webservice, wiki is being retired (T314880) |
[tools.notwikilambda] |
| 10:04 |
<Deus> |
Rebuild mars-01 (T321763) |
[wikisp] |
|
2022-10-29
§
|
| 20:07 |
<wm-bot> |
<lucaswerkmeister> Double IRC messages to other bridges |
[tools.bridgebot] |
| 19:44 |
<wm-bot> |
<lucaswerkmeister> deployed 19f1300fa1 (mypy) |
[tools.speedpatrolling] |
| 19:07 |
<wm-bot> |
<lucaswerkmeister> deployed f4c38cd04e (rm dead code) |
[tools.speedpatrolling] |
| 17:30 |
<wm-bot> |
<lucaswerkmeister> deployed 4c9e42dbc8 (flake8) |
[tools.speedpatrolling] |
| 17:22 |
<wm-bot> |
<lucaswerkmeister> Double IRC messages to other bridges |
[tools.bridgebot] |
| 15:47 |
<wm-bot> |
<lucaswerkmeister> deployed ae8fce43f8 (gitignore update – pulled without webservice restart) |
[tools.speedpatrolling] |
| 15:31 |
<wm-bot> |
<anticomposite> ./SULWatcher/manage.sh restart # SULWatchers disconnected |
[tools.stewardbots] |
| 15:10 |
<wm-bot> |
<lucaswerkmeister> deployed ee9bd42513 (README update – pulled without webservice restart) |
[tools.speedpatrolling] |
| 15:08 |
<wm-bot> |
<lucaswerkmeister> deployed 9058cfad77 (use pip-tools, update dependencies) |
[tools.speedpatrolling] |
| 11:25 |
<taavi> |
deploy patch for T321971 |
[production] |
| 11:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 11:21 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 11:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 11:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
|
2022-10-28
§
|
| 22:06 |
<wm-bot> |
<anticomposite> ./SULWatcher/manage.sh restart # "Not connected." spam |
[tools.stewardbots] |
| 20:58 |
<wm-bot> |
<bd808> Use short nicks for wikipedia-abstract-tech bridge (T321868) |
[tools.bridgebot] |
| 20:54 |
<wm-bot> |
<bd808> Change wikipedia-abstract-tech bridge telegram channel id (T321868) |
[tools.bridgebot] |
| 20:42 |
<mutante> |
clouddumps* - deployed gerrit:848444 - as kind of expected it fails - most likely the project dirs are not automatically created before rsync runs the first time - T57503 |
[production] |
| 20:37 |
<mutante> |
clouddumps1001 - puppet run after merging gerrit:848441 for kiwix, changed ferm status from "stopped" to "running". manually ran 'sudo systemctl start kiwix-mirror-update' T57503 |
[production] |
| 20:33 |
<wm-bot> |
<bd808> Add wikipedia-abstract-tech bridge (T321868) |
[tools.bridgebot] |
| 20:24 |
<mutante> |
- removing from Horizon / project-wide hiera: profile::phabricator::main::manage_scap_user: true (set in the repo) |
[devtools] |
| 20:24 |
<wm-bot> |
<lucaswerkmeister> Double IRC messages to other bridges |
[tools.bridgebot] |
| 20:23 |
<mutante> |
- removing from Horizon / project-wide hiera: profile::keyholder::server::require_encrypted_keys: 'no' (set in the repo) |
[devtools] |
| 20:19 |
<mutante> |
- removing from Horizon / project-wide hiera: profile::gerrit::daemon_user: gerrit2, profile::gerrit::manage_scap_user: true, profile::gerrit::scap_user: gerrit-deploy (all of these are set in the repo) |
[devtools] |
| 19:17 |
<mutante> |
contint* - changing source for scap repo to gitlab - gerrit:850246 T321847 |
[production] |
| 18:54 |
<ebernhardson@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@2326f9c]: Import cirrus indexes to hdfs (duration: 02m 07s) |
[production] |
| 18:52 |
<ebernhardson@deploy1002> |
Started deploy [wikimedia/discovery/analytics@2326f9c]: Import cirrus indexes to hdfs |
[production] |
| 18:48 |
<andrewbogott> |
deleting project and contents; leftovers from pen-testing |
[2022-testing-2] |