4851-4900 of 10000 results (64ms)
2022-06-15 §
05:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1098:3317 (T302659)', diff saved to https://phabricator.wikimedia.org/P29745 and previous config saved to /var/cache/conftool/dbconfig/20220615-054252-marostegui.json [production]
05:42 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
05:42 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1098.eqiad.wmnet with reason: Maintenance [production]
05:34 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1139.eqiad.wmnet with reason: Maintenance [production]
05:34 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1139.eqiad.wmnet with reason: Maintenance [production]
05:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1173.eqiad.wmnet with OS bullseye [production]
05:17 <marostegui> dbmaint es5@codfw T310485 [production]
05:17 <marostegui> dbmaint es4@codfw T310485 [production]
05:17 <marostegui> dbmaint es3@codfw T310485 [production]
05:17 <marostegui> dbmaint es2@codfw T310485 [production]
05:17 <marostegui> dbmaint es1@codfw T310485 [production]
05:07 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1173.eqiad.wmnet with reason: host reimage [production]
05:04 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1173.eqiad.wmnet with reason: host reimage [production]
05:03 <marostegui> Reboot dbproxy1016 and dbproxy1021 T310484 [production]
04:53 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db1173.eqiad.wmnet with OS bullseye [production]
02:31 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
02:30 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
02:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
02:29 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
02:25 <tstarling@deploy1002> Synchronized php-1.39.0-wmf.16/includes/cache/MessageCache.php: (no justification provided) (duration: 03m 36s) [production]
02:24 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
02:21 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
02:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
02:17 <tstarling@deploy1002> Synchronized php-1.39.0-wmf.15/includes/cache/MessageCache.php: T310532 (duration: 03m 29s) [production]
02:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
2022-06-14 §
23:52 <mutante> gitlab-runner1001/1002 - clean revert not possible, icinga alerting about failed buildkitd service, manually deleting systemd unit and trying to clean up T308271 [production]
23:49 <mutante> gitlab-runner1002 - systemctl restart docker; run-puppet-agent ; systemctl start buildkitd - fails though T308271 [production]
23:39 <mutante> gitlab-runner1001 - systemctl start buildkitd [production]
23:32 <mutante> gitlab-runner1001 - restarting docker [production]
23:08 <mutante> disabling puppet in gitlab-runners (via cumin /disable-puppet) before deploying gerrit:791655 to provide gitlab-runners with buildkit and new docker network - T308271 [production]
22:19 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
22:18 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
22:18 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
22:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
22:15 <urbanecm@deploy1002> Synchronized wmf-config/: e3fe6c04c95717f0f914bbfa366f5f827f392b6b: phpcs: fix more SpaceBeforeSingleLineComment.NewLineComment (T171115) (duration: 03m 39s) [production]
22:05 <urbanecm@deploy1002> Synchronized w/: ca3b94f2d9bc755d92839e5e69072615ea9008df: phpcs: start to fix SpaceBeforeSingleLineComment.NewLineComment (T171115) (duration: 03m 18s) [production]
22:02 <urbanecm@deploy1002> Synchronized src/: ca3b94f2d9bc755d92839e5e69072615ea9008df: phpcs: start to fix SpaceBeforeSingleLineComment.NewLineComment (T171115) (duration: 03m 32s) [production]
22:00 <mutante> wtp1026 - manually running '/usr/bin/sudo -u root -- /usr/local/sbin/check-and-restart-php php7.2-fpm 9223372036854775807' [production]
21:58 <urbanecm@deploy1002> Synchronized rpc/: ca3b94f2d9bc755d92839e5e69072615ea9008df: phpcs: start to fix SpaceBeforeSingleLineComment.NewLineComment (T171115) (duration: 03m 31s) [production]
21:57 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:54 <urbanecm@deploy1002> Synchronized multiversion/: ca3b94f2d9bc755d92839e5e69072615ea9008df: phpcs: start to fix SpaceBeforeSingleLineComment.NewLineComment (T171115) (duration: 03m 29s) [production]
21:54 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1003.eqiad.wmnet [production]
21:53 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:53 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:51 <urbanecm@deploy1002> Synchronized docroot/: ca3b94f2d9bc755d92839e5e69072615ea9008df: phpcs: start to fix SpaceBeforeSingleLineComment.NewLineComment (T171115) (duration: 03m 38s) [production]
21:49 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc-gp1003.eqiad.wmnet [production]
21:49 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:47 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1002.eqiad.wmnet [production]
21:40 <aokoth@cumin1001> START - Cookbook sre.hosts.reboot-single for host mc-gp1002.eqiad.wmnet [production]
21:38 <aokoth@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-gp1001.eqiad.wmnet [production]