1951-2000 of 10000 results (59ms)
2022-09-01 §
05:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Set db1181 with weight 0 T316111', diff saved to https://phabricator.wikimedia.org/P33726 and previous config saved to /var/cache/conftool/dbconfig/20220901-051701-ladsgroup.json [production]
05:16 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 T316111 [production]
05:16 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 28 hosts with reason: Primary switchover s7 T316111 [production]
01:20 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase201[3-8].codfw.wmnet: Restart to apply new certificates (T316697) - eevans@cumin1001 [production]
00:21 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase201[3-8].codfw.wmnet: Restart to apply new certificates (T316697) - eevans@cumin1001 [production]
2022-08-31 §
23:31 <krinkle@deploy1002> Synchronized wmf-config/: I493b5e4662 (duration: 03m 43s) [production]
23:22 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
23:21 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
23:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
23:20 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
23:17 <krinkle@deploy1002> Synchronized private/: (no justification provided) (duration: 03m 42s) [production]
23:15 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
23:14 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
23:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
23:13 <krinkle@deploy1002> Synchronized wmf-config/: Ibdac0a (duration: 03m 44s) [production]
23:13 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
23:12 <Krinkle> krinkle@deploy1002 Change /srv/mediawiki-staging/private to remove wmgElectronSecret [production]
23:08 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
23:07 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
23:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
23:06 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
22:45 <ejegg> payments-wiki upgraded from 80657b06 to 648842f9 [production]
22:38 <eileen> config revision changed from 4331ef59 to d3696af7 [production]
22:00 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase1030.eqiad.wmnet: Restart to apply new certificates (T316697) - eevans@cumin1001 [production]
21:59 <mutante> etherpad (etherpad1003) - rebooting for maintenance [production]
21:58 <mutante> mw1383 start php7.2-fpm_check_restart.service [production]
21:52 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase1029.eqiad.wmnet: Restart to apply new certificates (T316697) - eevans@cumin1001 [production]
21:48 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase1030.eqiad.wmnet: Restart to apply new certificates (T316697) - eevans@cumin1001 [production]
21:42 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase1029.eqiad.wmnet: Restart to apply new certificates (T316697) - eevans@cumin1001 [production]
21:42 <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase1028.eqiad.wmnet: Restart to apply new certificates (T316697) - eevans@cumin1001 [production]
21:42 <ebernhardson> run search index creation for pcmwiki [production]
21:41 <ebernhardson> run search index creation for bjnwiktionary [production]
21:40 <ebernhardson> run search index creation for guwwiktionary [production]
21:32 <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase1028.eqiad.wmnet: Restart to apply new certificates (T316697) - eevans@cumin1001 [production]
21:30 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on etherpad1003.eqiad.wmnet with reason: kernel upgrade [production]
21:30 <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on etherpad1003.eqiad.wmnet with reason: kernel upgrade [production]
21:30 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on etherpad1003.eqiad.wmnet with reason: kernel upgrade [production]
21:29 <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on etherpad1003.eqiad.wmnet with reason: kernel upgrade [production]
21:00 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@94b160c]: drop_old_data: Add new required param --allowed-interval (duration: 02m 07s) [production]
20:58 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@94b160c]: drop_old_data: Add new required param --allowed-interval [production]
20:57 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab2003.wikimedia.org with OS bullseye [production]
20:43 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab2003.wikimedia.org with reason: host reimage [production]
20:39 <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab2003.wikimedia.org with reason: host reimage [production]
20:38 <denisse@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
20:36 <eileen> config revision changed from b1bd9422 to 4331ef59 [production]
20:31 <denisse@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
20:31 <denisse|m> rebooting netmon1003 for a kernel upgrade [production]
20:23 <dzahn@cumin2002> START - Cookbook sre.hosts.reimage for host gitlab2003.wikimedia.org with OS bullseye [production]
20:15 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:14 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]