3751-3800 of 10000 results (27ms)
2020-10-28 §
07:22 <godog> swift codfw-prod: bump object weight for ms-be2057 - T261633 [production]
04:43 <ryankemper> T266492 Finished rolling restart of codfw cirrus cluster [production]
04:43 <ryankemper@cumin2001> END (PASS) - Cookbook sre.elasticsearch.rolling-restart (exit_code=0) [production]
02:58 <ryankemper> T266492 Beginning rolling restart of codfw cirrus cluster, 3 nodes at a time, on `ryankemper@cumin2001` tmux session `elasticsearch_restart_codfw` [production]
02:57 <ryankemper@cumin2001> START - Cookbook sre.elasticsearch.rolling-restart [production]
02:12 <eileen> tools revision changed from a2a91d6c6a to 087a596d3a [production]
00:40 <eileen> civicrm revision changed from 4fdfb8408b to e1d65b0f3a, config revision is f16003ab62 [production]
2020-10-27 §
22:20 <mutante> systemctl reset-failed on various servers to see which are coming back later from failed auto_restart and which don't [production]
21:40 <mutante> mwmaint2001 - systemctl reset-failed - mediawiki_job_parser_cache_purging.service [production]
20:56 <mutante> ms-be1057 is network down but running, NO-CARRIER on NIC, cable disconnected? [production]
20:43 <mutante> releases2002 - systemctl reset-failed .. after removing wmf_auto_restart_rsync [production]
20:13 <mutante> gerrit1001/gerrit2001: manually deleting list_mediawiki_extensions cron job (T266024) [production]
19:40 <eileen> civicrm revision changed from bb7c08bf6d to 4fdfb8408b, config revision is f16003ab62 [production]
18:35 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
17:55 <otto@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
17:55 <otto@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
17:46 <otto@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'canary' . [production]
17:46 <otto@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
17:44 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics-external' for release 'production' . [production]
17:22 <mutante> gerrit1001/2001 - sudo rm /var/www/mediawiki-extensions.txt [production]
17:18 <ejegg> updated payments-wiki from 4c1503ad91 to adc3369cb3 [production]
16:34 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.cf (exit_code=0) [production]
16:34 <ayounsi@cumin1001> START - Cookbook sre.network.cf [production]
16:05 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.cf (exit_code=0) [production]
16:05 <ayounsi@cumin1001> START - Cookbook sre.network.cf [production]
16:05 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.cf (exit_code=0) [production]
16:05 <ayounsi@cumin1001> START - Cookbook sre.network.cf [production]
16:05 <pt1979@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:59 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
15:42 <mepps> updated payments-wiki-staging from 5fdd29bc16 to 4c1503ad91 [production]
15:25 <ema> cp4032: downgrade varnish to 6.0.4 T264398 [production]
15:13 <ema> cp4032: varnish-frontend-restart with libvmod-netmapper 1.9-1 T266567 [production]
14:55 <ema> upload libvmod-netmapper 1.9-1 to buster-wikimedia component/varnish6 T266567 [production]
14:49 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) [production]
14:48 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
14:40 <_joe_> restarting envoyproxy on the jobrunners in codfw [production]
14:36 <akosiaris> rolling restart of all pods in codfw changeprop-jobqueue [production]
14:27 <_joe_> restart php-fpm on jobrunners in codfw [production]
14:17 <cdanis> ran puppet on alert1001 [production]
14:16 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) [production]
14:15 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-update-tendril [production]
14:15 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-run-puppet-on-db-masters (exit_code=0) [production]
14:11 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-run-puppet-on-db-masters [production]
14:11 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) [production]
14:11 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:11 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:09 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance [production]
14:09 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) [production]
14:09 <rzl@cumin1001> MediaWiki read-only period ends at: 2020-10-27 14:09:02.873019 [production]
14:09 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite [production]