1301-1350 of 10000 results (23ms)
2020-09-22 §
04:59 <marostegui@cumin1001> dbctl commit (dc=all): 'es2016 (re)pooling @ 25%: Slowly repool after recloning es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12698 and previous config saved to /var/cache/conftool/dbconfig/20200922-045928-root.json [production]
04:59 <marostegui@cumin1001> dbctl commit (dc=all): 'es2013 (re)pooling @ 25%: Slowly repool after recloning es2032 T261717 ', diff saved to https://phabricator.wikimedia.org/P12697 and previous config saved to /var/cache/conftool/dbconfig/20200922-045919-root.json [production]
01:35 <ryankemper> `sudo cumin C:profile::services_proxy::envoy 'enable-puppet "adding cloudelastic to the service proxy --rkemper"'` done [production]
01:35 <ryankemper> woot! `curl -X GET -s 'http://localhost:6105/_cluster/health'` gives a response as expected. (As do 6106 and 6107). Re-enabling puppet across the fleet... [production]
01:32 <ryankemper> `sudo run-puppet-agent -e "adding cloudelastic to the service proxy --rkemper"` on `mwdebug1002.eqiad.wmnet` [production]
01:28 <ryankemper> `sudo puppet-merge` done, now will run puppet on a single eqiad appserver and verify we can curl `localhost:610{5,6,7}` [production]
01:17 <ryankemper> Disabling puppet on affected nodes via `sudo cumin C:profile::services_proxy::envoy 'disable-puppet "adding cloudelastic to the service proxy --rkemper"'` [production]
01:17 <ryankemper> Going to test patch to stick envoy in front of `cloudelastic`, see https://gerrit.wikimedia.org/r/c/operations/puppet/+/628243 [production]
2020-09-21 §
23:42 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
23:39 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
23:37 <mholloway-shell@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
23:36 <mutante> debmonitor2002 - systemctl reset-failed [production]
22:59 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
22:57 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
22:55 <mholloway-shell@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
22:20 <mutante> releases.wikimedia.org has been converted to an active-active service with geodns/ backends in both DCs [production]
21:56 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
21:54 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
21:51 <mholloway-shell@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
21:28 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
21:23 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
21:18 <pt1979@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:12 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
20:49 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: adjust enwiktionary completion search ranking (duration: 00m 57s) [production]
20:47 <ebernhardson@deploy1001> Synchronized php-1.36.0-wmf.9/extensions/CirrusSearch/: Remove pages from completion search by page id (duration: 01m 00s) [production]
20:04 <herron> moving prometheus instance from bast3004 to prometheus3001 T243057 [production]
19:46 <herron> moving prometheus instance from bast4002 to prometheus4001 T243057 [production]
19:38 <mholloway-shell@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Push notifications deployment (4/5) (duration: 00m 57s) [production]
19:34 <mholloway-shell@deploy1001> Synchronized wmf-config/CommonSettings.php: Push notifications deployment (3/5) (duration: 00m 57s) [production]
19:28 <mholloway-shell@deploy1001> Synchronized wmf-config/ProductionServices.php: Push notifications deployment (2/5) (duration: 00m 57s) [production]
19:26 <mholloway-shell@deploy1001> Synchronized wmf-config/LabsServices.php: Push notifications deployment (1/5) (duration: 00m 57s) [production]
19:19 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
19:18 <mepps> updated crm to 8f32b6301f [production]
19:15 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
19:14 <ejegg> updated fundraising CiviCRM from e5ebf9d18a to 8f32b6301f [production]
19:13 <mholloway-shell@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
18:59 <ppchelko@deploy1001> Synchronized wmf-config/CommonSettings.php: gerrit:622863 T249745 (duration: 00m 56s) [production]
18:57 <ebernhardson@deploy1001> Finished deploy [search/mjolnir/deploy@8afe8d2]: mjolnir daemons update I336365 (duration: 06m 54s) [production]
18:53 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable and configure GrowthExperiments on plwiki (T254239) and ptwiki (T255027) (duration: 00m 56s) [production]
18:50 <ebernhardson@deploy1001> Started deploy [search/mjolnir/deploy@8afe8d2]: mjolnir daemons update I336365 [production]
18:33 <mepps> updated crm from cc1f7e6d13 to e5ebf9d18a [production]
18:26 <catrope@deploy1001> Synchronized wmf-config/CommonSettings.php: Define Chinese logo variants for Modern Vector (no-op) (part 2) (T261153) (duration: 00m 56s) [production]
18:25 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Define Chinese logo variants for Modern Vector (no-op) (T261153) (duration: 00m 57s) [production]
18:21 <catrope@deploy1001> Synchronized static/images/mobile/copyright/: Update Chinese logo variants for Modern Vector (T261153) (duration: 00m 56s) [production]
18:08 <XioNoX> add NAT rule to pfw3-codfw - T263488 [production]
17:42 <papaul> rebooting ps1-a8-codfw firmware upgrade [production]
16:46 <papaul> shutting down ms-be2019 for BBU replacing [production]
16:24 <marostegui@cumin1001> dbctl commit (dc=all): 'db2127 (re)pooling @ 100%: Slowly repool after on-site maintenance T262247 ', diff saved to https://phabricator.wikimedia.org/P12696 and previous config saved to /var/cache/conftool/dbconfig/20200921-162433-root.json [production]
16:17 <papaul> replacing msw-c8-codfw [production]
16:16 <jayme@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]