2551-2600 of 10000 results (65ms)
2020-03-25 ยง
20:16 <rzl@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
20:16 <rzl@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
20:01 <rzl@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
20:01 <rzl@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
19:36 <hasharDinner> Jenkins restarted on all machines [production]
19:30 <rzl@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
19:30 <rzl@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
19:29 <rlazarus> updating eventstreams to envoy 1.13.1 T246868 [production]
19:28 <twentyafterfour> group1 looks good after deploying wmf.25 refs T233873 [production]
19:27 <hashar> upgrading Jenkins # T248122 [production]
19:26 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.25 refs T233873 [production]
19:26 <twentyafterfour> scap sync-proxies failed on mw1251 [production]
18:53 <ppchelko@deploy1001> Finished deploy [restbase/deploy@a1c3be4]: Add restbase202[123] T244178 (duration: 14m 00s) [production]
18:39 <ppchelko@deploy1001> Started deploy [restbase/deploy@a1c3be4]: Add restbase202[123] T244178 [production]
18:39 <ppchelko@deploy1001> Finished deploy [restbase/deploy@777b881]: Remove experimental PCS endpoints (duration: 14m 28s) [production]
18:24 <ppchelko@deploy1001> Started deploy [restbase/deploy@777b881]: Remove experimental PCS endpoints [production]
18:21 <tgr@deploy1001> Synchronized php-1.35.0-wmf.25/extensions/GrowthExperiments/modules/homepage/: re-sync, mw1251 failed (duration: 03m 18s) [production]
18:13 <tgr@deploy1001> Synchronized php-1.35.0-wmf.25/extensions/GrowthExperiments/modules/homepage/: SWAT: [[gerrit:583393|Mentorship module: Update for root screen refactor (T248422)]] (duration: 03m 23s) [production]
18:06 <ppchelko@deploy1001> Finished deploy [changeprop/deploy@4bdf55b]: Stop rerendering experimental PCS endpoints (duration: 01m 40s) [production]
18:05 <ppchelko@deploy1001> Started deploy [changeprop/deploy@4bdf55b]: Stop rerendering experimental PCS endpoints [production]
17:43 <mvolz@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'citoid' for release 'production' . [production]
17:38 <mvolz@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'citoid' for release 'production' . [production]
17:33 <mvolz@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'citoid' for release 'staging' . [production]
16:50 <moritzm> installing python-bleach security updates [production]
16:47 <moritzm> updated jenkins packages on apt.wikimedia.org to 2.222.1 [production]
16:33 <rzl@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'blubberoid' for release 'production' . [production]
16:32 <sukhe> upload cescout 0.1.0-1 to apt.wm.o (buster) - T247273 [production]
16:17 <rzl@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'blubberoid' for release 'production' . [production]
16:15 <rzl@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
16:07 <rlazarus> updating blubberoid to envoy 1.13.1 T246868 [production]
15:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2115 after reimage to Buster', diff saved to https://phabricator.wikimedia.org/P10767 and previous config saved to /var/cache/conftool/dbconfig/20200325-152148-marostegui.json [production]
15:14 <moritzm> installing deneb.codfw.wmnet T248165 [production]
14:51 <cdanis> repool codfw T248394 [production]
14:46 <mutante> closed port 80 for caching servers on misc backends https://gerrit.wikimedia.org/r/q/topic:%22applayer-tls%22+(status:open%20OR%20status:merged) as final step per service on T210411 [production]
14:39 <mutante> static microsites (annual.wikimedia.org, research.wikimedia.org, static-bugzilla etc). closed port 80 for caching servers, finalizing switch to https behind caching servers [production]
14:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:53 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
13:48 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
13:26 <_joe_> cumin A:puppetmaster 'apt-get -y install puppet-common' [production]
13:03 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
13:02 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:56 <marostegui> Deploy schema change on db1139:3316 [production]
12:45 <marostegui> Stop MySQL on db2115 for reimage to buster [production]
11:50 <cdanis> cr1-codfw: `set chassis fpc 5 inline-services flex-flow-sizing` and `request chassis fpc restart slot 5` T248394 [production]
11:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2115 for upgrade', diff saved to https://phabricator.wikimedia.org/P10763 and previous config saved to /var/cache/conftool/dbconfig/20200325-114655-marostegui.json [production]
11:39 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
11:37 <mutante> decom mw1250 - mw1253 [production]
11:37 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
11:35 <cdanis> depool codfw for router maintenance T248394 [production]