4101-4150 of 10000 results (70ms)
2019-02-25 §
15:30 <akosiaris@deploy1001> scap-helm citoid cluster staging completed [production]
15:30 <akosiaris@deploy1001> scap-helm citoid install -n staging -f citoid-staging-values.yaml stable/citoid [namespace: citoid, clusters: staging] [production]
15:28 <jiji@deploy1001> Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 07s) [production]
15:28 <jiji@deploy1001> Started deploy [3d2png/deploy@ca39432]: (no justification provided) [production]
15:27 <jiji@deploy1001> deploy aborted: (no justification provided) (duration: 00m 04s) [production]
15:27 <jiji@deploy1001> Started deploy [3d2png/deploy@ca39432]: (no justification provided) [production]
15:20 <vgutierrez> shutting down certcentral VMs for decommission - T207389 [production]
15:18 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Increase API traffic for db1085 after MySQL upgrade (duration: 00m 45s) [production]
15:04 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1085 after MySQL upgrade (duration: 00m 45s) [production]
14:50 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1085 after MySQL upgrade (duration: 00m 45s) [production]
14:49 <jiji@deploy1001> Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 15s) [production]
14:49 <jiji@deploy1001> Started deploy [3d2png/deploy@ca39432]: (no justification provided) [production]
14:47 <jiji@deploy1001> deploy aborted: (no justification provided) (duration: 00m 19s) [production]
14:46 <jiji@deploy1001> Started deploy [3d2png/deploy@ca39432]: (no justification provided) [production]
14:32 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool into API db1085 after MySQL upgrade (duration: 00m 45s) [production]
14:15 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1085 after MySQL upgrade (duration: 00m 45s) [production]
14:04 <marostegui> Stop MySQL on db1085 for mysql upgrade [production]
13:55 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1085 for MySQL upgrade and schema change (duration: 00m 46s) [production]
13:32 <akosiaris> upgrade etherpad-lite to 1.7.5 [production]
12:38 <gilles@deploy1001> Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 07s) [production]
12:38 <gilles@deploy1001> Started deploy [3d2png/deploy@ca39432]: (no justification provided) [production]
12:27 <gilles@deploy1001> Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 05s) [production]
12:27 <gilles@deploy1001> Started deploy [3d2png/deploy@ca39432]: (no justification provided) [production]
12:22 <gilles@deploy1001> Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 01m 15s) [production]
12:21 <gilles@deploy1001> Started deploy [3d2png/deploy@ca39432]: (no justification provided) [production]
11:49 <moritzm> rolling out intel-microcode 3.20180807a.2 on all jessie/stretch servers, tests on a number of previously unsupported servers with Westmere CPU were successful and I've verified that all other microcode files are identical compared to the current 3.20180807a.1 microcode [production]
11:19 <jijiki> Reimageing thumbor1001 - T214597 [production]
10:40 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:492636| Bumping portals to master (T128546, T202497)]] (duration: 00m 46s) [production]
10:39 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:492636| Bumping portals to master (T128546, T202497)]] (duration: 00m 46s) [production]
10:31 <gtirloni> labstore1004 restarted nfsd and killed stuck rpc.mountd.real processed (T216988) [production]
10:16 <jijiki> Depooling thumbor1001 to reimage - T214597 [production]
09:54 <marostegui> Deploy schema change on db1074, this will generate lag on labsdb:s2 - T187295 [production]
09:07 <marostegui@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Increase ParserCache TTL from 24 days to 30 - T210992 (duration: 00m 46s) [production]
08:52 <marostegui> Deploy schema change on s2 on codfw master - lag will happen on s2 codfw - T187295 [production]
08:49 <_joe_> generating mcrouter certificate for mw2151 T192457 [production]
07:09 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1104 after MySQL upgrade (duration: 00m 45s) [production]
06:28 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1104 in API after MySQL upgrade (duration: 00m 45s) [production]
06:13 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1104 after MySQL upgrade (duration: 00m 45s) [production]
06:02 <marostegui> Stop MySQL on db1104 for mysql upgrade [production]
06:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1104 for MySQL upgrade (duration: 00m 50s) [production]
2019-02-24 §
21:49 <eileen> civicrm revision changed from 1b5d974569 to d1fc603677, config revision is 00f9c08766 [production]
18:20 <elukey> clean up 2017/2018 log files in /var/log/jmxtrans on kafka1013-22 - root partitions filling up [production]
18:15 <elukey> clean up 2017/2018 log files in /var/log/jmxtrans - root partition almost filled up [production]
10:22 <elukey> force remount of /mnt/hdfs on an-coord1001 (fuse-hdfs stuck) [production]
2019-02-22 §
18:02 <gehel> rolling upgrade on elasticsearch / cirrus / eqiad completed - T215931 [production]
18:00 <gehel@cumin2001> END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) [production]
18:00 <gehel@cumin2001> START - Cookbook sre.elasticsearch.force-shard-allocation [production]
17:33 <gehel@cumin2001> END (PASS) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=0) [production]
17:33 <gehel@cumin2001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
17:33 <gehel@cumin2001> END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97) [production]