2251-2300 of 10000 results (63ms)
2019-06-03 §
09:46 <moritzm> upgrading Druid/Kafka-Jumbo servers to new Java security release (will be picked up by forthcoming MDS reboots) [production]
09:43 <moritzm> upgrading AQS servers to new Java security release (will be picked up by forthcoming MDS reboots) [production]
09:33 <moritzm> upgrading Hadoop servers to new Java security release (will be picked up by forthcoming MDS reboots) [production]
08:18 <ema> cp1077: restart varnish-be [production]
08:17 <elukey> manually removed phab_clean_tmp from www-data's crontab on phab1001 to reduce cronspam [production]
08:16 <ema> cp1075: restart varnish-be [production]
08:03 <marostegui> Stop MySQL on db1064 T223217 [production]
08:01 <marostegui> Remove db1064 from tendril and zarcillo T223217 [production]
07:58 <elukey> refresh field list for logstash (via kibana Management -> Index patterns -> etc..) [production]
07:48 <marostegui> Repool db1103 after upgrade T224852 [production]
07:29 <marostegui> Stop MySQL on db1103 (s2 and s4) for upgrade T224852 [production]
07:29 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1103 for upgrade (duration: 00m 47s) [production]
07:10 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1081 into API after upgrade (duration: 00m 48s) [production]
06:50 <elukey> roll restart varnishkafka (via puppet) for a config change - T224236 [production]
06:46 <kartik@deploy1001> scap-helm cxserver finished [production]
06:46 <kartik@deploy1001> scap-helm cxserver cluster eqiad completed [production]
06:45 <kartik@deploy1001> scap-helm cxserver upgrade -f cxserver-eqiad-values.yaml production stable/cxserver [namespace: cxserver, clusters: eqiad] [production]
06:45 <kartik@deploy1001> scap-helm cxserver finished [production]
06:45 <kartik@deploy1001> scap-helm cxserver cluster codfw completed [production]
06:45 <kartik@deploy1001> scap-helm cxserver upgrade -f cxserver-codfw-values.yaml production stable/cxserver [namespace: cxserver, clusters: codfw] [production]
06:44 <kartik@deploy1001> scap-helm cxserver finished [production]
06:44 <kartik@deploy1001> scap-helm cxserver cluster staging completed [production]
06:44 <kartik@deploy1001> scap-helm cxserver upgrade -f cxserver-staging-values.yaml staging stable/cxserver [namespace: cxserver, clusters: staging] [production]
06:39 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1081 into API after upgrade (duration: 00m 49s) [production]
06:26 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1081 after upgrade (duration: 00m 46s) [production]
06:04 <marostegui> Stop MySQL on db1081 for upgrade - T224852 [production]
06:00 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1081 for upgrade (duration: 00m 47s) [production]
05:50 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool es1019 T213422 (duration: 00m 46s) [production]
05:45 <marostegui> Upgrade mariadb on dbstore1004 - T224852 [production]
05:17 <marostegui> Upgrade MariaDB on codfw hosts in preparation for s4 master failover T217396 [production]
05:15 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to es1019 T213422 (duration: 00m 46s) [production]
05:05 <marostegui> Remove db2037 from tendril and zarcillo T224720 [production]
05:04 <marostegui> Stop MySQL on db2037 for decommission T224720 [production]
04:56 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool es1019 T213422 (duration: 00m 51s) [production]
2019-06-02 §
20:28 <onimisionipe> pooled wdqs1007. It caught up on lag [production]
15:24 <onimisionipe> depooled wdqs1007 to catch up on lags [production]
15:22 <onimisionipe> depool wdqs internal cluster to allow them catch up on lags. depool one at a time [production]
03:09 <andrewbogott> restarting pdns-recursor on cloudservices 1003 and 1004 (but not at the same time) [production]
2019-06-01 §
22:49 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.7/extensions/3D/modules/mmv.3d.js: T224812 / bd4fbfddbe1a0 (duration: 01m 07s) [production]
2019-05-31 §
21:47 <aaron@deploy1001> Synchronized wmf-config/db-eqiad.php: Set "secret" field in $wgLBFactoryConf for ChronologyProtector HMACs (duration: 00m 47s) [production]
21:46 <aaron@deploy1001> Synchronized wmf-config/db-codfw.php: Set "secret" field in $wgLBFactoryConf for ChronologyProtector HMACs (duration: 00m 50s) [production]
21:10 <bblack> cp3034: repool - T222937 [production]
20:04 <bblack> cp3034: depool for reimage - T222937 [production]
18:44 <marostegui> Start MySQL on es1019 - T213422 [production]
18:34 <jgleeson> payments-wiki updated from a76658f0a3 to c6c7bbf71e [production]
17:29 <andrewbogott> added jeh to the 'ops' group in ldap [production]
16:20 <ariel@deploy1001> Finished deploy [dumps/dumps@fd6100a]: remove orderrevs config option, unneeded now (duration: 00m 03s) [production]
16:20 <ariel@deploy1001> Started deploy [dumps/dumps@fd6100a]: remove orderrevs config option, unneeded now [production]
15:05 <bblack> cp3039: restart varnish-be for mbox lag (likely induced by 3049's depool for ATS conversion!) [production]
15:00 <Krinkle> krinkle@deploy1001: pulling down 6f91b41 for php-1.34-wmf.7/extensions/ORES (without deploy), commit seems test-only [production]