3901-3950 of 10000 results (70ms)
2019-06-03 §
06:50 <elukey> roll restart varnishkafka (via puppet) for a config change - T224236 [production]
06:46 <kartik@deploy1001> scap-helm cxserver finished [production]
06:46 <kartik@deploy1001> scap-helm cxserver cluster eqiad completed [production]
06:45 <kartik@deploy1001> scap-helm cxserver upgrade -f cxserver-eqiad-values.yaml production stable/cxserver [namespace: cxserver, clusters: eqiad] [production]
06:45 <kartik@deploy1001> scap-helm cxserver finished [production]
06:45 <kartik@deploy1001> scap-helm cxserver cluster codfw completed [production]
06:45 <kartik@deploy1001> scap-helm cxserver upgrade -f cxserver-codfw-values.yaml production stable/cxserver [namespace: cxserver, clusters: codfw] [production]
06:44 <kartik@deploy1001> scap-helm cxserver finished [production]
06:44 <kartik@deploy1001> scap-helm cxserver cluster staging completed [production]
06:44 <kartik@deploy1001> scap-helm cxserver upgrade -f cxserver-staging-values.yaml staging stable/cxserver [namespace: cxserver, clusters: staging] [production]
06:39 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1081 into API after upgrade (duration: 00m 49s) [production]
06:26 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1081 after upgrade (duration: 00m 46s) [production]
06:04 <marostegui> Stop MySQL on db1081 for upgrade - T224852 [production]
06:00 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1081 for upgrade (duration: 00m 47s) [production]
05:50 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool es1019 T213422 (duration: 00m 46s) [production]
05:45 <marostegui> Upgrade mariadb on dbstore1004 - T224852 [production]
05:17 <marostegui> Upgrade MariaDB on codfw hosts in preparation for s4 master failover T217396 [production]
05:15 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: More traffic to es1019 T213422 (duration: 00m 46s) [production]
05:05 <marostegui> Remove db2037 from tendril and zarcillo T224720 [production]
05:04 <marostegui> Stop MySQL on db2037 for decommission T224720 [production]
04:56 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool es1019 T213422 (duration: 00m 51s) [production]
2019-06-02 §
20:28 <onimisionipe> pooled wdqs1007. It caught up on lag [production]
15:24 <onimisionipe> depooled wdqs1007 to catch up on lags [production]
15:22 <onimisionipe> depool wdqs internal cluster to allow them catch up on lags. depool one at a time [production]
03:09 <andrewbogott> restarting pdns-recursor on cloudservices 1003 and 1004 (but not at the same time) [production]
2019-06-01 §
22:49 <krinkle@deploy1001> Synchronized php-1.34.0-wmf.7/extensions/3D/modules/mmv.3d.js: T224812 / bd4fbfddbe1a0 (duration: 01m 07s) [production]
2019-05-31 §
21:47 <aaron@deploy1001> Synchronized wmf-config/db-eqiad.php: Set "secret" field in $wgLBFactoryConf for ChronologyProtector HMACs (duration: 00m 47s) [production]
21:46 <aaron@deploy1001> Synchronized wmf-config/db-codfw.php: Set "secret" field in $wgLBFactoryConf for ChronologyProtector HMACs (duration: 00m 50s) [production]
21:10 <bblack> cp3034: repool - T222937 [production]
20:04 <bblack> cp3034: depool for reimage - T222937 [production]
18:44 <marostegui> Start MySQL on es1019 - T213422 [production]
18:34 <jgleeson> payments-wiki updated from a76658f0a3 to c6c7bbf71e [production]
17:29 <andrewbogott> added jeh to the 'ops' group in ldap [production]
16:20 <ariel@deploy1001> Finished deploy [dumps/dumps@fd6100a]: remove orderrevs config option, unneeded now (duration: 00m 03s) [production]
16:20 <ariel@deploy1001> Started deploy [dumps/dumps@fd6100a]: remove orderrevs config option, unneeded now [production]
15:05 <bblack> cp3039: restart varnish-be for mbox lag (likely induced by 3049's depool for ATS conversion!) [production]
15:00 <Krinkle> krinkle@deploy1001: pulling down 6f91b41 for php-1.34-wmf.7/extensions/ORES (without deploy), commit seems test-only [production]
14:59 <Krinkle> krinkle@deploy1001: git status in php-1.34-wmf.7/ is dirty (extensions/ORES) [production]
14:52 <bblack> pool cp3049 back into service - T222937 [production]
14:32 <onimisionipe> depool maps2004 (again) - T224395 [production]
14:32 <elukey> powercycle notebook1003 - host stuck due to user processes, no ssh available, OOM didn't trigger [production]
14:20 <_joe_> rolling restart of php-fpm across production to pick up the shorter revalidate frequency for T224491 [production]
14:10 <bblack> reboot cp3049 - T222937 [production]
13:16 <bblack> depool cp3049 for reimage - T222937 [production]
11:46 <jynus> stop and upgrade db2084 [production]
11:09 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1099 after maintenance (duration: 00m 48s) [production]
10:54 <jynus> depool labsdb1010 for maintenance [production]
10:47 <arturo> merging multiple commits to labs/private.git. We now require `puppet-merge --labsprivate` and people may not be yet aware of that [production]
09:28 <jynus> stop and upgrade db2073 [production]
09:11 <jynus> stop and upgrade db2095 (s2, s4, s6, s7) [production]