2251-2300 of 10000 results (61ms)
2019-09-16 §
10:51 <vgutierrez@puppetmaster1001> conftool action : set/pooled=no; selector: name=ncredir2002.codfw.wmnet [production]
10:50 <jdrewniak@deploy1001> Synchronized portals: Wikimedia Portals Update: [[gerrit:537060| Bumping portals to master (T128546)]] (duration: 01m 03s) [production]
10:49 <jdrewniak@deploy1001> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:537060| Bumping portals to master (T128546)]] (duration: 01m 04s) [production]
10:45 <vgutierrez> Enabling OCSP prefetched responses for the non-canonical redirect service - T232988 [production]
10:29 <_joe_> installing a patched php-memcached on mw1347 T232613 [production]
10:16 <vgutierrez> upgrade acme-chief production servers to acme-chief 0.21 - T219765 [production]
10:16 <moritzm> upload libtrapperkeeper-webserver-jetty9-clojure 1.7.0-2+wmf1 to buster-wikimedia [production]
10:05 <vgutierrez> restarting acmechief servers to get latest kernel upgrades [production]
09:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:24 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:57 <vgutierrez> replacing nginx with ATS in cp3034 (upload cluster) - T231433 [production]
08:56 <mobrovac@deploy1001> Synchronized wmf-config/CommonSettings.php: Beta: enable the Parsoid extension - T231569 (duration: 01m 01s) [production]
08:50 <marostegui> Apply grants for dbproxy1021 on db1133 (m5 master) with replication - T202367 [production]
08:16 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:14 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:11 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:09 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:38 <moritzm> installing faad2 security updates [production]
07:15 <moritzm> repooling restbase2009 [production]
06:48 <marostegui> Stop MySQL on db1114 to upgrade it to 10.3 [production]
06:03 <marostegui> Stop MySQL on db2054 for decommissioning T232969 [production]
06:01 <marostegui> Remove db2054 from tendril and zarcillo T232969 [production]
05:59 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2054 from config T232969 (duration: 01m 03s) [production]
05:58 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2054 from config T232969 (duration: 01m 05s) [production]
2019-09-15 §
16:51 <Krinkle> Fixed a dozen abuse filtes, listed at https://phabricator.wikimedia.org/T156096#5494060. The trailing pipe character was removed from filters that had it which is no longer supported in a future version of AbuseFilter. [production]
14:35 <_joe_> test: setting opcache.interned_strings_buffer to 0 on mw1348 for T232613 [production]
2019-09-14 §
23:42 <onimisionipe> force shard allocation (dewiki_content_1566659363[4]) on eqiad cluster [production]
04:39 <effie> Depool and reload mw1286 [production]
01:14 <ejegg> updated fundraising python tools from 1e405864d7 to e1b81688c6 [production]
00:29 <ejegg> updated payments-wiki from 1f556670cf to fc82318180 [production]
2019-09-13 §
23:44 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
23:42 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
23:06 <gehel> re-enable puppet on maps - T232817 [production]
20:23 <chaomodus> restarting netbox1001.wikimedia.org [production]
20:00 <twentyafterfour> hotfixing T232600 due to severity of the bug and relative safety of the fix (if this breaks, yell at James_F who twisted my arm and made me do it) [production]
19:54 <urandom> bootstrapping Cassandra, restbase2009-c -- T224553 [production]
17:24 <urandom> bootstrapping Cassandra, restbase2009-b -- T224553 [production]
16:10 <XioNoX> fix bgp group netflow on cr2-codfw [production]
15:47 <urandom> bootstrapping Cassandra, restbase2009-a -- T224553 [production]
15:43 <effie> reverting live hacks on mw1348 [production]
15:34 <hashar@deploy1001> Synchronized wmf-config/CommonSettings.php: Disable adhoc core dump logging - T232613 (duration: 01m 04s) [production]
15:14 <akosiaris> upload apertium-dan_0.6.0-1+wmf3 apertium-nno_1.0.0-1+wmf1 apertium-nob_1.0.0-2+wmf1 apertium-swe_0.8.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia T218184 [production]
15:11 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:11 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:08 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:08 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:02 <hashar@deploy1001> Synchronized php-1.34.0-wmf.22/includes/libs/rdbms/lbfactory/LBFactoryMulti.php: Add more log and context for T232613 logging - T232613 (duration: 01m 04s) [production]
15:02 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:02 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
14:51 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]