2901-2950 of 10000 results (50ms)
2017-04-03 §
12:37 <elukey> reimage analytics10[29,30,31] to Debian Jessie [production]
12:28 <ema> banning 200px-Status_iucn3.1_LC_cs.svg.png from esams frontends T162035 [production]
11:49 <volans@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=eqiad,name=ms-fe1006.eqiad.wmnet [production]
11:45 <volans@puppetmaster1001> conftool action : set/pooled=no; selector: dc=eqiad,name=ms-fe1006.eqiad.wmnet [production]
11:35 <joal@tin> Finished deploy [analytics/refinery@cc73c40]: (no justification provided) (duration: 07m 23s) [production]
11:31 <volans@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=eqiad,name=ms-fe1008.eqiad.wmnet [production]
11:28 <joal@tin> Started deploy [analytics/refinery@cc73c40]: (no justification provided) [production]
11:28 <volans@puppetmaster1001> conftool action : set/pooled=no; selector: dc=eqiad,name=ms-fe1008.eqiad.wmnet [production]
11:21 <volans@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=eqiad,name=ms-fe1007.eqiad.wmnet [production]
11:18 <volans@puppetmaster1001> conftool action : set/pooled=no; selector: dc=eqiad,name=ms-fe1007.eqiad.wmnet [production]
11:08 <volans@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=eqiad,name=ms-fe1006.eqiad.wmnet [production]
11:05 <volans@puppetmaster1001> conftool action : set/pooled=no; selector: dc=eqiad,name=ms-fe1006.eqiad.wmnet [production]
11:04 <volans@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=eqiad,name=ms-fe1005.eqiad.wmnet [production]
10:43 <volans@puppetmaster1001> conftool action : set/pooled=no; selector: dc=eqiad,name=ms-fe1005.eqiad.wmnet [production]
10:38 <volans> upgrading swift-proxy in eqiad to use discovery URLs [production]
08:46 <marostegui> Deploy alter table db1086 (s7) on revision table to unify PK and indexes - T160390 [production]
08:38 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1086 - T160390 (duration: 00m 44s) [production]
07:39 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=mw1261.eqiad.wmnet [production]
07:25 <marostegui> Deploy alter table dbstore2001 (s7) on revision table to unify PK and indexes - T160390 [production]
07:25 <_joe_> rebooting copper to clean up at least partially the docker mess [production]
07:14 <moritzm> switched default kernel for jessie installations to Linux 4.9 [production]
07:06 <_joe_> removing stale files on copper for docker, all local images will be wiped away [production]
07:03 <moritzm> instaling gnutls security updates on trusty [production]
06:51 <marostegui> Deploy InnoDB compression on dewiki - db1070 - T150438 [production]
06:41 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1070 to compress it - T153743 (duration: 00m 44s) [production]
06:40 <_joe_> manually restarted replication for etcd [production]
06:25 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1057 entry - T160435 (duration: 00m 44s) [production]
06:21 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1057 entry - T160435 (duration: 00m 54s) [production]
06:12 <marostegui> Remove partitions from metawiki.pagelinks (s7) on codfw master (db2029) this will generate lag on codfw - T153300 [production]
05:59 <marostegui> Resume pt-table-checksum on wikidata - T161294 [production]
05:53 <_joe_> powercycling mw2256, unresponsive to ping, blank console [production]
02:24 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.18) (duration: 09m 39s) [production]
2017-04-02 §
08:18 <elukey> powercycle ms-be1016 (stuck in console, answers pings but not ssh) [production]
07:25 <ariel@tin> Finished deploy [dumps/dumps@1ac3fb3]: var/method name cleanups, refactor, pregenerate page ranges for page content jobs, auto retry of failed page ranges (duration: 00m 03s) [production]
07:25 <ariel@tin> Started deploy [dumps/dumps@1ac3fb3]: var/method name cleanups, refactor, pregenerate page ranges for page content jobs, auto retry of failed page ranges [production]
02:24 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.18) (duration: 09m 44s) [production]
2017-04-01 §
19:01 <elukey> restart hhvm on mw1191 (dump debug in /tmp/hhvm.16619.bt.) - threads stuck in HPHP::Treadmill::getAgeOldestRequest [production]
02:37 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Apr 1 02:37:30 UTC 2017 (duration 5m 20s) [production]
02:32 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.18) (duration: 13m 27s) [production]
2017-03-31 §
23:11 <mutante> ruthenium: logrotate --force /etc/logrotate.d/parsoid (note this is existing file "parsoid" not new file "parsoid_testing") (T161920) [production]
20:18 <elukey> stopping jobrunners on mw116[89] and restarting hhvm after https://gerrit.wikimedia.org/r/345881 [production]
19:44 <Reedy> Stop badge hacks from messing up the entire page on IE 11 on MonoBook T161869 [production]
19:42 <reedy@tin> Synchronized php-1.29.0-wmf.18/extensions/Echo: Stop badge hacks from messing up the entire page on IE 11 on MonoBook T161689 (duration: 00m 50s) [production]
19:16 <mutante> ruthenium also deleting ancient "htmldumper" data, gwicke confirmed it's not needed anymore [production]
18:27 <mutante> ruthenium mounting /dev/mapper/ruthenium--vg-tank into /srv/visualdiff/pngs | deleted "mysql" and "dumps" data that was on previously unmounted partition , subbu checked that wasn't needed anymore, we still need logrotate (T161920) [production]
18:14 <mutante> ruthenium mounting /dev/mapper/ruthenium--vg-tank which wasnt used at all.. bam.. over 477GB of free space [production]
16:02 <volans@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,name=ms-fe2008.codfw.wmnet [production]
16:00 <volans@puppetmaster1001> conftool action : set/pooled=no; selector: dc=codfw,name=ms-fe2008.codfw.wmnet [production]
15:59 <volans@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,name=ms-fe2007.codfw.wmnet [production]
15:57 <volans@puppetmaster1001> conftool action : set/pooled=no; selector: dc=codfw,name=ms-fe2007.codfw.wmnet [production]