1751-1800 of 10000 results (38ms)
2017-04-03 §
06:40 <_joe_> manually restarted replication for etcd [production]
06:25 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1057 entry - T160435 (duration: 00m 44s) [production]
06:21 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1057 entry - T160435 (duration: 00m 54s) [production]
06:12 <marostegui> Remove partitions from metawiki.pagelinks (s7) on codfw master (db2029) this will generate lag on codfw - T153300 [production]
05:59 <marostegui> Resume pt-table-checksum on wikidata - T161294 [production]
05:53 <_joe_> powercycling mw2256, unresponsive to ping, blank console [production]
02:24 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.18) (duration: 09m 39s) [production]
2017-04-02 §
08:18 <elukey> powercycle ms-be1016 (stuck in console, answers pings but not ssh) [production]
07:25 <ariel@tin> Finished deploy [dumps/dumps@1ac3fb3]: var/method name cleanups, refactor, pregenerate page ranges for page content jobs, auto retry of failed page ranges (duration: 00m 03s) [production]
07:25 <ariel@tin> Started deploy [dumps/dumps@1ac3fb3]: var/method name cleanups, refactor, pregenerate page ranges for page content jobs, auto retry of failed page ranges [production]
02:24 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.18) (duration: 09m 44s) [production]
2017-04-01 §
19:01 <elukey> restart hhvm on mw1191 (dump debug in /tmp/hhvm.16619.bt.) - threads stuck in HPHP::Treadmill::getAgeOldestRequest [production]
02:37 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Apr 1 02:37:30 UTC 2017 (duration 5m 20s) [production]
02:32 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.18) (duration: 13m 27s) [production]
2017-03-31 §
23:11 <mutante> ruthenium: logrotate --force /etc/logrotate.d/parsoid (note this is existing file "parsoid" not new file "parsoid_testing") (T161920) [production]
20:18 <elukey> stopping jobrunners on mw116[89] and restarting hhvm after https://gerrit.wikimedia.org/r/345881 [production]
19:44 <Reedy> Stop badge hacks from messing up the entire page on IE 11 on MonoBook T161869 [production]
19:42 <reedy@tin> Synchronized php-1.29.0-wmf.18/extensions/Echo: Stop badge hacks from messing up the entire page on IE 11 on MonoBook T161689 (duration: 00m 50s) [production]
19:16 <mutante> ruthenium also deleting ancient "htmldumper" data, gwicke confirmed it's not needed anymore [production]
18:27 <mutante> ruthenium mounting /dev/mapper/ruthenium--vg-tank into /srv/visualdiff/pngs | deleted "mysql" and "dumps" data that was on previously unmounted partition , subbu checked that wasn't needed anymore, we still need logrotate (T161920) [production]
18:14 <mutante> ruthenium mounting /dev/mapper/ruthenium--vg-tank which wasnt used at all.. bam.. over 477GB of free space [production]
16:02 <volans@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,name=ms-fe2008.codfw.wmnet [production]
16:00 <volans@puppetmaster1001> conftool action : set/pooled=no; selector: dc=codfw,name=ms-fe2008.codfw.wmnet [production]
15:59 <volans@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,name=ms-fe2007.codfw.wmnet [production]
15:57 <volans@puppetmaster1001> conftool action : set/pooled=no; selector: dc=codfw,name=ms-fe2007.codfw.wmnet [production]
15:55 <mobrovac@tin> Finished deploy [trending-edits/deploy@26b5eb4]: Config change: lower min_edits to 15 T160127 (duration: 06m 37s) [production]
15:55 <volans@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,name=ms-fe2006.codfw.wmnet [production]
15:52 <volans@puppetmaster1001> conftool action : set/pooled=no; selector: dc=codfw,name=ms-fe2006.codfw.wmnet [production]
15:49 <mobrovac@tin> Started deploy [trending-edits/deploy@26b5eb4]: Config change: lower min_edits to 15 T160127 [production]
15:44 <volans@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,name=ms-fe2005.codfw.wmnet [production]
15:22 <volans@puppetmaster1001> conftool action : set/pooled=true; selector: dnsdisc=imagescaler-rw,name=eqiad [production]
15:17 <volans@puppetmaster1001> conftool action : set/pooled=false; selector: dnsdisc=imagescaler-rw,name=eqiad [production]
15:02 <volans@puppetmaster1001> conftool action : set/pooled=no; selector: dc=codfw,name=ms-fe2005.codfw.wmnet [production]
15:01 <oblivian@puppetmaster1001> conftool action : set/ttl=300; selector: dnsdisc=restbase-async [production]
15:01 <oblivian@puppetmaster1001> conftool action : set/pooled=false; selector: dnsdisc=restbase-async,name=eqiad [production]
14:58 <oblivian@puppetmaster1001> conftool action : set/pooled=true; selector: dnsdisc=restbase-async,name=eqiad [production]
14:56 <oblivian@puppetmaster1001> conftool action : set/ttl=10; selector: dnsdisc=restbase-async [production]
14:55 <_joe_> reducing ttl on the restbase-async discovery record, then flipping eqiad to active [production]
14:55 <volans> deploying the use of discovery URL to swift-proxy hosts in codfw T160178#3136906 [production]
14:09 <_joe_> performing a rolling restart of changeprop after puppet runs on scb* [production]
14:00 <oblivian@puppetmaster1001> conftool action : set/pooled=true; selector: dnsdisc=restbase-async,name=codfw [production]
13:23 <elukey> restart hhvm on mw116[89] after https://gerrit.wikimedia.org/r/345829 [production]
13:19 <gehel> rolling restart of maps-test cluster for kernel upgrade [production]
13:09 <moritzm> rebooting bromine to Linux 4.9 [production]
12:10 <moritzm> rebooting mwdebug* to Linux 4.9 [production]
12:05 <moritzm> rebooting pybal-test* to Linux 4.9 [production]
11:22 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1066 after maintenance (duration: 00m 49s) [production]
10:47 <akosiaris> uploaded jessie-wikimedia kubernetes_1.4.6-4 on apt.wikimedia.org/jessie-wikimedia [production]
09:59 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw2244.codfw.wmnet [production]
09:56 <elukey> set pooled=yes mw210[56789], mw2260 and mw2213 (and cleaned up old /srv/mediawiki dirs that were causing rsync spam in scap pull) [production]