2651-2700 of 10000 results (46ms)
2017-08-14 §
20:35 <ppchelko@tin> Started deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation [production]
17:09 <gehel> resetting mode on stat1005:/srv/published-datasets/discovery recursively [production]
16:29 <elukey> execute sudo find -type f -mtime +60 -exec rm {} \; in /var/lib/carbon on graphite2001 to free some space in / [production]
13:32 <elukey> Execute systemctl mask nfacctd on rhenium.wikimedia.org for T172681 [production]
13:08 <zeljkof> nothing for EU SWAT [production]
12:21 <jynus> stopping replication on all instances of dbstore2001 T169516 [production]
11:01 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add db2076 - T170662 T151029 (duration: 00m 47s) [production]
11:00 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add db2076 - T170662 T151029 (duration: 01m 02s) [production]
10:34 <marostegui> Restart MySQL on db1069 to pick up new replcation filters [production]
10:27 <marostegui> Restart MySQL on db1095 to pick up new replication filters [production]
09:16 <oblivian@puppetmaster1001> conftool action : set/pooled=yes; selector: service=thumbor,name=thumbor2001.codfw.wmnet [production]
09:07 <_joe_> stopping thumbor on thumbor2001 after depooling it for testing [production]
09:05 <oblivian@puppetmaster1001> conftool action : set/pooled=no; selector: service=thumbor,name=thumbor2001.codfw.wmnet [production]
03:15 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Aug 14 03:15:31 UTC 2017 (duration 7m 6s) [production]
03:08 <l10nupdate@tin> scap sync-l10n completed (1.30.0-wmf.13) (duration: 06m 47s) [production]
02:39 <l10nupdate@tin> scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 29s) [production]
2017-08-13 §
22:45 <ebernhardson> restart elsaticsearch on elastic1017 after setting md2 readahead to 256 to match md2 on 1032-152 [production]
19:35 <godog> upload python-thumbor-wikimedia 1.2 - T161719 [production]
19:01 <godog> bounce pdfrender on scb1001 and scb1003 - T159922 [production]
18:46 <godog> bounce carbon and uwsgi on graphite1003 [production]
17:53 <reedy@tin> Synchronized wmf-config/db-eqiad.php: fix comment (duration: 00m 47s) [production]
17:52 <reedy@tin> Synchronized wmf-config/db-codfw.php: fix comment (duration: 00m 47s) [production]
17:51 <reedy@tin> Synchronized refresh-dblist: phpcs (duration: 00m 48s) [production]
2017-08-12 §
20:00 <krinkle@tin> Synchronized php-1.30.0-wmf.13/includes/jobqueue/JobQueueGroup.php: T171371 - Log job pushes to bogus wikis (duration: 00m 53s) [production]
16:09 <Reedy> Deleted some bogus user languages from commonswiki.user_properties [production]
15:25 <elukey> powercycle mw2256 (able to use com2 but not to login as root, regular ssh hanging) - T163346 [production]
2017-08-11 §
21:47 <legoktm@tin> Finished scap: (no justification provided) (duration: 07m 09s) [production]
21:41 <jmm@puppetmaster1001> conftool action : set/pooled=inactive; selector: mw2256.codfw.wmnet [production]
21:40 <legoktm@tin> Started scap: (no justification provided) [production]
21:40 <legoktm@tin> scap failed: LockFailedError Failed to acquire lock "/var/lock/scap.unknown-but-probably-mediawiki.lock"; owner is "legoktm"; reason is "Deploying Timeless (try 2) - T154371" (duration: 00m 00s) [production]
21:02 <ebernhardson> unban elastic1017 from elasticsearch cluster [production]
20:52 <bblack> varnish backend restart on cp1099 [production]
20:46 <legoktm@tin> Started scap: Deploying Timeless (try 2) - T154371 [production]
20:42 <legoktm@tin> scap aborted: Deploying Timeless - T154371 (duration: 05m 10s) [production]
20:36 <legoktm@tin> Started scap: Deploying Timeless - T154371 [production]
20:19 <bblack> varnish backend restart on cp1049 + cp1074 (mailbox lag) [production]
19:59 <ebernhardson> ban elastic1017 from eqiad search cluster [production]
18:28 <moritzm> installing subversion security updates [production]
17:19 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool db2075 (duration: 00m 47s) [production]
17:05 <twentyafterfour> Deploying phabricator security update [production]
16:21 <jynus_> stop db1069:s6 replication and dropping frwiki, jawiki, ruwiki [production]
15:41 <jynus_> stopping db2075 to clone it to dbstore2001 [production]
15:39 <moritzm> installing git security updates on trusty (jessie/stretch already fixed) [production]
15:38 <jynus@tin> Synchronized wmf-config/db-codfw.php: Depool db2075 (duration: 00m 48s) [production]
13:51 <elukey> moved the eventbus scap deployment dirs on kafka[12]00[123] to deploy-service:deploy-service to allow scap to depool/pool - T171506 [production]
13:49 <mobrovac@tin> Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 27s) [production]
13:49 <mobrovac@tin> Started deploy [eventlogging/eventbus@41e3418]: (no justification provided) [production]
13:42 <mobrovac@tin> Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 13s) [production]
13:42 <mobrovac@tin> Started deploy [eventlogging/eventbus@41e3418]: (no justification provided) [production]
13:25 <mobrovac@tin> Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 34s) [production]