2851-2900 of 10000 results (33ms)
2016-06-24 §
07:10 <elukey> memcached on mc1007 restarted with growth factor 1.05 (T129963) [production]
03:54 <robh> data copy for labmon1001 verified complete with proper permissions, re-enabling and running puppet to start back up services [production]
03:19 <l10nupdate@tin> ResourceLoader cache refresh completed at Fri Jun 24 03:19:55 UTC 2016 (duration 7m 4s) [production]
03:12 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.7) (duration: 17m 24s) [production]
02:38 <mwdeploy@tin> scap sync-l10n completed (1.28.0-wmf.6) (duration: 17m 08s) [production]
01:22 <bblack> stream.wikimedia.org (RCStream) DNS moved to cache_misc termination. If anyone reports bugs with rcstream services, revert https://gerrit.wikimedia.org/r/295385 [production]
2016-06-23 §
23:16 <maxsem@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/295600/ (duration: 00m 29s) [production]
23:15 <maxsem@tin> Synchronized dblists/mobilemainpagelegacy.dblist: https://gerrit.wikimedia.org/r/#/c/295600/ (duration: 00m 28s) [production]
22:33 <chasemp> reimage labstore1005 post io testing [production]
22:12 <chasemp> powercycle labstore1005 [production]
21:24 <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: group2 wikis to wmf.6 [production]
21:11 <chasemp> silence alerts for labstore1004 for setup [production]
20:31 <ebernhardson> synced out latest logstash-plugins via trebuchet [production]
20:17 <Dereckson> Run initSiteStats.php on cebwiki (T138533) [production]
20:04 <jzerebecki@tin> Synchronized wmf-config/CommonSettings.php: Log PHP/HHVM errors in CLI mode to stderr, not stdout T138291 (duration: 00m 28s) [production]
20:03 <robh> labmon1001 data restore at 100gb 50minutes in, 298gb total for restoration [production]
19:29 <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.7 [production]
19:24 <greg-g> 19:21 < RoanKatto> !log Synced patches for T137288 and T137593 [production]
18:31 <elukey> mw130[0134] - new jobrunners installed and pooled (happened automatically after the fist puppet run) [production]
18:09 <robh> labmon1001 powering down for reimage [production]
17:45 <subbu> finished deploying parsoid sha 18022c96 [production]
17:40 <subbu> synced new code; restarted parsoid on wtp1001 as a canary [production]
17:37 <subbu> starting parsoid deploy [production]
17:29 <robh> labmon1001 cpy changed back to local usb, errors on network transfer for ownership. resumed rsync with append flag to local usb disk. [production]
17:03 <bblack> cache perf tuning marker: start rollout of tcp_no_metrics_save:0 [production]
16:27 <chasemp> remove old log files on ytterbium for T114395 [production]
16:18 <godog> swift: add ms-be202[234] weight 1000 - T136630 [production]
15:30 <thcipriani@tin> Synchronized wmf-config/CommonSettings-labs.php: SWAT: [[gerrit:295580|LABS: Enable geoshapes graph protocol]] (duration: 00m 29s) [production]
15:26 <akosiaris> stop etherpad-lite, etherpad is down [production]
15:16 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:295454|Deploy Compact Language Links as default (Stage 2)]] PART III (duration: 00m 24s) [production]
15:16 <thcipriani@tin> Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:295454|Deploy Compact Language Links as default (Stage 2)]] PART II (duration: 00m 28s) [production]
15:15 <thcipriani@tin> Synchronized dblists/clldefault.dblist: SWAT: [[gerrit:295454|Deploy Compact Language Links as default (Stage 2)]] PART I (duration: 00m 41s) [production]
15:11 <robh> puppet disabled on labmon1001 along with all icinga alerting. data migration to usb in progress via root screen session [production]
15:05 <robh> starting data backup of labmon1001, halting statsite/graphite/carbon-relay on system [production]
14:47 <akosiaris> change the default message in etherpad to indicate problems [production]
14:47 <mobrovac> change-prop deploying 05c72ed24ca [production]
14:45 <akosiaris> debugging etherpad. Started the service with a blank db, looks like it's working [production]
14:38 <akosiaris> stopping etherpad-lite on etherpad1001, disabling puppet [production]
14:32 <jynus> restarting etherpad-lite.service [production]
13:53 <hashar> Zuul/CI are slowly catching up. I had to drop a few changes that got force merged on the SmashPig repo. [production]
13:37 <awight> update SmashPig from a435adeb130217bda8b95d3c5c6331ace8ad1228 to 917138e159f0341e3dfbb35818c3ce479927875b [production]
13:36 <hashar> CI is slowed down due to surge of jobs and lack of instances to build them on ( T133911 ). Queue is 50 for Jessie and 25 for Trusty. [production]
13:30 <jynus> db1059 backup and reimage [production]
13:28 <awight> update SmashPig from c0cc2a1a6062ad8d114473ea1a444786a0d50833 to a435adeb130217bda8b95d3c5c6331ace8ad1228 [production]
13:16 <jynus> running scap pool on mw1301 [production]
13:13 <mobrovac> restarting zotero on sca, 6g mem [production]
13:13 <jynus> running scap pool on mw1300 [production]
13:11 <mobrovac> citoid deploying 0129ab0b [production]
13:11 <elukey> purged some puppet output logs on compiler02.puppet3-diffs.eqiad.wmflabs to free space (disk full) [production]
13:09 <moritzm> depooled jessie image scaler (mw1291) again, works fine, to be permanently pooled on Monday [production]