3301-3350 of 10000 results (30ms)
2015-08-26 §
16:34 <andrewbogott> stopping keystone, updating db, restarting [production]
16:23 <andrewbogott> correction on that last: Upgrading nova and glance services piecemeal [production]
16:22 <andrewbogott> running dist-upgrade on labcontrol1001 [production]
16:18 <marxarelli> deleted udp2log.log and restarted service. so far nothing out of `tail -fn0 udp2log.log` [releng]
16:18 <andrewbogott> switching labcontrol1001 hiera to Juno which will add the cloud-archive repo for Juno. [production]
16:16 <marxarelli> stopping udp2log on deployment-flourine [releng]
16:14 <marxarelli> udp2log is mostly "egrep: writing output: Broken pipe" [releng]
16:11 <andrewbogott> backing up labs openstack databases into /home/andrew/openstackdbbackups on db1009 [production]
16:11 <andrewbogott> starting labs openstack update to Juno [production]
16:10 <marxarelli> disk space at 97% on deployment-flourine, mainly due to 15G /var/log/udp2log/udp2log.log [releng]
16:01 <bd808> sudo rm -rf integration-slave-precise-1014:/mnt/jenkins-workspace/workspace/mediawiki-core-phplint/.git [releng]
15:53 <moritzm> ferm enabled on elastic1023 [production]
15:45 <godog> repool restbase1009 in pybal [production]
15:28 <thcipriani@tin> Synchronized php-1.26wmf20/extensions/Wikidata: SWAT: Update Wikidata - wrap usage tracking batch updates in transaction [[gerrit:233970]] (duration: 00m 23s) [production]
14:30 <YuviPanda> rebooted wikidata-build1 instance, seems to be down from the disk issues on labvirt1007 from a few days ago [wikidata-build]
13:47 <andrewbogott> rebooting/reimaging labnet1001 [production]
13:11 <mobrovac> restbase deploying 1dfba85 [production]
12:54 <yurik> git synced kartotherian [production]
11:02 <jynus> dropping optin_survey_old table on all wikis [production]
10:33 <godog> reenable puppet on ms-fe/ms-be, base::firewall still not enabled [production]
09:58 <godog> test-reboot ms-be2001 [production]
09:57 <hashar> Bumping our JJB mirror a3aef64..f01628c Required for the Android Emulator plugin support ( https://phabricator.wikimedia.org/T110307 ) [releng]
08:17 <godog> disable puppet on ms-be/ms-fe in preparation for merging firewall changes [production]
07:53 <l10nupdate@tin> ResourceLoader cache refresh completed at Wed Aug 26 07:53:31 UTC 2015 (duration 53m 30s) [production]
07:39 <hashar_> puppet is back in action on beta cluster [releng]
07:38 <hashar_> enabling puppet agent on deployment-puppetmaster. It is disable with no reason given [releng]
07:24 <hashar_> resetted beta cluster puppet master to origin/production . We have lost any cherry pick that might have existed [releng]
07:16 <hashar_> started puppetmaster on deployment-puppetmaster [releng]
07:11 <hashar> puppet fails on most beta cluster instances :-( [releng]
07:01 <jynus> restarting mw1239 HHVM, which is unresponsive [production]
04:47 <ori@tin> Synchronized wmf-config: I73721936: Enable ParsoidBatchAPI everywhere (duration: 00m 13s) [production]
03:11 <l10nupdate@tin> LocalisationUpdate completed (1.26wmf20) at 2015-08-26 03:11:29+00:00 [production]
03:06 <awight@tin> Synchronized wmf-config/InitialiseSettings-labs.php: Push labs config to keep in sync with master (duration: 00m 13s) [production]
03:05 <l10nupdate@tin> Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 45s) [production]
02:37 <l10nupdate@tin> LocalisationUpdate completed (1.26wmf19) at 2015-08-26 02:37:51+00:00 [production]
02:34 <l10nupdate@tin> Synchronized php-1.26wmf19/cache/l10n: l10nupdate for 1.26wmf19 (duration: 06m 29s) [production]
02:00 <ottomata> kafka topic webrequest_upload has finished rebalancing across new brokers. starting move of last topic webrequest_text [production]
01:50 <mattflaschen@tin> Synchronized php-1.26wmf19/extensions/Flow/: Sync Flow for reply fix (duration: 00m 15s) [production]
00:28 <ori@tin> Synchronized php-1.26wmf20/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: (no message) (duration: 00m 13s) [production]
00:26 <ori@tin> Synchronized php-1.26wmf19/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: (no message) (duration: 00m 13s) [production]
00:26 <Danny_B> 2586dd1c7c obviously broke many pages [production]
00:19 <ori@tin> Synchronized php-1.26wmf19/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: (no message) (duration: 00m 14s) [production]
00:14 <ori@tin> Synchronized wmf-config/CommonSettings.php: I79ffa78fa: Collection/OCG: Turn on plain text output format in Book Creator (duration: 00m 12s) [production]
00:12 <ori@tin> Synchronized php-1.26wmf20/extensions/Scribunto/engines/LuaCommon/LuaCommon.php: 2586dd1c7c: Updated mediawiki/core Project: mediawiki/extensions/Scribunto (duration: 00m 13s) [production]
2015-08-25 §
23:48 <thcipriani> stopping puppetmaster and disabling puppet runs on deployment-puppetmaster until we get a change to diagnose/rebuild (tomorrow) [releng]
23:47 <thcipriani> deployment-puppetmaster showing signs of a corrupt disk "error: object file .git/objects/cc/026ba0cdc872490ef6a616b2bac4bb829639cd is empty" shutting it off for now. [releng]
23:43 <thcipriani> reboot deployment-puppetmaster unreachable from other vms (labvirt1007 thing, probably) [releng]
23:39 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233860/ (duration: 00m 12s) [production]
23:16 <krenair@tin> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233872/ (duration: 00m 13s) [production]
23:13 <krenair@tin> Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/232963/ (duration: 00m 12s) [production]