851-900 of 10000 results (30ms)
2015-10-12 §
08:54 <hashar> zuul-merger process leaked file descriptors and ended up unable to open any more files. Fixed by restarting the service on gallium. https://phabricator.wikimedia.org/T115243 [production]
08:44 <hashar> Zuul CI in trouble. zuul-merger can't not apply patches anymore https://phabricator.wikimedia.org/T115243 [production]
02:33 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Oct 12 02:33:13 UTC 2015 (duration 33m 12s) [production]
02:33 <l10nupdate@tin> LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-12 02:33:13+00:00 [production]
02:29 <l10nupdate@tin> Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 50s) [production]
2015-10-11 §
04:57 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Oct 11 04:57:40 UTC 2015 (duration 57m 39s) [production]
02:28 <l10nupdate@tin> LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-11 02:28:34+00:00 [production]
02:25 <l10nupdate@tin> Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 17s) [production]
2015-10-10 §
06:40 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Oct 10 06:40:04 UTC 2015 (duration 40m 3s) [production]
02:22 <l10nupdate@tin> LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-10 02:22:30+00:00 [production]
02:19 <l10nupdate@tin> Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 05m 58s) [production]
2015-10-09 §
19:33 <ori@tin> Synchronized multiversion/MWWikiversions.php: I9d4cbd3d67: Provide a smooth migration path of dblist files to dblists/ (duration: 01m 13s) [production]
18:13 <krenair@tin> Synchronized php-1.27.0-wmf.2/extensions/OpenStackManager/nova/OpenStackNovaProject.php: https://gerrit.wikimedia.org/r/#/c/244707/ (duration: 01m 13s) [production]
16:04 <godog> rolling restart cassandra test cluster T95253 [production]
15:35 <urandom> performing Cassandra cleanup on restbase-test2003.codfw [production]
15:15 <urandom> bouncing Cassandra on restbase-test2001-a [production]
15:06 <urandom> starting nodetool cleanup on restbase-test2002 [production]
14:37 <jynus> more configuration testing (with puppet disabled) and several mysql restarts on db1022 [production]
13:53 <hashar> Restarted Zuul, had a deadlocked job [production]
11:50 <godog> bounce mathoid on sca100[12], stray instance found not running firejail [production]
11:22 <mobrovac> restbase deploying aaee7c31 [production]
11:16 <godog> force-run puppet on restbase after merging https://gerrit.wikimedia.org/r/#/c/244656/ [production]
11:00 <jynus> restarting db1022's mysql (depooled) for configuration testing [production]
09:17 <jynus> deployed visual glitch fix to dbtree [production]
09:11 <akosiaris> poweroff rhodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing. [production]
09:11 <akosiaris> poweroff sodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing. [production]
08:20 <ori> Purged graphite[12]001:/var/lib/carbon/whisper/servers/*/TcpConnStatesCollector and graphite[12]001:/var/lib/carbon/whisper/servers/*/network/work; cleaning up after https://gerrit.wikimedia.org/r/#/c/244637/ [production]
05:13 <mutante> @RoanKattouw re: sync-file ssh to mira.codfw.wmnet: fixed! sorry. -> T115075#1714464 [production]
02:32 <l10nupdate@tin> Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 11s) [production]
01:24 <catrope@tin> Synchronized php-1.27.0-wmf.2/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add period to try to flush out ResourceLoader issue (duration: 01m 26s) [production]
01:18 <RoanKattouw> Getting failures from sync-file / scap because mira.codfw.wmnet doesn't respond to ssh [production]
01:17 <catrope@tin> Synchronized php-1.27.0-wmf.1/extensions/CentralNotice/resources/subscribing/ext.centralNotice.display.js: Add trailing newline to try to flush out ResourceLoader issue (duration: 02m 15s) [production]
2015-10-08 §
23:56 <catrope@tin> Finished scap: SWAT (duration: 29m 10s) [production]
23:51 <yurik> reverted Kartotherian to HEAD^^ - the service wouldn't start [production]
23:27 <catrope@tin> Started scap: SWAT [production]
23:06 <yurik> deployed kartotherian [production]
18:20 <twentyafterfour@tin> rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.27.0-wmf.2 [production]
16:45 <mobrovac> mathoid deploying 110abaf [production]
15:12 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Fix labs settings for foreign uploads (syncing out so it doesnt surprise future SWATters) [[gerrit:244332]] (duration: 00m 18s) [production]
15:05 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable CX suggestions in ast, bn, ml, nb, ta and ukwiki [[gerrit:244142]] (duration: 00m 17s) [production]
14:53 <paravoid> replacing cr1-codfw<->asw-a-codfw QSFPs [production]
14:48 <reedy@tin> Synchronized docroot and w: (no message) (duration: 00m 17s) [production]
14:13 <jynus> performing schema change on the m4/analytics/eventlogging databases (db1046, db1047, dbstore2002) [production]
13:59 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Reduce db1035 & db1044 weight, repool at 100% db1051 & db1055 (duration: 00m 17s) [production]
13:11 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Reduce db1035 weight (duration: 00m 16s) [production]
12:09 <godog> update facts on puppet compiler from palladium [production]
11:38 <_joe_> rebooting mc2002 [production]
10:39 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool db1055 (duration: 05m 10s) [production]
10:37 <_joe_> rebooting mw1008, in oom spiral [production]
10:15 <paravoid> salt rm /home/*/.ssh/authorized_keys [production]