3301-3350 of 10000 results (27ms)
2015-01-20 §
02:43 <akosiaris> restarted puppet on strontium [production]
02:19 <LocalisationUpdate> completed (1.25wmf15) at 2015-01-20 02:19:04+00:00 [production]
02:19 <l10nupdate> Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 03s) [production]
02:11 <LocalisationUpdate> completed (1.25wmf14) at 2015-01-20 02:10:56+00:00 [production]
02:11 <l10nupdate> Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) [production]
01:06 <paravoid> rebooting netmon1001, upstart stuck(?!) [production]
01:02 <paravoid> power-cycling rhenium [production]
2015-01-19 §
23:46 <springle> Synchronized wmf-config/db-eqiad.php: depool db1063 (duration: 00m 06s) [production]
23:41 <springle> Synchronized wmf-config/db-eqiad.php: repool db1060 (duration: 00m 07s) [production]
23:14 <reedy> Synchronized wmf-config/CommonSettings.php: Remove old texvccheck config (duration: 00m 05s) [production]
21:51 <bd808> Updated Wikimania scholarships to 1f8d8f8 (disable PDO persistent connections; review display fixes) [production]
19:40 <hoo> Set email of commons user Tatobot to the email of the owning account [production]
19:30 <akosiaris> https://gerrit.wikimedia.org/r/185610 merged, tested on wtp1024, wtp1023, caused 0 problems, rolling out to the rest of parsoid machines [production]
19:23 <akosiaris> disable puppet on wtp* hosts for https://gerrit.wikimedia.org/r/#/c/185610/ merge [production]
17:27 <akosiaris> manually running wikidatajsondump.sh in a screen on datasets1003 after https://gerrit.wikimedia.org/r/185840 was merged [production]
16:21 <springle> Synchronized wmf-config/db-eqiad.php: depool db1060 (duration: 00m 05s) [production]
14:57 <springle> Synchronized wmf-config/db-eqiad.php: repool db1054, warm up (duration: 00m 05s) [production]
03:56 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Jan 19 03:56:15 UTC 2015 (duration 56m 14s) [production]
02:20 <LocalisationUpdate> completed (1.25wmf15) at 2015-01-19 02:20:40+00:00 [production]
02:20 <l10nupdate> Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) [production]
02:12 <LocalisationUpdate> completed (1.25wmf14) at 2015-01-19 02:12:37+00:00 [production]
02:12 <l10nupdate> Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) [production]
00:43 <andrewbogott> restarted keystone service on virt1000 [production]
2015-01-18 §
04:03 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Jan 18 04:03:03 UTC 2015 (duration 3m 2s) [production]
02:18 <LocalisationUpdate> completed (1.25wmf15) at 2015-01-18 02:17:57+00:00 [production]
02:17 <l10nupdate> Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) [production]
02:10 <LocalisationUpdate> completed (1.25wmf14) at 2015-01-18 02:10:18+00:00 [production]
02:10 <l10nupdate> Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s) [production]
2015-01-17 §
06:36 <YuviPanda> restarted dnsmasq on labnet1001 (see https://wikitech.wikimedia.org/wiki/Labs_DNS#DHCP_and_internal_DNS for how to) [production]
04:47 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Jan 17 04:47:42 UTC 2015 (duration 47m 41s) [production]
02:31 <LocalisationUpdate> completed (1.25wmf15) at 2015-01-17 02:30:57+00:00 [production]
02:30 <l10nupdate> Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) [production]
02:18 <LocalisationUpdate> completed (1.25wmf14) at 2015-01-17 02:18:24+00:00 [production]
02:18 <l10nupdate> Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) [production]
02:17 <ori> mw1148: threads in stuck in __lll_lock_wait (); restarted HHVM. [production]
02:16 <krinkle> Synchronized php-1.25wmf14/includes/content/JsonContent.php: Ic1d10393912fcefa22d (duration: 00m 06s) [production]
02:16 <krinkle> Synchronized php-1.25wmf14/resources/src/mediawiki/mediawiki.content.json.css: Ic1d10393912fcefa22d (duration: 00m 06s) [production]
02:14 <krinkle> Synchronized php-1.25wmf15/includes/content/JsonContent.php: Ic1d10393912fcefa22d (duration: 00m 05s) [production]
02:14 <krinkle> Synchronized php-1.25wmf15/resources/src/mediawiki/mediawiki.content.json.css: Ic1d10393912fcefa22d (duration: 00m 06s) [production]
02:10 <ori> Synchronized wmf-config/StartProfiler.php: I4e3871d3d: xenon: Annotate file scope and closure scope with filename (duration: 00m 05s) [production]
00:13 <bd808> restarted elasticserch on logstash1001 & logstash1003; OOM [production]
2015-01-16 §
23:33 <bd808> ran `LTRIM logstash -50000 9999999` on redis queues to drop ~4M events in backlog [production]
22:14 <bd808> restarted elasticsearch on logstash1001; OOM errors [production]
21:21 <bd808> restarted elasticsearch on logstash1001 [production]
21:18 <marktraceur> Finished scap: Fix UploadWizard regression and EventLogging errors (duration: 31m 06s) [production]
21:17 <bd808> OOM for elasticsearch on logstash1001 caused a dropped shard and icinga alerts [production]
20:47 <marktraceur> Started scap: Fix UploadWizard regression and EventLogging errors [production]
20:17 <bd808> Synchronized wmf-config/InitialiseSettings.php: Allow wgDebugLogGroups to exclude logstash append (e808e690) (duration: 00m 05s) [production]
20:17 <bd808> Synchronized wmf-config/logging.php: Allow wgDebugLogGroups to exclude logstash append (e808e690) (duration: 00m 07s) [production]
18:13 <bd808> document count not changing for logstash-2015.01.16 index [production]