2401-2450 of 10000 results (27ms)
2015-01-19 §
21:51 <bd808> Updated Wikimania scholarships to 1f8d8f8 (disable PDO persistent connections; review display fixes) [production]
19:40 <hoo> Set email of commons user Tatobot to the email of the owning account [production]
19:30 <akosiaris> https://gerrit.wikimedia.org/r/185610 merged, tested on wtp1024, wtp1023, caused 0 problems, rolling out to the rest of parsoid machines [production]
19:23 <akosiaris> disable puppet on wtp* hosts for https://gerrit.wikimedia.org/r/#/c/185610/ merge [production]
17:27 <akosiaris> manually running wikidatajsondump.sh in a screen on datasets1003 after https://gerrit.wikimedia.org/r/185840 was merged [production]
16:21 <springle> Synchronized wmf-config/db-eqiad.php: depool db1060 (duration: 00m 05s) [production]
14:57 <springle> Synchronized wmf-config/db-eqiad.php: repool db1054, warm up (duration: 00m 05s) [production]
03:56 <LocalisationUpdate> ResourceLoader cache refresh completed at Mon Jan 19 03:56:15 UTC 2015 (duration 56m 14s) [production]
02:20 <LocalisationUpdate> completed (1.25wmf15) at 2015-01-19 02:20:40+00:00 [production]
02:20 <l10nupdate> Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) [production]
02:12 <LocalisationUpdate> completed (1.25wmf14) at 2015-01-19 02:12:37+00:00 [production]
02:12 <l10nupdate> Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) [production]
00:43 <andrewbogott> restarted keystone service on virt1000 [production]
2015-01-18 §
04:03 <LocalisationUpdate> ResourceLoader cache refresh completed at Sun Jan 18 04:03:03 UTC 2015 (duration 3m 2s) [production]
02:18 <LocalisationUpdate> completed (1.25wmf15) at 2015-01-18 02:17:57+00:00 [production]
02:17 <l10nupdate> Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) [production]
02:10 <LocalisationUpdate> completed (1.25wmf14) at 2015-01-18 02:10:18+00:00 [production]
02:10 <l10nupdate> Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s) [production]
2015-01-17 §
06:36 <YuviPanda> restarted dnsmasq on labnet1001 (see https://wikitech.wikimedia.org/wiki/Labs_DNS#DHCP_and_internal_DNS for how to) [production]
04:47 <LocalisationUpdate> ResourceLoader cache refresh completed at Sat Jan 17 04:47:42 UTC 2015 (duration 47m 41s) [production]
02:31 <LocalisationUpdate> completed (1.25wmf15) at 2015-01-17 02:30:57+00:00 [production]
02:30 <l10nupdate> Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) [production]
02:18 <LocalisationUpdate> completed (1.25wmf14) at 2015-01-17 02:18:24+00:00 [production]
02:18 <l10nupdate> Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) [production]
02:17 <ori> mw1148: threads in stuck in __lll_lock_wait (); restarted HHVM. [production]
02:16 <krinkle> Synchronized php-1.25wmf14/includes/content/JsonContent.php: Ic1d10393912fcefa22d (duration: 00m 06s) [production]
02:16 <krinkle> Synchronized php-1.25wmf14/resources/src/mediawiki/mediawiki.content.json.css: Ic1d10393912fcefa22d (duration: 00m 06s) [production]
02:14 <krinkle> Synchronized php-1.25wmf15/includes/content/JsonContent.php: Ic1d10393912fcefa22d (duration: 00m 05s) [production]
02:14 <krinkle> Synchronized php-1.25wmf15/resources/src/mediawiki/mediawiki.content.json.css: Ic1d10393912fcefa22d (duration: 00m 06s) [production]
02:10 <ori> Synchronized wmf-config/StartProfiler.php: I4e3871d3d: xenon: Annotate file scope and closure scope with filename (duration: 00m 05s) [production]
00:13 <bd808> restarted elasticserch on logstash1001 & logstash1003; OOM [production]
2015-01-16 §
23:33 <bd808> ran `LTRIM logstash -50000 9999999` on redis queues to drop ~4M events in backlog [production]
22:14 <bd808> restarted elasticsearch on logstash1001; OOM errors [production]
21:21 <bd808> restarted elasticsearch on logstash1001 [production]
21:18 <marktraceur> Finished scap: Fix UploadWizard regression and EventLogging errors (duration: 31m 06s) [production]
21:17 <bd808> OOM for elasticsearch on logstash1001 caused a dropped shard and icinga alerts [production]
20:47 <marktraceur> Started scap: Fix UploadWizard regression and EventLogging errors [production]
20:17 <bd808> Synchronized wmf-config/InitialiseSettings.php: Allow wgDebugLogGroups to exclude logstash append (e808e690) (duration: 00m 05s) [production]
20:17 <bd808> Synchronized wmf-config/logging.php: Allow wgDebugLogGroups to exclude logstash append (e808e690) (duration: 00m 07s) [production]
18:13 <bd808> document count not changing for logstash-2015.01.16 index [production]
17:59 <bd808> Synchronized wmf-config/logging-labs.php: beta: Allow wgDebugLogGroups to exclude logstash append (03c3ab27) (duration: 00m 06s) [production]
17:50 <bblack> depooled amssq42 text cache in esams [production]
17:44 <ejegg> updated tools from 88b57fea517d2232e8ae906df550f426b6574f24 to 84442d51a841af4265ff103827cda83d5dd9dc54 [production]
17:24 <demon> Synchronized wmf-config/: (no message) (duration: 00m 05s) [production]
17:21 <ejegg> updated civicrm from d648ededf5c9fc2b0ebf989300ca2037956418e3 to 4fa10ec9e3afbf65e6cbd523138cdc4b4485c482 [production]
17:17 <demon> Synchronized wmf-config/: (no message) (duration: 00m 06s) [production]
16:48 <ottomata> finished hadoop namenode migration. Hadoop cluster is back online [production]
16:48 <bd808> Upgraded elasticsearch and restarted on all logstash nodes [production]
16:43 <bd808> shutdown whole elasticsearch cluster for logstash [production]
16:39 <bd808> restarted elasticsearch on logstash1001 [production]