801-850 of 10000 results (20ms)
2015-01-06 §
10:36 <hashar> The Zuul issues have their root cause in Gerrit which cannot open ReviewDB. Filled as https://phabricator.wikimedia.org/T85916 [production]
10:21 <hashar> Restarted Zuul. Gerrit transiently died out just like ~10 hours ago which locked Zuul entirely [production]
09:41 <springle> started an analytics ETL run on dbstore1002. to disable: set global event_scheduler=0; [production]
09:41 <godog> starting hhvm-profiler-to-carbon on tungsten T85641 [production]
09:16 <mutante> upgraded python version on zirconium [production]
09:13 <hashar> Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 05s) [production]
08:38 <andrewbogott> restarted gerrit service on ytterbium [production]
08:11 <hashar> Zuul stalled for some reason :( [production]
08:07 <andrewbogott> restarted pdns on virt1000 and labcontrol2001 to handle the change to nembus (just in case pdns is upset by change!) [production]
08:07 <andrewbogott> moved codfw ldap service to nembus [production]
02:03 <aude> Finished scap: Add Wikidata other projects message (duration: 37m 18s) [production]
01:26 <aude> Started scap: Add Wikidata other projects message [production]
01:23 <aude> Synchronized php-1.25wmf13/extensions/WikimediaMessages: Add Wikidata other projects message (duration: 00m 06s) [production]
01:16 <aude> Synchronized php-1.25wmf13/extensions/MobileFrontend: Fix MobileFrontend bugs (duration: 00m 06s) [production]
00:57 <aude> Synchronized php-1.25wmf12/extensions/MoodBar: Bug fixes for MoodBar (duration: 00m 09s) [production]
00:49 <aude> Synchronized php-1.25wmf13/extensions/MoodBar: Bug fixes for MoodBar (duration: 00m 06s) [production]
00:44 <aude> Synchronized wmf-config/InitialiseSettings.php: Update collabwiki bureaucrat permissions (duration: 00m 06s) [production]
00:25 <greg-g> restarting jenkins, hope that kicks it enough [production]
00:23 <awight> updated payments from 62c81d4574e5e994ff8f3cac7115eff335bd5265 to c23cf16407ef200da446d81fb990abbe429fd378 [production]
00:01 <ori> Synchronized wmf-config/StartProfiler.php: I341a50bef: 'Re-enable xhprof for single-request profiling' (duration: 00m 06s) [production]
2015-01-05 §
22:06 <subbu> deployed parsoid version 0e2997d2 [production]
21:40 <bblack> hard rebooted wtp1020, unresponsive in every way [production]
19:14 <hoo> Made the ruwikinews sites table entry on wikidatawiki use https URLs rather than protocol relative ones [production]
18:41 <YuviPanda> imported trusty pyparsing package into precise-wikimedia [production]
17:44 <hoo> Synchronized php-1.25wmf13/extensions/Wikidata/: Update Wikibase: Fix SpecialEntityData and enhance populateSitesTable (duration: 00m 24s) [production]
17:43 <hoo> Synchronized php-1.25wmf12/extensions/Wikidata/: Update Wikibase: Fix SpecialEntityData and enhance populateSitesTable (duration: 00m 14s) [production]
17:22 <bd808> Synchronized wmf-config/logging-labs.php: Beta logging config change (5b628827) (duration: 00m 06s) [production]
17:09 <bd808> Synchronized wmf-config/logging-labs.php: Beta logging config change (b47ee787) (duration: 00m 06s) [production]
17:00 <bd808> restarted logstash on logstash1001 to see if that will make syslog events come back [production]
16:58 <bd808> syslog events not being recorded in logstash as expected (apache2, hhvm) [production]
16:21 <manybubbles> Synchronized php-1.25wmf13/extensions/VisualEditor/: SWAT fix switching between wikitext and VE on mobile (duration: 00m 14s) [production]
16:14 <manybubbles> Synchronized wmf-config/InitialiseSettings.php: SWAT disable creating books in the wikipedia namespace AND shuffle some upload permissions on kowiki (duration: 00m 05s) [production]
16:12 <manybubbles> Synchronized wmf-config/InitialiseSettings.php: SWAT disable creating books in the wikipedia namespace (duration: 00m 06s) [production]
16:03 <manybubbles> Synchronized wmf-config/Wikibase.php: SWAT Display links to Wikidata in the other project sidebar (duration: 00m 06s) [production]
10:40 <springle> xtrabackup clone: db1037 to db1061, db1039 to db1062 [production]
10:23 <springle> Synchronized wmf-config/db-eqiad.php: depool db1061 db1062 (duration: 00m 06s) [production]
10:14 <springle> Synchronized wmf-config/db-eqiad.php: pool db1057, warm up (duration: 00m 07s) [production]
2015-01-04 §
17:35 <springle> xtrabackup clone db1061 to db1057 [production]
16:49 <springle> restarted zuul [production]
15:59 <springle> upgrade db1057 trusty [production]
15:23 <springle> limiting exim/otrs concurrent connections on m2-master to 250 [production]
14:29 <springle> xtrabackup clone db1020 to db2011 [production]
13:49 <springle> dbproxy1002 failed m2-master traffic over to m2-slave. services up. investigating cause [production]
2015-01-03 §
23:23 <subbu> Try #2: hotfix synced to parsoid cores (to return 500 for urwiki:نام_مقامات_اے); git sha 85d8818ec1b692aaab440630a119c539d63d5ca5 [production]
22:38 <YuviPanda> restarted parsoid on wtp1010 [production]
22:38 <YuviPanda> restarted parsoid on wtp1006 [production]
22:37 <YuviPanda> restarted parsoid on wtp1004 [production]
22:29 <subbu> hotfix synced to parsoid cores (to return 500 for urwiki:نام_مقامات_اے); restart coming next [production]
22:15 <YuviPanda> restarted parsoid on wtp* hosts agian [production]
21:19 <YuviPanda> restarting parsoid on wtp* hosts again [production]