2151-2200 of 10000 results (35ms)
2015-12-18 §
11:40 <hashar> logstash: reorganized list of dashboards per sections https://logstash.wikimedia.org/#/dashboard/elasticsearch/default [production]
09:43 <akosiaris> rebooting planet1001, memory exhaustion, OOM showed up [production]
09:20 <hashar> Killed Zuul entirely, the queues were full / deadlocked. Patches need to be retriggered [production]
06:47 <gwicke> restbase1004: nodetool stop -- COMPACTION to avoid running out of disk space [production]
03:07 <ori@tin> Synchronized php-1.27.0-wmf.9/includes/api/ApiStashEdit.php: ab32f4e740: Make ApiStashEdit use statsd metrics (duration: 00m 49s) [production]
02:29 <l10nupdate@tin> ResourceLoader cache refresh completed at Fri Dec 18 02:29:10 UTC 2015 (duration 6m 55s) [production]
02:22 <mwdeploy@tin> sync-l10n completed (1.27.0-wmf.9) (duration: 08m 45s) [production]
01:52 <ori> re-enabled puppet on rdb* / mc* [production]
01:25 <ori> in preparation for Iaefb2d191e, disabling puppet on mc* and rdb* [production]
01:21 <krinkle@tin> Synchronized docroot and w: (no message) (duration: 00m 32s) [production]
00:53 <catrope@tin> Synchronized php-1.27.0-wmf.9/extensions/Flow: Revert Nuke-Flow integration, doesn't work (duration: 00m 32s) [production]
00:42 <catrope@tin> Synchronized php-1.27.0-wmf.9/extensions/Flow: SWAT: Nuke support for Flow, part 3 (duration: 00m 32s) [production]
00:34 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Add completion suggester to BetaFeatures whitelist (duration: 00m 30s) [production]
00:26 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: grumble grumble touch InitialiseSettings grumble (duration: 00m 30s) [production]
00:25 <catrope@tin> Synchronized php-1.27.0-wmf.9/extensions/Flow: SWAT: Nuke support for Flow, part 2 (duration: 00m 32s) [production]
00:23 <catrope@tin> Synchronized wmf-config/CirrusSearch-production.php: SWAT: enable completion suggester beta on all wikis except wikidata (duration: 00m 30s) [production]
00:23 <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: enable completion suggester beta on all wikis except wikidata (duration: 00m 29s) [production]
00:20 <catrope@tin> Synchronized php-1.27.0-wmf.9/extensions/Nuke/: SWAT: Nuke support in Flow, part 1 (duration: 00m 30s) [production]
00:18 <catrope@tin> Synchronized php-1.27.0-wmf.9/resources/src/mediawiki.messagePoster/mediawiki.messagePoster.factory.js: SWAT: fix error in messagePoster (duration: 00m 29s) [production]
00:17 <catrope@tin> Synchronized php-1.27.0-wmf.9/extensions/MobileFrontend: SWAT: Schema:MobileWebSectionUsage: always log the isTestA field (duration: 00m 31s) [production]
00:08 <catrope@tin> Synchronized wmf-config/CommonSettings.php: SWAT: cleanup (duration: 00m 30s) [production]
00:00 <ori> restarted mathoid on sca1001 [production]
2015-12-17 §
23:42 <mobrovac> mathoid deploying 8d2295 [production]
22:47 <hashar> ssh to tin is back https://gerrit.wikimedia.org/r/#/c/259876/ [production]
22:39 <hashar> Only tin lost SSH user keys apparently due to https://gerrit.wikimedia.org/r/#/c/253465/ overriding the admin::groups to simply "eventlogging-admins" [production]
22:34 <hashar> Ssh User keys are gone on deployment servers ( tin / mira ) [production]
21:33 <hashar> gallium: upgrading Zuul from 2.1.0-60-g1cc37f7-wmf2precise1 .. 2.1.0-60-g1cc37f7-wmf4precise1 . Should be noop, only change zuul-cloner which is not used there [production]
21:29 <mutante> add zuul_2.1.0-60-g1cc37f7-wmf4jessie1 to jessie-wikimedia repo [production]
21:28 <mutante> add zuul_2.1.0-60-g1cc37f7-wmf4trusty1 to trusty-wikimedia repo [production]
21:22 <mutante> add zuul_2.1.0-60-g1cc37f7-wmf4precise1 to precise-wikimedia APT [production]
19:42 <mobrovac> mathoid deploying a2187a6 [production]
19:07 <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.9 [production]
19:05 <godog> disable puppet on graphite2001, brief testing cluster aggregations [production]
19:00 <thcipriani> starting update of all wikis to 1.27.0-wmf.9 [production]
18:11 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Repool x1-slave (db1031), increase db1041 load to 100% (duration: 00m 30s) [production]
18:06 <gwicke> running `nodetool cleanup` on restbase1001 and restbase1005 [production]
18:04 <robh> calcium is supposed to be down, reclaiming to spares, ignore any irc alerts (its in maint mode in icinga) [production]
17:55 <gwicke> running `nodetool cleanup` on restbase1002 [production]
17:18 <jynus> setting mysql db1031 as db2009's master [production]
17:05 <jynus> restarting and reconfiguring mysql at db2009 [production]
16:49 <thcipriani@tin> Synchronized php-1.27.0-wmf.8/extensions/Math: SWAT: Make math usable without RESTbase [[gerrit:259734]] (duration: 00m 30s) [production]
16:41 <jynus> problems with corruption on x1-slave for cebwiki. Fixed them. Will leave db1031 depooled for a while to check they are gone. [production]
16:39 <thcipriani@tin> Synchronized php-1.27.0-wmf.9/extensions/ContentTranslation/includes/Translation.php: SWAT: Fix Undefined index: targetRevisionId in ContentTranslation [[gerrit:259649]] (duration: 00m 29s) [production]
16:23 <thcipriani@tin> Synchronized php-1.27.0-wmf.9/extensions/WikimediaEvents/WikimediaEventsHooks.php: SWAT: Actually define tags for cross-wiki upload A/B test [[gerrit:259729]] (duration: 00m 31s) [production]
16:17 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable cross-wiki upload A/B test in additional languages [[gerrit:259665]] (duration: 00m 30s) [production]
16:11 <thcipriani@tin> Synchronized php-1.27.0-wmf.9/extensions/CirrusSearch/includes/CirrusSearch.php: SWAT: Fix array-to-string conversion [[gerrit:259633]] (duration: 00m 30s) [production]
15:57 <moritzm> stopping opendj LDAP servers on nembus/neptunium (read-only since about days now due to migration to openldap) [production]
15:56 <akosiaris> depool sca1001, playing with cxserver config [production]
15:24 <jynus> reinstall, reboot and reconfigure mysql at db1031 [production]
15:03 <moritzm> installing git security updates [production]