2001-2050 of 10000 results (41ms)
2017-01-06 §
04:27 <matt_flaschen> Started FlowFixInconsistentBoards.php (production mode) on all wikis [production]
04:10 <mutante> Icinga now using Letsencrypt cert and all good [production]
03:42 <mutante> icinga - debugging issue with cert change [production]
03:05 <papaul> OS instalaltion on elastic2025-elastic2036 [production]
02:34 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.7) (duration: 13m 23s) [production]
01:27 <bd808> Restarted logstash on logstash1002 (T154732) [production]
01:06 <Krinkle> stream.wikimedia.org problems - nginx responds with HTTP 502 Bad Gateway to most requests [production]
01:01 <ostriches> gerrit: back up from upgrade [production]
01:00 <ostriches> gerrit: down for upgrade [production]
00:13 <mutante> analytics1036, ms-fe1003 - ran puppet to fix Icinga [production]
00:11 <mutante> carbon - stopping ganglia-monitor-aggregator for good [production]
2017-01-05 §
23:49 <mutante> rolling out exim4 upgrades (DSA 3747-1) on all remaining eqiad (all-eqiad) [production]
23:46 <godog> fix root-owned files on puppetmaster1001:/var/lib/git/operations/private/ causing /srv/private post-commit hook to fail [production]
23:18 <mutante> switching eqiad ganglia aggregator - running puppet on install1001 - disabling on carbon, re-enabling puppet across eqiad [production]
23:05 <mutante> temp disabling puppet on all eqiad hosts via salt - during ganglia aggregator switch [production]
22:51 <demon@tin> Synchronized php-1.29.0-wmf.7/extensions/Echo/includes/api: silence api warnings (duration: 02m 46s) [production]
22:40 <demon@tin> Synchronized php-1.29.0-wmf.7/extensions/VisualEditor: silence some api warnings (duration: 02m 48s) [production]
22:22 <demon@tin> Synchronized php-1.29.0-wmf.7/extensions/CodeReview/api/ApiQueryCodeComments.php: silence some warnings (duration: 02m 46s) [production]
21:59 <mutante> rolling out exim4 upgrades (DSA 3747-1) on all remaning ones in codfw (all-codfw) [production]
21:53 <ejegg> updated payments wiki from 21ea9bc4a46c96f84685b33ef06fd1b4c40e797e to 1f9ea809b3c32060222f4f6a70e725c433fe15b7 [production]
21:53 <mutante> mx1001 - upgrading exim4 packages, exim4-daemon-heavy, forcing puppet run [production]
21:35 <mutante> mx2001 - upgrading exim4 packages, daemon-heavey, forcing puppet run [production]
21:22 <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.7 [production]
20:58 <mutante> mendelevium (OTRS) - upgrade exim4 packages, force puppet run [production]
20:45 <mutante> iridium (phabricator) - upgrade exim4 packages, force puppet run [production]
20:29 <demon@tin> Synchronized docroot/wikipedia.org: removing junk 15 stuff (duration: 04m 50s) [production]
20:18 <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.7 [production]
20:07 <demon@tin> Synchronized docroot: Ok last docroot thing for today I promise (duration: 05m 00s) [production]
19:52 <demon@tin> Synchronized docroot: Final bit of this round of docroot cleanup (duration: 05m 00s) [production]
19:44 <thcipriani@tin> Finished scap: SemanticForms l10n cache rebuild for 1.29.0-wmf.7 (duration: 39m 21s) [production]
19:35 <mutante> mira - upgrade exim, python-requests, linux-image [production]
19:29 <mutante> rolled out exim4 upgrades (DSA 3747-1) on contint1001/2001, dbmonitor1001/2001, tungsten, seaborgium, hassium, labstore* [production]
19:05 <thcipriani@tin> Started scap: SemanticForms l10n cache rebuild for 1.29.0-wmf.7 [production]
18:34 <mutante> fermium (lists server) - upgrading exim packages, exim4-daemon-heavy, forcing puppet run [production]
18:29 <yurik@tin> Finished deploy [graphoid/deploy@d20b00e]: (no message) (duration: 03m 11s) [production]
18:26 <yurik@tin> Starting deploy [graphoid/deploy@d20b00e]: (no message) [production]
18:25 <arlolra> Updated Parsoid to 974dd5b3 (T143183, T102134, T113044) [production]
18:25 <mutante> rolling out exim4 upgrades (DSA 3747-1) on prometheus, mwlog, pollux, labmon, lithium, hassaleh, dubnium, graphite1002, tin, serpens, bromine, dataset1001 [production]
18:16 <arlolra@tin> Finished deploy [parsoid/deploy@465f9c4]: Updating Parsoid to 974dd5b3 (duration: 10m 46s) [production]
18:15 <mutante> rolling out exim4 upgrades (DSA 3747-1) on puppetmaster, yubiauth, oresrdb, (oresrdb1001 - Unknown installation error.. eh.. this is new) [production]
18:05 <arlolra@tin> Starting deploy [parsoid/deploy@465f9c4]: Updating Parsoid to 974dd5b3 [production]
17:57 <robh> shutting down mw2075-2089 for decom per T154621 [production]
17:35 <robh> diabling puppet on mw2075-2089 to decommission them today. [production]
17:15 <mutante> rolling out exim4 upgrades (DSA 3747-1) on ruthenium, einsteinium (icinga), etherpad1001, rhodium, | einsteinium: upgrade python packages, kernel | xenon: apt-get autoremove, upgrade python- arcconf, libs... [production]
17:07 <mutante> scandium (zuul merger), upgrade exim, python-requests, kernel version [production]
17:06 <mutante> rolling out exim4 upgrades (DSA 3747-1) on kraz, wdqs2003, wezen, zosma, tegmen, rutherfordium. upgrade kernel and python-requests on zosma [production]
16:59 <mutante> rolling out exim4 upgrades (DSA 3747-1) on all-db-noncore, all-mw-eqiad, restbase-eqiad, kafka-main [production]
16:52 <mutante> rolling out exim4 upgrades (DSA 3747-1) on notebook, lvs-canary, lvs, mw-maintenance, all-mw-codfw [production]
16:33 <mutante> cobalt upgrading exim packages [production]
16:28 <mutante> rolling out exim4 upgrades (DSA 3747-1) on mw-api, swift-fe, swift-be, sca, scb, misc-analytics [production]