2101-2150 of 10000 results (37ms)
2017-03-10 §
18:28 <smalyshev@tin> Started deploy [wdqs/wdqs@1f2973c]: Deploy new updater on 1003 for potential connection drop fix [production]
17:28 <ottomata> installed librdkafka 0.9.4 via dpkg -i on cp1052 (cache text) and restarted varnishkafka in preparation for fleet upgrade next week [production]
17:24 <ottomata> installed librdkafka 0.9.4 via dpkg -i on cp1058 (cache text) and restarted varnishkafka in preparation for fleet upgrade next week [production]
16:44 <papaul> oresrdb2002 - signing puppet certs, salt-key, initial run [production]
16:25 <elukey> reboot mw22(5[1-9]|60) to enable mw-cgroup mountpoint [production]
15:58 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1030 - T159414 (duration: 02m 42s) [production]
15:18 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1022 - T159414 (duration: 00m 45s) [production]
15:03 <marostegui> Stop slave db2033 for maintenance - T159707 [production]
14:05 <hashar> contint1001 and contint2001 : Migrating git-daemon to systemd . Would stop zuul merger briefly [production]
13:58 <elukey> added 3 new MW api-appservers (mw2251-53) and 7 new appservers (mw2254-60) to codfw [production]
13:35 <hashar> Restarting Jenkins. Deadlocks in ssh connections. T160168 [production]
07:28 <moritzm> upgrading libarchive on trusty systems (jessie already fixed) [production]
07:13 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Added weight 1 for db1061 - T159414 (duration: 00m 40s) [production]
07:13 <marostegui> Deploy alter table s6 revision table on db1022 - T159414 [production]
07:09 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1022 - T159414 (duration: 00m 41s) [production]
04:29 <mutante> codfw mw jobrunner: they start but then fail again shortly after: mw2248 jobrunner[67314]: [Fri Mar 10 04:23:07 2017] [hphp] [67314:7f6a34b746c0:0:000024] [] LightProcess::closeShadow failed due to exception: Failed in afdt::sendRaw: Broken pipe [production]
04:12 <mutante> more mw appservers ... - systemctl start jobchron, systemctl start jobrunner (both were failed but are now active (running) [production]
04:09 <mutante> mw2155 - systemctl start jobchron, systemctl start jobrunner (both were failed but are now active (running) [production]
04:02 <mutante> mw2249 systemctl start jobrunner - now Active: active (running) [production]
03:56 <mutante> codfw appserver jobrunner service fail related to https://gerrit.wikimedia.org/r/#/c/259660/ ? [production]
03:54 <mutante> codfw appservers showing "systemd degraded" alerts are failed jobrunner service unit. after puppet-agent "Mediawiki::Jobrunner/Package[jobrunner]/ensure) ensure changed..." ..then jobrunner.service: main process exited, code=exited, status=143/n/a [production]
02:51 <AaronSchulz> Restarted job services for 510142425d268df (statsd batching) after monitoring mw1161 [production]
02:39 <l10nupdate@tin> ResourceLoader cache refresh completed at Fri Mar 10 02:39:25 UTC 2017 (duration 5m 28s) [production]
02:33 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.15) (duration: 12m 17s) [production]
00:54 <ppchelko@tin> Finished deploy [trending-edits/deploy@1673068]: Replayed events are purged based on current timestamp T160136 (duration: 06m 24s) [production]
00:48 <ppchelko@tin> Started deploy [trending-edits/deploy@1673068]: Replayed events are purged based on current timestamp T160136 [production]
00:39 <ppchelko@tin> Finished deploy [trending-edits/deploy@a5716b9]: Replayed events are purged based on current timestamp T160136 (duration: 02m 23s) [production]
00:38 <dereckson@tin> Synchronized php-1.29.0-wmf.15/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.ArticleTargetLoader.js: ArticleTargetLoader: wikitext switch shouldn't require FullRestbaseURL (T158692) (duration: 00m 41s) [production]
00:37 <ppchelko@tin> Started deploy [trending-edits/deploy@a5716b9]: Replayed events are purged based on current timestamp T160136 [production]
00:31 <ppchelko@tin> Finished deploy [trending-edits/deploy@a5716b9]: Replayed events are purged based on current timestamp T160136 (duration: 07m 17s) [production]
00:30 <eileen> update CiviCRM from d20ed40b52180252a99ecbcb4bb24308e10bc1a4 to 31f19d6901fc6e4bf3b8a57ecde0ea62c1256c95 [production]
00:24 <ppchelko@tin> Started deploy [trending-edits/deploy@a5716b9]: Replayed events are purged based on current timestamp T160136 [production]
00:22 <dereckson@tin> Synchronized wmf-config/CommonSettings.php: Move NavigationTiming config to EventLogging section + Remove setting of unused $wgPercentHHVM ([[Gerrit:342147]] and [[Gerrit:342149]], no-op) (duration: 00m 40s) [production]
00:19 <maxsem@tin> Finished deploy [tilerator/deploy@160f314]: https://gerrit.wikimedia.org/r/#/c/342153/ - revert submodule updates due to broken manik->libc dependency (duration: 00m 16s) [production]
00:19 <maxsem@tin> Started deploy [tilerator/deploy@160f314]: https://gerrit.wikimedia.org/r/#/c/342153/ - revert submodule updates due to broken manik->libc dependency [production]
2017-03-09 §
22:50 <mutante> prometheus1003/1004 - systemctl stop prometheus (as opposed to /etc/init.d/prometheus), as they are low on disk but are not in production yet [production]
22:49 <maxsem@tin> Finished deploy [tilerator/deploy@fb06c99]: https://gerrit.wikimedia.org/r/#/c/342140/ (duration: 00m 05s) [production]
22:48 <maxsem@tin> Started deploy [tilerator/deploy@fb06c99]: https://gerrit.wikimedia.org/r/#/c/342140/ [production]
22:46 <mutante> prometheus1003 - stopping service: [....] Stopping monitoring system and time series database: prometheusInvalid --pidfile argument: '/var/run/prometheus/prometheus.pid' (Parent directory does not exist) [production]
22:46 <maxsem@tin> Finished deploy [tilerator/deploy@fb06c99]: https://gerrit.wikimedia.org/r/#/c/342140/ (duration: 00m 21s) [production]
22:45 <maxsem@tin> Started deploy [tilerator/deploy@fb06c99]: https://gerrit.wikimedia.org/r/#/c/342140/ [production]
22:18 <maxsem@tin> Finished deploy [tilerator/deploy@367df80]: no-op (duration: 00m 22s) [production]
22:18 <maxsem@tin> Started deploy [tilerator/deploy@367df80]: no-op [production]
22:00 <mobrovac@tin> Finished deploy [trending-edits/deploy@57a654e]: Bump max_pages for T156411 (duration: 06m 07s) [production]
21:54 <mobrovac@tin> Started deploy [trending-edits/deploy@57a654e]: Bump max_pages for T156411 [production]
21:37 <mutante> fluorine - puppet node clean, puppet node deactivate, salt-key -d, remove from Icinga.. (T159996) [production]
21:35 <mutante> fluorine - shutdown -h now (decom) T159996 [production]
20:09 <twentyafterfour@tin> rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.15 [production]
20:02 <mutante> cobalt: remove crontab entry of user gerrit2 that created reviewer counts, gzip /var/www/reviewer-counts.json and moved to /root/ for backup (re: gerrit:341592) T54329 [production]
19:53 <reedy@tin> Synchronized php-1.29.0-wmf.15/extensions/ConfirmEdit: Fixup maintenance script (duration: 00m 43s) [production]