9351-9400 of 10000 results (26ms)
2010-11-23 §
14:40 <mark> Shutdown BGP session to amslvs1 on csw1-esams for testing, traffic failing over to amslvs3 [production]
13:50 <mark> Removed redistribution of BGP LVS routes into OSPF; rely on iBGP exclusively for LVS routes [production]
12:06 <catrope> synchronized php-1.5/includes/api/ApiDelete.php 'r77146' [production]
02:27 <RobH> had to restart morebots, it had left all channels. also had to change admin pass for listserv [production]
2010-11-22 §
22:09 <mark> Restarted varnish on knsq4 [production]
22:05 <mark> Restarting varnish on knsq1 [production]
21:44 <Stifle> Does this work? [production]
21:44 <Ryan_Lane> err - restarting varnish on knsq2,4,5 [production]
21:44 <Ryan_Lane> restarting varnish on knsq2,3,4 [production]
21:43 <Ryan_Lane> restarting knsq2,3,4 [production]
21:29 <mark> Restarted varnish on knsq2,4,5 [production]
21:24 <mark> Restarting Varnish on knsq1 [production]
21:09 <mark> Set static routes 91.198.174.232/31 to 91.198.174.247 (csw1-esams) on br1-knams [production]
21:03 <mark> Started pybal on amslvs1 again [production]
20:53 <mark> Killed PyBal on amslvs1 [production]
20:39 <mark> Lowered cache_mem and cache_dir sizes on amssq31 for testing [production]
20:28 <Ryan_Lane> depooling knsq24 [production]
20:25 <mark> Restarted pybal on amslvs1 with depool-threshold = 1 (temporarily) [production]
17:27 <Ryan_Lane> restarting nagios [production]
17:27 <Ryan_Lane> adding warning notifications via IRC to nagios [production]
16:40 <mark> Suppressed announcements to AS16265 on csw1-esams [production]
16:30 <mark> Suppressed BGP announcements to AS13030 on br1-knams [production]
2010-11-21 §
08:29 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25929 - Import sources for Spanish Wikipedia' [production]
05:49 <tomasz> purging binary logs on db9 to mysql-bin.001600 [production]
05:43 <tomasz> syncing mysql-bin.001562 - mysql-bin.001599 to tridge [production]
2010-11-20 §
21:12 <JeLuF> added download.wikipedia.org as ServerAlias for download.wikimedia.org [production]
20:14 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25982 - Enable subpages in the main namespace for ten.wikipedia.org' [production]
20:11 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25923 - Namespaces on br.wikisource' [production]
20:09 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25929 - Import sources for Spanish Wikipedia' [production]
20:06 <jeluf> synchronized php-1.5/wmf-config/InitialiseSettings.php '25996 - Import sources for pfl.wikipedia' [production]
13:00 <JeLuF> added more error patterns to puppetmon [production]
2010-11-19 §
23:04 <Ryan_Lane> changed https check on payments to 1 retry in nagios, via puppet [production]
20:59 <atglenn> restarted torrus. guess why :-P [production]
17:52 <mark> Readded notification of Service[nagios] when changing nagios types in puppet [production]
13:59 <richcole> set raid 10 up on DB41 [production]
08:36 <catrope> synchronized php-1.5/wmf-config/CommonSettings.php 'bug 25850 - Hide Take me Back link on all wikis' [production]
07:32 <JeLuF> started puppet on sq42 sq41 sq47 sq45 sq46 sq44 sq50 sq52 sq53 sq51 sq48 sq55 sq54 sq43 sq56 sq58 sq64 sq62 sq61 sq65 sq66 sq60 sq63 sq72 sq75 sq71 sq73 sq74 sq76 sq78 sq77 sq81 sq80 sq79 sq83 sq84 sq82 sq85 sq86 [production]
05:21 <apergos> on sq85 I was seeing complaints from cron about restart of puppet: unknown option -w. removed that from /etc/default/puppet and restart, but that fails: Could not parse for environment production: Could not find file /agent.pp [production]
01:00 <atglenn> added monitoring mechanism in root's crontab on sq85 (don't need it everywhere) that will sms me when ms4 is acting up. I'd do it in puppet if someone told me how they would want it to be added there. [production]
00:22 <tomasz> turning db9 watchdog back on. setting at 5minutes [production]
2010-11-18 §
21:57 <richcole> DB42 shutdown for service [production]
21:40 <JeLuF> CORRECTION: started puppet manually on sq59 sq61 sq73 sq60 sq62 sq65 sq77 sq63 sq64 sq72 sq75 sq74 sq76 sq71 sq78 sq66, startup script is broken. [production]
21:40 <JeLuF> started squid manually on sq59 sq61 sq73 sq60 sq62 sq65 sq77 sq63 sq64 sq72 sq75 sq74 sq76 sq71 sq78 sq66, startup script is broken. [production]
21:26 <mark> Fixed puppet on formey [production]
21:24 <mark> Fixed puppet on linne [production]
19:57 <JeLuF> blocked UDP from srv124 on nfs1 aka syslog [production]
17:00 <JeLuF> restarted puppet on srv215, srv235, srv244, srv257, srv262, srv288 [production]
16:33 <JeLuF> fixed puppet on srv185 and srv200 [production]
15:00 <aaron> synchronized php-1.5/wmf-config/flaggedrevs.php 'Set FR_INCLUDES_CURRENT on mediawikiwiki' [production]
2010-11-17 §
20:19 <JeLuF> syslog is being spammed with one week old messages from srv124 [production]