2951-3000 of 10000 results (32ms)
2014-10-01 §
15:27 <anomie> Synchronized php-1.25wmf1/extensions/Wikidata/: SWAT: Fix js error that breaks editing properties on Wikidata [[gerrit:164079]] (duration: 00m 16s) [production]
15:12 <anomie> Synchronized wmf-config: SWAT: Disable the old mwlib PDF render service [[gerrit:163609]] (duration: 00m 09s) [production]
15:05 <andrewbogott> switched icinga over to the new ldap servers. Seems to still work so far... [production]
15:02 <godog> switched syslog to lithium [production]
15:01 <anomie> Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Enable thumbnail prerendering at upload time on Beta [[gerrit:163836]] (duration: 00m 09s) [production]
14:59 <hashar> Jenkins changed git executable path from 'git' to '/usr/bin/git' [production]
14:41 <godog> testing syslog change on mw1060 [production]
14:40 <mark> Shutdown mchenry [production]
14:32 <andrewbogott> turning virt0 off again. Soon we won't have a choice about this, trying to flush out issues in the meantime. [production]
14:27 <bblack> mexia powered off [production]
14:08 <mark> Stopped pdns_recursor on mchenry [production]
14:02 <mark> Shutting down dobson [production]
13:49 <godog> temporarily override syslog.eqiad.wmnet on mw1053 for testing [production]
13:28 <mark> Stopped DNS recursor on dobson [production]
13:22 <mark> Stopped OpenDJ on sanger [production]
11:58 <godog> reboot lithium for installation [production]
10:38 <_joe_> re-enabled puppet on mw1018, repooling in a few [production]
10:16 <paravoid> killed cp4006's stale puppet agent_disabled.lock, ran puppet [production]
10:16 <akosiaris> started spare disk zeroing process on nas1-a [production]
10:16 <akosiaris> destroyed backups aggregate on nas1-a [production]
09:58 <akosiaris> destroyed baculasd1, baculasd2 and fr_archive volumes on nas1 [production]
09:47 <akosiaris> umount /home on fenari. fenari user homes no longer available [production]
09:42 <akosiaris> touch /etc/nologin on fenari. Non root logins disallowed [production]
09:04 <akosiaris> breaking the snapmirror relationships between nas1-a, nas1001-a. Effect: no more fr_archive syncing, fenari /home no longer is synced [production]
08:35 <_joe_> disabling puppet on mw1018, enabling debug logging to get more details about fcgi reported errors [production]
03:51 <LocalisationUpdate> ResourceLoader cache refresh completed at Wed Oct 1 03:51:23 UTC 2014 (duration 51m 22s) [production]
02:42 <mutante> jenkins config used virt0, login was needed though to change the config. blocked Krinkle [production]
02:38 <mutante> bringing virt0 back up did indeed fix login on jenkins , also analytics-kafka appears to be still using it [production]
02:36 <LocalisationUpdate> completed (1.25wmf1) at 2014-10-01 02:36:43+00:00 [production]
02:31 <mutante> virt0 - powering back up, suspecting it broke jenkins login [production]
02:31 <ori> mw1053 flooding exception logs with: "Unrecognized job type 'EchoNotificationDeleteJob'." Disabling jobrunner & Puppet [production]
02:25 <LocalisationUpdate> completed (1.24wmf22) at 2014-10-01 02:25:11+00:00 [production]
2014-09-30 §
23:29 <spage> Synchronized wmf-config/InitialiseSettings.php: Create log group for Echo (duration: 00m 11s) [production]
23:27 <spage> Synchronized php-1.25wmf1/extensions/Echo: Echo no-op (change reverted) (duration: 00m 09s) [production]
22:55 <ori> re-enabling puppet on mw1019 [production]
22:36 <ori> disabling puppet on mw1019 to enable debug logging in apache [production]
22:09 <mutante> removing linne from DNS - was already shutdown about 24 hours before [production]
21:57 <K4-713> updated prod civicrm to 477a5107a0c93ceac5214 [production]
21:44 <ori> Spike of bitter irony from Nemo_bis on #wikimedia-operations starting 21:43 UTC [production]
21:33 <ori> Synchronized php-1.25wmf1/languages/Language.php: I672c699c (2/2) (duration: 00m 03s) [production]
21:33 <ori> Synchronized php-1.25wmf1/includes/specialpage/SpecialPageFactory.php: I672c699c (1/2) (duration: 00m 07s) [production]
21:23 <Nemo_bis> widespread reproducible 503 errors on wikidata and elsewhere [production]
20:55 <andrewbogott> powering down virt0, just to see what breaks [production]
20:48 <andrewbogott> shutting down pdns on virt0 [production]
20:48 <andrewbogott> shutting down opendj on virt (temporary, a preview of tomorrow) [production]
18:50 <reedy> Synchronized wmf-config/: (no message) (duration: 00m 16s) [production]
18:49 <reedy> Synchronized database lists: (no message) (duration: 00m 15s) [production]
18:41 <mutante> pc1001-1003 - can't generate tmp files for percona monitoring checks -> puppet fail [production]
18:24 <mutante> killing silver from icinga and puppet [production]
18:23 <reedy> Synchronized wmf-config/: (no message) (duration: 00m 16s) [production]