|
2014-10-21
§
|
| 11:56 |
<godog> |
silenced *-lb.ulsfo.wikimedia.org |
[production] |
| 11:43 |
<godog> |
drained ulsfo via DNS, GTT link problems |
[production] |
| 11:10 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1066, warm up (duration: 00m 06s) |
[production] |
| 09:09 |
<akosiaris> |
enabled icinga-wm again |
[production] |
| 08:50 |
<akosiaris> |
temporarily killed icinga-wm |
[production] |
| 04:21 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Oct 21 04:21:19 UTC 2014 (duration 21m 18s) |
[production] |
| 03:06 |
<LocalisationUpdate> |
completed (1.25wmf4) at 2014-10-21 03:06:47+00:00 |
[production] |
| 02:33 |
<LocalisationUpdate> |
completed (1.25wmf3) at 2014-10-21 02:33:35+00:00 |
[production] |
| 02:26 |
<mutante> |
rebooting iron for upgrades |
[production] |
| 02:12 |
<mutante> |
restarting Apache on strontium |
[production] |
| 01:58 |
<mutante> |
restarting apache on puppetmaster, temp. stopping icinga-wm |
[production] |
| 01:31 |
<K4-713> |
adjusted fraud filters on payments |
[production] |
| 00:32 |
<ori> |
Synchronized wmf-config: Id01fe7aac: Turn off spammy message cache log (duration: 00m 05s) |
[production] |
|
2014-10-20
§
|
| 23:56 |
<awight> |
updated payments to 4cf8eb06a4746478c6424648c94688bf460cf63d |
[production] |
| 23:31 |
<springle> |
upgrade db1004 trusty and reboot |
[production] |
| 23:20 |
<mutante> |
installing package upgrades on iron |
[production] |
| 23:04 |
<demon> |
Synchronized wmf-config/CommonSettings-labs.php: no-op, for completeness (duration: 00m 05s) |
[production] |
| 22:38 |
<awight|purgreen> |
updating payments config with French Snowflake hack |
[production] |
| 22:37 |
<Tim> |
doing load testing on mw1189 |
[production] |
| 22:28 |
<Tim> |
enabled puppet on mw1189 |
[production] |
| 21:08 |
<cscott> |
redis-cli srem "deploy:ocg/ocg:minions" tantalum.eqiad.wmnet |
[production] |
| 21:08 |
<bblack> |
baham running gdnsd-2.1.0 test pkg |
[production] |
| 21:08 |
<bd808> |
Deployed iegreview 203d509 (Disable strict variables for twig) |
[production] |
| 21:01 |
<cscott> |
updated OCG to version ea10c93aca9bc1cae34f284fd74bb05d4b6a8cc6 |
[production] |
| 20:33 |
<jgage> |
ulsfo repooled in dns |
[production] |
| 20:17 |
<jgage> |
ulsfo fpc 1 mic 1 card swap complete |
[production] |
| 20:12 |
<jgage> |
beginning router card swap in ulsfo |
[production] |
| 20:11 |
<subbu> |
deployed Parsoid version d4567e9f |
[production] |
| 20:01 |
<paravoid> |
cr1-ulsfo: reenable ospf/ospf3 (GTT is stable) |
[production] |
| 19:13 |
<mutante> |
importing schema, data, users into mysql for iegreview |
[production] |
| 17:42 |
<apergos> |
all trusty and lucid hosts now running salt 2014.1.11 (this includes labcontrol2001, salt master for future codfw labs) |
[production] |
| 16:48 |
<reedy> |
Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 14s) |
[production] |
| 16:47 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: orwikisource |
[production] |
| 16:44 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 15s) |
[production] |
| 16:44 |
<reedy> |
Synchronized database lists: orwikisource (duration: 00m 13s) |
[production] |
| 16:43 |
<cmjohnson> |
reseating pem3 cr2-eqiad |
[production] |
| 16:43 |
<apergos> |
all precise hosts salt updated to 2014.1.11, this includes tin (deployment) and virt1000 (salt master for labs). Not updated: virt1006 (inaccessible) |
[production] |
| 15:58 |
<anomie> |
Synchronized wmf-config: SWAT: Enable wgSecurePollUseNamespace for testwiki [[gerrit:167592]] (duration: 00m 10s) |
[production] |
| 15:57 |
<anomie> |
Finished scap: (no message) (duration: 18m 22s) |
[production] |
| 15:39 |
<anomie> |
Started scap: (no message) |
[production] |
| 15:39 |
<anomie> |
Synchronized php-1.25wmf4/extensions/SecurePoll/: SWAT: Update SecurePoll for testing on testwiki [[gerrit:167586]] (duration: 00m 10s) |
[production] |
| 15:20 |
<_joe_> |
disabling puppet on mw1189 to do some hhvm testing |
[production] |
| 15:20 |
<anomie> |
Synchronized php-1.25wmf3/resources/lib/oojs-ui/: SWAT: OOJS-UI bug fixes [[gerrit:167344]] (duration: 00m 12s) |
[production] |
| 15:19 |
<anomie> |
Synchronized php-1.25wmf3/extensions/VisualEditor/: SWAT: VE bug fixes [[gerrit:167344]] (duration: 00m 10s) |
[production] |
| 15:08 |
<anomie> |
Synchronized php-1.25wmf4/extensions/VisualEditor/: SWAT: VE bug fixes [[gerrit:167577]] (duration: 00m 10s) |
[production] |
| 14:26 |
<paravoid> |
cr1-ulsfo: deactivating ospf/ospf3 on GTT ulsfo-eqiad link |
[production] |
| 13:26 |
<paravoid> |
cr2-ulsfo: "request chassis mic {off,on}line fpc-slot 1 mic-slot 1" to reboot broken card |
[production] |
| 12:28 |
<apergos> |
upgraded salt master (plus minion) on palladium to 2014.1.11, all neww precise installs will get this version now, other minion upgrades to follow shortly |
[production] |
| 11:51 |
<godog> |
temporarily stopped ircecho/icinga-wm on neon, shower of alarms |
[production] |
| 11:42 |
<godog> |
killed stray/old copy of diamond that was filling up conntrack on virt1000 |
[production] |