|
2014-01-30
§
|
| 17:41 |
<cmjohnson1_> |
labnet1001 down for card install |
[production] |
| 17:26 |
<demon> |
synchronized wmf-config/InitialiseSettings.php 'Id0dea6e4: Revert "New extra language for wikidata: Ottoman Turkish (ota)"' |
[production] |
| 17:20 |
<awight_> |
crm updated from 704a029a01e2cbe0e99625f78cd472edbd86cd64 to 7c675028cf9a51d4b2d6c8f0d716c343225eb0db |
[production] |
| 04:09 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1020 for schema changes' |
[production] |
| 03:21 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1042' |
[production] |
| 02:09 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-30 02:09:15+00:00 |
[production] |
| 02:03 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-30 02:02:54+00:00 |
[production] |
| 02:02 |
<LocalisationUpdate> |
completed (1.23wmf11) at 2014-01-30 02:02:05+00:00 |
[production] |
| 01:04 |
<awight> |
crm updated from f71526fa021fe3963a5d94151881f5a8a5c40dab to e0dbcef91fdad7223c78707c83c8163a160776bc |
[production] |
| 00:52 |
<awight> |
rollback crm to f71526fa021fe3963a5d94151881f5a8a5c40dab |
[production] |
| 00:51 |
<awight> |
crm updated from f71526fa021fe3963a5d94151881f5a8a5c40dab to 37c6dc5c5034b196679b4321e7fdc1b452741827 |
[production] |
| 00:40 |
<awight> |
rollback crm to f71526fa021fe3963a5d94151881f5a8a5c40dab |
[production] |
| 00:39 |
<awight> |
updated crm from f71526fa021fe3963a5d94151881f5a8a5c40dab to fa36e6c13397d86455bd4963dff9d91181c05af3 |
[production] |
| 00:18 |
<awight> |
updated payments from 3cda9052104d90a746ba5ca243c22ab8d36e0274 to fac21c9d394f6d7c9da172cee18c805304c8fe1b |
[production] |
|
2014-01-29
§
|
| 23:58 |
<RobH> |
all lists.w.o updates done and now on individual certificate |
[production] |
| 23:26 |
<awight> |
rollback payments to 3cda9052104d90a746ba5ca243c22ab8d36e0274 |
[production] |
| 23:14 |
<awight> |
updated payments from 65d6fa8d44a483a3789dbdee2c4926143c3a2338 to bcf5ec47bd995f89f000f4e57ffeee527eb29e11 |
[production] |
| 22:41 |
<awight> |
updating payments from 3cda9052104d90a746ba5ca243c22ab8d36e0274 to 65d6fa8d44a483a3789dbdee2c4926143c3a2338 |
[production] |
| 22:13 |
<RobH> |
otrs now using its own cert per rt 6702, confirmed working chain |
[production] |
| 21:48 |
<RobH> |
svn.w.o on own cert, confirmed chain is properly functioning |
[production] |
| 21:45 |
<RobH> |
updating svn to use own cert, service disruption may (but shouldnt) occur |
[production] |
| 19:49 |
<cmjohnson1> |
tungsten replacing failing disk at slot 10 |
[production] |
| 19:42 |
<cmjohnson1> |
db1002 replacing disk at slot 1 |
[production] |
| 18:53 |
<manybubbles> |
shutting down searchidx1001 for hardware fix |
[production] |
| 17:27 |
<demon> |
synchronized wmf-config/db-eqiad.php 'Removing underscores from class names' |
[production] |
| 17:27 |
<demon> |
synchronized wmf-config/db-labs.php 'Removing underscores from class names' |
[production] |
| 17:26 |
<demon> |
synchronized wmf-config/db-pmtpa.php 'Removing underscores from class names' |
[production] |
| 17:06 |
<demon> |
synchronized wmf-config/InitialiseSettings.php 'Enable Cirrus for huwiki + some shard config' |
[production] |
| 17:05 |
<demon> |
synchronized wmf-config/CirrusSearch-common.php 'Turn commons file searching back on' |
[production] |
| 16:53 |
<LeslieCarr> |
aggregated labnet1001 secondary port |
[production] |
| 14:50 |
<springle> |
synchronized wmf-config/db-eqiad.php 'db1040 LB full steam' |
[production] |
| 14:14 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1040, warm up' |
[production] |
| 12:06 |
<andrewbogott> |
deployed (broken!) havana/nova on virt1000, 1001, 1002, 1003, labnet1001. Should be safe, but any recent breakage on virt1000 is most likely a side-effect. |
[production] |
| 09:15 |
<paravoid> |
reenable ospfv3 on the eqiad/esams link |
[production] |
| 08:54 |
<paravoid> |
applying NTP access lists on cr{1,2}-{esams,knams,eqiad,pmtpa,sdtpa,ulsfo}, csw2-esams, pfw1-eqiad |
[production] |
| 07:46 |
<springle> |
powercycled unresponsive db1042, /a tank data mount failed on boot, vgchange -a y + mount + xfs_check. still investigating |
[production] |
| 06:51 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1042 for schema changes' |
[production] |
| 02:58 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-29 02:58:21+00:00 |
[production] |
| 02:31 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-29 02:31:30+00:00 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.23wmf11) at 2014-01-29 02:16:57+00:00 |
[production] |
| 00:34 |
<reedy> |
updated /a/common to {{Gerrit|I2294bac73}}: Throttle now handles IP ranges. |
[production] |
| 00:34 |
<ori> |
another recurrent error in antimony:/var/log/upstart/gitblit.log : "org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed." repeats for each repository. traces: <http://p.defau.lt/?VKPpRReIO7aGDxmHfSJeWw> |
[production] |
| 00:34 |
<reedy> |
synchronized wmf-config/missing.php |
[production] |
| 00:29 |
<ori> |
gitblit on antimony crashed with org.eclipse.jetty.io.EofException. trace: <http://p.defau.lt/?1E8Wwj7_089XS6dtu6nqfA>. lots of java.lang.NullPointerException due to malformed URLs, but these appear to happen continuously. |
[production] |
| 00:20 |
<reedy> |
synchronized wmf-config/ |
[production] |