|
2014-01-29
§
|
| 12:06 |
<andrewbogott> |
deployed (broken!) havana/nova on virt1000, 1001, 1002, 1003, labnet1001. Should be safe, but any recent breakage on virt1000 is most likely a side-effect. |
[production] |
| 09:15 |
<paravoid> |
reenable ospfv3 on the eqiad/esams link |
[production] |
| 08:54 |
<paravoid> |
applying NTP access lists on cr{1,2}-{esams,knams,eqiad,pmtpa,sdtpa,ulsfo}, csw2-esams, pfw1-eqiad |
[production] |
| 07:46 |
<springle> |
powercycled unresponsive db1042, /a tank data mount failed on boot, vgchange -a y + mount + xfs_check. still investigating |
[production] |
| 06:51 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1042 for schema changes' |
[production] |
| 02:58 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-29 02:58:21+00:00 |
[production] |
| 02:31 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-29 02:31:30+00:00 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.23wmf11) at 2014-01-29 02:16:57+00:00 |
[production] |
| 00:34 |
<reedy> |
updated /a/common to {{Gerrit|I2294bac73}}: Throttle now handles IP ranges. |
[production] |
| 00:34 |
<ori> |
another recurrent error in antimony:/var/log/upstart/gitblit.log : "org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed." repeats for each repository. traces: <http://p.defau.lt/?VKPpRReIO7aGDxmHfSJeWw> |
[production] |
| 00:34 |
<reedy> |
synchronized wmf-config/missing.php |
[production] |
| 00:29 |
<ori> |
gitblit on antimony crashed with org.eclipse.jetty.io.EofException. trace: <http://p.defau.lt/?1E8Wwj7_089XS6dtu6nqfA>. lots of java.lang.NullPointerException due to malformed URLs, but these appear to happen continuously. |
[production] |
| 00:20 |
<reedy> |
synchronized wmf-config/ |
[production] |
|
2014-01-28
§
|
| 23:36 |
<Reedy> |
Created FlaggedRevs tables on cewiki |
[production] |
| 22:27 |
<reedy> |
updated /a/common to {{Gerrit|I6b16562d0}}: Rename phase1 dblist to group0 |
[production] |
| 22:19 |
<reedy> |
synchronized database lists files: |
[production] |
| 22:19 |
<reedy> |
synchronized docroot and w |
[production] |
| 20:31 |
<RobH> |
fixing virt0 wikitech cert, wikitech may restart |
[production] |
| 19:46 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'touch' |
[production] |
| 19:46 |
<reedy> |
synchronized database lists files: |
[production] |
| 19:29 |
<reedy> |
synchronized php-1.23wmf11/extensions/Wikibase |
[production] |
| 19:23 |
<reedy> |
updated /a/common to {{Gerrit|I4abcbb866}}: Non Wikipedias to 1.23wmf11 |
[production] |
| 19:22 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Non Wikipedias to 1.23wmf11 |
[production] |
| 18:40 |
<reedy> |
updated /a/common to {{Gerrit|Ic2ea727b4}}: Fixup default file permissions |
[production] |
| 18:36 |
<reedy> |
updated /a/common to {{Gerrit|If45e33ae9}}: Move re-usable code from checkoutMediaWiki to checkoutMediaWiki.php |
[production] |
| 18:20 |
<reedy> |
updated /a/common to {{Gerrit|Iac50ca3fb}}: Deploy Extension:MobileApp to betalabs |
[production] |
| 17:33 |
<cmjohnson1> |
replacing ethernet cable db1024 rt6672 |
[production] |
| 11:45 |
<springle> |
running fresh s5 dump for toolserver on db73 |
[production] |
| 10:41 |
<mutante> |
professor: revoked puppet cert, deleted salt minion key, powering down. bye bye |
[production] |
| 10:34 |
<mutante> |
disabled host and service checks/notifications for professor. running puppet on icinga |
[production] |
| 10:28 |
<mutante> |
removed professor from dsh,puppet,puppetmaster: on palladium Killing professor.pmtpa.wmnet...done. |
[production] |
| 04:19 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1040 for schema changes' |
[production] |
| 04:18 |
<springle> |
synchronized wmf-config/db-pmtpa.php 'depool db1040 for schema changes' |
[production] |
| 03:32 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-28 03:32:36+00:00 |
[production] |
| 03:11 |
<springle> |
synchronized wmf-config/db-pmtpa.php 'rotate s6 master, demote db1027, promote db1023' |
[production] |
| 03:11 |
<springle> |
synchronized wmf-config/db-eqiad.php 'rotate s6 master, demote db1027, promote db1023' |
[production] |
| 02:58 |
<springle> |
synchronized wmf-config/db-eqiad.php 'db1006 to LB 400. prep db1027 for s6 master rotation' |
[production] |
| 02:53 |
<springle> |
wikitech /a full again as per Ryan Lane email to ops@ on 2014-01-14. Deleted two oldest backup sets *2014012[01]*. |
[production] |
| 02:52 |
<LocalisationUpdate> |
completed (1.23wmf11) at 2014-01-28 02:52:02+00:00 |
[production] |
| 02:49 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-28 02:25:44+00:00 |
[production] |
| 01:43 |
<^d> |
gerrit back up, running v2.8.1 stable now |
[production] |
| 01:36 |
<^d> |
gerrit down for upgrade |
[production] |
| 01:26 |
<bsitu> |
synchronized php-1.23wmf11/extensions/Flow 'Update Flow with some special contribs cherry-picks' |
[production] |
| 01:19 |
<ori> |
synchronized php-1.23wmf10/extensions/WikimediaEvents |
[production] |
| 01:18 |
<ori> |
synchronized php-1.23wmf11/extensions/WikimediaEvents |
[production] |
| 01:05 |
<ori> |
synchronized php-1.23wmf10/extensions/WikimediaEvents 'Ifc697cbe6: Revert I829790cd5, removing module storage logging' |
[production] |
| 00:57 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1006, warm up' |
[production] |
| 00:56 |
<ori> |
synchronized php-1.23wmf11/extensions/WikimediaEvents 'Ifc697cbe6: Revert I829790cd5, removing module storage logging' |
[production] |