|
2009-07-16
§
|
| 17:55 |
<brion> |
srv171 has some borkage; sudo config is broken can't run apache-restart as user |
[production] |
| 17:52 |
<brion> |
ran sync-common-all |
[production] |
| 17:51 |
<brion> |
running updated sync-common-all friendly to non-NFS boxes |
[production] |
| 17:49 |
<brion> |
swapped private SVN-managed /home/wikipedia/bin into place |
[production] |
| 15:09 |
<apergos> |
removing the last of our snapshots on ms1 :-( getting us a little more space |
[production] |
| 14:47 |
<apergos> |
disabled snapshots on ms1 in preparation for move of thumbnails to ms4 |
[production] |
| 14:38 |
<brion> |
updated wikibugs-l list config to allow bugzilla-daemon@wikimedia.org to post |
[production] |
| 14:34 |
<brion> |
restarted wikibugs bot |
[production] |
| 14:27 |
<brion> |
ms1 performance seems to be sucking again |
[production] |
| 14:17 |
<brion> |
synchronized php-1.5/InitialiseSettings.php 'adjusting throttle temporarily for outreach event' |
[production] |
| 11:55 |
<RoanKattouw> |
ExtensionDistributor repeatedly reported broken in the past 48 hrs |
[production] |
| 07:08 |
<Fred> |
traffic profile switched back to normal. Esams is back to normal. |
[production] |
| 06:11 |
<hcatlin> |
Mobile1 has returned to normal function. |
[production] |
| 05:58 |
<hcatlin> |
Error after restarting mobile1 stopped stats logging from working. Stats will be low for July 15th and higher for July 16th. Parsing of the 6 hour log file (about 1GB) might slow server for next few minutes until caught up. |
[production] |
| 04:24 |
<Rob> |
outage for esams servers started at approx 3:20 gmt |
[production] |
| 04:15 |
<Rob> |
still waiting on esams to update us about the rack(s), moving traffic to pmtpa |
[production] |
| 00:59 |
<tomaszf> |
started backup for latest xml snapshots from storage2 to ms4 |
[production] |
|
2009-07-15
§
|
| 22:30 |
<Rob> |
updated dns for new snapshot servers becasue tomasz did not want to be in charge of dump servers. |
[production] |
| 22:10 |
<brion> |
brion checking around for 0-byte files (not thumbs) to see if we can recover |
[production] |
| 21:33 |
<atglenn> |
verified that zfs patch is in place on ms4 (it got sucked in during river's update yesterday) |
[production] |
| 21:26 |
<brion> |
synchronized php-1.5/CommonSettings.php 'Restore fancy captcha mode' |
[production] |
| 21:16 |
<I_am_not_root> |
synchronized php-1.5/CommonSettings.php 're-enabling Uploads and removing site notice.' |
[production] |
| 21:01 |
<atglenn> |
rebooting ms1 after applying zfs patch. *cross fingers* |
[production] |
| 20:51 |
<brion> |
synchronized php-1.5/CommonSettings.php |
[production] |
| 20:51 |
<brion> |
synchronized php-1.5/InitialiseSettings.php |
[production] |
| 20:42 |
<brion> |
reenabled captcha in simple mode (no images; math q) |
[production] |
| 20:37 |
<brion> |
captcha system broken while images are offline, need to disable it temporarily |
[production] |
| 20:18 |
<brion> |
updated http://en.wikipedia.org/wiki/MediaWiki:Uploaddisabledtext & http://commons.wikimedia.org/wiki/MediaWiki:Uploaddisabledtext |
[production] |
| 19:43 |
<fvassard> |
synchronized php-1.5/CommonSettings.php 'Disabling Uploads while ms1 gets fixed (again with an s after upload).' |
[production] |
| 19:40 |
<fvassard> |
synchronized php-1.5/CommonSettings.php 'Disabling Uploads while ms1 gets fixed.' |
[production] |
| 19:40 |
<atglenn> |
bringing solaris up to current patch level on ms1 |
[production] |
| 19:34 |
<brion> |
Ok, we're going to temporarily shut off uploading and unmount the uploads dir while we muck about with ms1. |
[production] |
| 19:14 |
<brion> |
dropping export/upload@daily-2009-07-11_03:10:00 |
[production] |
| 19:08 |
<brion> |
restarting web server on ms1, see if that resets some connections to the backend scalers |
[production] |
| 19:05 |
<brion> |
restarting nfsd on ms1 |
[production] |
| 18:58 |
<brion> |
dropping zfs snapshot export/upload@daily-2009-07-09_03:10:00 |
[production] |
| 18:25 |
<RobH_A90> |
drac and physical setup done for dump1,2,3, will install remotely |
[production] |
| 17:52 |
<RobH_A90> |
updated dns for new dump processing servers public and management ips |
[production] |
| 17:41 |
<Fred> |
bounced apache on srv45 |
[production] |
| 17:37 |
<Fred> |
bounced apache on srv47 |
[production] |
| 17:09 |
<RobH_A90> |
pdf1 is not coming back, working on it |
[production] |
| 16:56 |
<RobH_A90> |
shutting down pdf1 and mobile1 to move their power too, weee |
[production] |
| 16:55 |
<RobH_A90> |
shutting down spence to move |
[production] |
| 16:50 |
<RobH_A90> |
shutting down singer to move its power, blogs and other associated services will be offline for approx. 5 minutes |
[production] |
| 16:47 |
<Andrew> |
Restarting apache on prototype |
[production] |
| 16:46 |
<RobH_A90> |
shutting down grosley for power move |
[production] |
| 16:45 |
<RobH_A90> |
all these power moves are to add the new dump processing servers to the rack |
[production] |
| 16:45 |
<RobH_A90> |
shutting down fenari for power move |
[production] |
| 16:43 |
<RobH_A90> |
shut down eiximenis and erzurumi to move their power |
[production] |
| 16:34 |
<RobH_A90> |
shutting down some servers and moving power around in a4-sdtpa |
[production] |