|
2012-03-19
§
|
| 23:20 |
<Ryan_Lane> |
restarting all nginx servers |
[production] |
| 23:20 |
<Ryan_Lane> |
added a new proxy to the ssl configuration to temporarily proxy access to wikimania videos being transcoded |
[production] |
| 21:38 |
<binasher> |
creating "ops" db and related grants on prod db clusters 2-7 to prep rollout of ishmael / pt-digest beyond s1 |
[production] |
| 21:17 |
<binasher> |
started enwiki.revision sha1 alter on production side |
[production] |
| 20:57 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Removing debugging code from MobileFormatter' |
[production] |
| 20:54 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter' |
[production] |
| 20:43 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter' |
[production] |
| 20:36 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Addin more debugging code to MobileFormatter' |
[production] |
| 20:31 |
<awjrichards> |
synchronized php/extensions/MobileFrontend/MobileFormatter.php 'Adding debugging code to MobileFormatter' |
[production] |
| 20:07 |
<catrope> |
synchronized php-1.19/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js '[[rev:114176|r114176]]' |
[production] |
| 19:41 |
<Ryan_Lane> |
bringing virt3 instances back up |
[production] |
| 19:34 |
<binasher> |
deploying new frontend squid conf to add support for mf_useformat cookie [rt 2645] |
[production] |
| 19:18 |
<K4-713> |
CiviCRM 4.1.1 update script finished executing on prod. |
[production] |
| 19:12 |
<Ryan_Lane> |
shutting down virt3 for memory reseating |
[production] |
| 19:09 |
<K4-713> |
Started the CiviCRM 4.1.1 update script on prod. |
[production] |
| 19:08 |
<mark> |
Rebuilding RAID arrays on brewster |
[production] |
| 18:58 |
<K4-713> |
Put production civicrm / drupal instance in offline mode for upgrade |
[production] |
| 18:54 |
<K4-713> |
Disabled all production CiviCRM Jenkins jobs, for CiviCRM upgrade. |
[production] |
| 18:54 |
<cmjohnson1> |
brewster HDD replacement complete |
[production] |
| 18:42 |
<mark> |
Shutting down brewster for HDD replacement |
[production] |
| 18:26 |
<Jeff_Green> |
killed kill-slow-queries on db1008 for the duration of the civicrm upgrade |
[production] |
| 18:19 |
<nikerabbit> |
synchronized php-1.19/includes/Linker.php 'i18ndeploy [[rev:114160|r114160]]' |
[production] |
| 18:19 |
<nikerabbit> |
synchronized php-1.19/extensions/WebFonts/resources/ext.webfonts.fontlist.js 'i18ndeploy [[rev:114160|r114160]]' |
[production] |
| 18:14 |
<mark> |
Running smartctl -t long /dev/sdb on brewster |
[production] |
| 12:58 |
<hashar> |
synchronized php-1.19/includes/SiteStats.php 'Reenable SiteStatsInit::articles() for [[bugzilla:35169|bug 35169]]. SiteStatsInit::doAllAndCommit() still disabled since it breaks the site' |
[production] |
| 10:28 |
<tstarling> |
synchronized wmf-config/PoolCounterSettings.php 'increased max queue from 50 to 100 on reports that the limit was reached on the enwiki main page in normal operation' |
[production] |
| 09:11 |
<mutante> |
nomcom and langcom wikis look kind of broken , redirecting to pages on incubator with "Error: This page is unprefixed! " |
[production] |
| 08:49 |
<mutante> |
making (almost) all private wikis https-only per RT-2565, vi remnant.conf,sync,graceful... |
[production] |
| 07:30 |
<mutante> |
running sync-apache after making a change to remnant.conf to make grants.wm https-only |
[production] |
| 05:09 |
<Ryan_Lane> |
bringing up most instances on virt3, doing so by project priority |
[production] |
| 04:42 |
<Ryan_Lane> |
bringing up all instances on virt4, waiting 30 seconds between instances |
[production] |
| 04:25 |
<Ryan_Lane> |
bringing up all instances on virt2, waiting 30 seconds between instances |
[production] |
| 04:09 |
<Ryan_Lane> |
bringing up all instances on virt1, waiting 30 seconds between instances |
[production] |
| 04:00 |
<Ryan_Lane> |
attempting to bring some instances up |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.19) at Mon Mar 19 02:17:17 UTC 2012 |
[production] |
| 01:15 |
<mutante> |
killed, updated, restarted wikibus bot per request in [[RT:2656]], should have fixed [[bugzilla:18831]] |
[production] |
|
2012-03-18
§
|
| 23:18 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 35308 - Install [[mw:Extension:DynamicPageList (Wikimedia)]] on Portuguese Wikipedia (ptwiki)' |
[production] |
| 19:20 |
<Ryan_Lane> |
stopping all labs instances, manually recovering gluster volume |
[production] |
| 15:39 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 35295 - Missing a in abusefilter-hide-log permission for oversighters' |
[production] |
| 10:49 |
<Ryan_Lane> |
rebooting virt4 thanks to defunct libvirt process |
[production] |
| 03:43 |
<Ryan_Lane> |
bringing all labs instances up |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.19) at Sun Mar 18 02:18:51 UTC 2012 |
[production] |
| 01:09 |
<Ryan_Lane> |
rebooting all of the virt hosts, gluster is having major issues |
[production] |
| 00:43 |
<Ryan_Lane> |
rebooting virt2 |
[production] |
| 00:40 |
<Ryan_Lane> |
restarting glusterfs on virt2 |
[production] |
| 00:11 |
<Ryan_Lane> |
rebooting virt3 libirt is non-responsive |
[production] |
| 00:00 |
<Ryan_Lane> |
bringing up instances that were downed on virt3 |
[production] |