|
2012-03-19
§
|
| 19:12 |
<Ryan_Lane> |
shutting down virt3 for memory reseating |
[production] |
| 19:09 |
<K4-713> |
Started the CiviCRM 4.1.1 update script on prod. |
[production] |
| 19:08 |
<mark> |
Rebuilding RAID arrays on brewster |
[production] |
| 18:58 |
<K4-713> |
Put production civicrm / drupal instance in offline mode for upgrade |
[production] |
| 18:54 |
<K4-713> |
Disabled all production CiviCRM Jenkins jobs, for CiviCRM upgrade. |
[production] |
| 18:54 |
<cmjohnson1> |
brewster HDD replacement complete |
[production] |
| 18:42 |
<mark> |
Shutting down brewster for HDD replacement |
[production] |
| 18:26 |
<Jeff_Green> |
killed kill-slow-queries on db1008 for the duration of the civicrm upgrade |
[production] |
| 18:19 |
<nikerabbit> |
synchronized php-1.19/includes/Linker.php 'i18ndeploy [[rev:114160|r114160]]' |
[production] |
| 18:19 |
<nikerabbit> |
synchronized php-1.19/extensions/WebFonts/resources/ext.webfonts.fontlist.js 'i18ndeploy [[rev:114160|r114160]]' |
[production] |
| 18:14 |
<mark> |
Running smartctl -t long /dev/sdb on brewster |
[production] |
| 12:58 |
<hashar> |
synchronized php-1.19/includes/SiteStats.php 'Reenable SiteStatsInit::articles() for [[bugzilla:35169|bug 35169]]. SiteStatsInit::doAllAndCommit() still disabled since it breaks the site' |
[production] |
| 10:28 |
<tstarling> |
synchronized wmf-config/PoolCounterSettings.php 'increased max queue from 50 to 100 on reports that the limit was reached on the enwiki main page in normal operation' |
[production] |
| 09:11 |
<mutante> |
nomcom and langcom wikis look kind of broken , redirecting to pages on incubator with "Error: This page is unprefixed! " |
[production] |
| 08:49 |
<mutante> |
making (almost) all private wikis https-only per RT-2565, vi remnant.conf,sync,graceful... |
[production] |
| 07:30 |
<mutante> |
running sync-apache after making a change to remnant.conf to make grants.wm https-only |
[production] |
| 05:09 |
<Ryan_Lane> |
bringing up most instances on virt3, doing so by project priority |
[production] |
| 04:42 |
<Ryan_Lane> |
bringing up all instances on virt4, waiting 30 seconds between instances |
[production] |
| 04:25 |
<Ryan_Lane> |
bringing up all instances on virt2, waiting 30 seconds between instances |
[production] |
| 04:09 |
<Ryan_Lane> |
bringing up all instances on virt1, waiting 30 seconds between instances |
[production] |
| 04:00 |
<Ryan_Lane> |
attempting to bring some instances up |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.19) at Mon Mar 19 02:17:17 UTC 2012 |
[production] |
| 01:15 |
<mutante> |
killed, updated, restarted wikibus bot per request in [[RT:2656]], should have fixed [[bugzilla:18831]] |
[production] |
|
2012-03-18
§
|
| 23:18 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 35308 - Install [[mw:Extension:DynamicPageList (Wikimedia)]] on Portuguese Wikipedia (ptwiki)' |
[production] |
| 19:20 |
<Ryan_Lane> |
stopping all labs instances, manually recovering gluster volume |
[production] |
| 15:39 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 35295 - Missing a in abusefilter-hide-log permission for oversighters' |
[production] |
| 10:49 |
<Ryan_Lane> |
rebooting virt4 thanks to defunct libvirt process |
[production] |
| 03:43 |
<Ryan_Lane> |
bringing all labs instances up |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.19) at Sun Mar 18 02:18:51 UTC 2012 |
[production] |
| 01:09 |
<Ryan_Lane> |
rebooting all of the virt hosts, gluster is having major issues |
[production] |
| 00:43 |
<Ryan_Lane> |
rebooting virt2 |
[production] |
| 00:40 |
<Ryan_Lane> |
restarting glusterfs on virt2 |
[production] |
| 00:11 |
<Ryan_Lane> |
rebooting virt3 libirt is non-responsive |
[production] |
| 00:00 |
<Ryan_Lane> |
bringing up instances that were downed on virt3 |
[production] |
|
2012-03-17
§
|
| 23:50 |
<Ryan_Lane> |
virt3 crashed, powercycling it |
[production] |
| 23:34 |
<reedy> |
synchronized wmf-config/CommonSettings.php 'Remove old comments' |
[production] |
| 23:24 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Remove old comments' |
[production] |
| 23:18 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Simplify config somewhat' |
[production] |
| 23:11 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Simplify config somewhat' |
[production] |
| 23:02 |
<catrope> |
synchronizing Wikimedia installation... : Have to scap for that AFTv5 change to propagate i18n change |
[production] |
| 22:52 |
<catrope> |
synchronized php-1.19/extensions/ArticleFeedbackv5/ '[[rev:114087|r114087]]' |
[production] |
| 21:42 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Bug 35289 - Add wikisource logo to mobile wikisource gateway' |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.19) at Sat Mar 17 02:21:03 UTC 2012 |
[production] |
| 01:23 |
<AaronSchulz> |
FindFilesMissingDBRows.php done, list under aaron/output/missingFileDBRows |
[production] |
| 00:11 |
<AaronSchulz> |
Running FindFilesMissingDBRows.php on all wikis |
[production] |