|
2016-01-20
ยง
|
| 20:37 |
<akosiaris> |
s#/dev/md1#/dev/mapper/tank-data# on labvirt1010, reverted by puppet with Notice: /Stage[main]/Role::Labs::Openstack::Nova::Compute/Mount[/var/lib/nova/instances]/device: device changed '/dev/mapper/tank-data' to '/dev/md1' |
[production] |
| 20:37 |
<akosiaris> |
s#/dev/md1#/dev/mapper/tank-data# |
[production] |
| 19:32 |
<dduvall@tin> |
dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.11 |
[production] |
| 19:13 |
<marxarelli> |
including labswiki and labtestwiki in group1 promotion after all |
[production] |
| 19:09 |
<marxarelli> |
starting promotion of group1, but holding back labswiki and labtestwiki until Jan 21 'all' promotion |
[production] |
| 18:54 |
<paravoid> |
manually triggering an ubuntu mirror update ("sudo -u mirror /usr/local/sbin/update-ubuntu-mirror" on carbon) |
[production] |
| 18:41 |
<jynus> |
schema change on wikidatawiki (wb_terms) finished- slaves already catching up |
[production] |
| 18:34 |
<mutante> |
restart hhvm on mw1206 |
[production] |
| 18:32 |
<godog> |
bounce stuck hhvm on mw1205 |
[production] |
| 18:06 |
<paravoid> |
turning up BGP with Zayo in codfw |
[production] |
| 17:48 |
<jynus> |
restarting replication on db1026 after schema change |
[production] |
| 17:09 |
<gwicke> |
restbase cassandra: set DTCS max_window_size_seconds to 70736000, large enough to accommodate a two-year window |
[production] |
| 16:56 |
<thcipriani@tin> |
thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set default graph vega version back to 1 [[gerrit:265289]] (duration: 00m 32s) |
[production] |
| 16:46 |
<thcipriani@tin> |
thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add davidabian.com to wgCopyUploadsDomains [[gerrit:265286]] (duration: 00m 32s) |
[production] |
| 16:42 |
<thcipriani@tin> |
thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Change default graph version param. Part II [[gerrit:265282]] (duration: 00m 32s) |
[production] |
| 16:42 |
<thcipriani@tin> |
thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Change default graph version param. Part I [[gerrit:265282]] (duration: 00m 36s) |
[production] |
| 16:33 |
<thcipriani@tin> |
thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add davidabian.com to wgCopyUploadsDomains [[gerrit:259003]] (duration: 00m 32s) |
[production] |
| 16:21 |
<thcipriani@tin> |
thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add *.bodleian.ox.ac.uk to wgCopyUploadsDomains [[gerrit:265165]] (duration: 00m 33s) |
[production] |
| 16:19 |
<thcipriani@tin> |
thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add *.archives.gov to wgCopyUploadsDomains [[gerrit:265163]] (duration: 00m 32s) |
[production] |
| 16:13 |
<godog> |
bounce hhvm on mw1191 and syntaxlight runaway processes |
[production] |
| 16:05 |
<thcipriani@tin> |
thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable active gadget user stats on enwiki since it takes too long [[gerrit:265185]] (duration: 00m 32s) |
[production] |
| 14:52 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.11/vendor/: Fix ?PHP properly from commit (duration: 00m 36s) |
[production] |
| 14:50 |
<godog> |
powercycle mw1123, hhvm oom |
[production] |
| 14:47 |
<ema> |
Finished reverting migration of mobile traffic to text cluster in codfw https://phabricator.wikimedia.org/T109286 |
[production] |
| 14:24 |
<hoo@tin> |
hoo@tin Synchronized wmf-config/db-eqiad.php: Set db1045 load to 0 (duration: 00m 32s) |
[production] |
| 14:23 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.11/: consistency (duration: 02m 38s) |
[production] |
| 14:15 |
<hoo@tin> |
hoo@tin Synchronized wmf-config/db-eqiad.php: Re-Pool lagged db1045 (duration: 00m 35s) |
[production] |
| 14:14 |
<_joe_> |
syncronizing /srv/deployment manually between the two deployment servers for the first time |
[production] |
| 14:11 |
<hoo@tin> |
hoo@tin Synchronized wmf-config/db-eqiad.php: Has not been synced before (duration: 00m 32s) |
[production] |
| 14:07 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.10/: consistency (duration: 02m 38s) |
[production] |
| 13:58 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.11/extensions/Validator/: noop for wikitech deploy (duration: 00m 32s) |
[production] |
| 13:58 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticMediaWiki/: noop for wikitech deploy (duration: 00m 34s) |
[production] |
| 13:57 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.11/extensions/SemanticResultFormats/: noop for wikitech deploy (duration: 00m 33s) |
[production] |
| 13:57 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.10/extensions/SemanticResultFormats/: noop for wikitech deploy (duration: 00m 33s) |
[production] |
| 13:55 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.10/extensions/SemanticMediaWiki/: noop for wikitech deploy (duration: 00m 33s) |
[production] |
| 13:54 |
<reedy@tin> |
reedy@tin Synchronized php-1.27.0-wmf.10/extensions/Validator/: noop for wikitech deploy (duration: 00m 33s) |
[production] |
| 13:52 |
<jynus@tin> |
jynus@tin Synchronized wmf-config/db-eqiad.php: Setting db1049 as API node as db1045 is lagged (duration: 00m 33s) |
[production] |
| 13:41 |
<ema> |
Revert migration of mobile traffic to text cluster in codfw https://phabricator.wikimedia.org/T109286 |
[production] |
| 12:55 |
<akosiaris> |
restart hhvm on mw1130 |
[production] |
| 12:43 |
<jynus> |
performing alter table on db1026 (ETA: 5 hours) |
[production] |
| 12:20 |
<jynus@tin> |
jynus@tin Synchronized wmf-config/db-eqiad.php: Setting s5 master as recentchanges role (duration: 00m 32s) |
[production] |
| 12:04 |
<jynus> |
trying schema change on wikidata (wb_terms) |
[production] |
| 09:36 |
<akosiaris> |
gnt-instance modify -H disk_aio=native cygnus.codfw.wmnet |
[production] |
| 09:18 |
<akosiaris> |
offline fr_archive volume on nas1001-a |
[production] |
| 09:15 |
<akosiaris> |
unexport /vol/fr_archive on nas1001-a |
[production] |
| 07:56 |
<_joe_> |
powercycling mw1162, unable to login from console, memory exhaustion |
[production] |
| 07:24 |
<ebernhardson@tin> |
ebernhardson@tin Synchronized php-1.27.0-wmf.10/extensions/CirrusSearch/includes/DataSender.php: stop checking for frozen indices while codfw elasticsearch recovers (duration: 01m 42s) |
[production] |
| 06:24 |
<ebernhardson> |
codfw elasticsearch cluster stopped responding during load test, idling test to see if it recovers |
[production] |
| 03:44 |
<l10nupdate@tin> |
l10nupdate@tin ResourceLoader cache refresh completed at Wed Jan 20 03:44:48 UTC 2016 (duration 7m 29s) |
[production] |
| 03:37 |
<mwdeploy@tin> |
mwdeploy@tin sync-l10n completed (1.27.0-wmf.11) (duration: 16m 21s) |
[production] |