|
2015-09-03
§
|
| 16:21 |
<robh> |
re-enabling puppet on all mw systems |
[production] |
| 16:14 |
<robh> |
disabling puppet on all mw systems for apache config update |
[production] |
| 16:01 |
<jynus> |
performing es3 master switchover from es1009 to es1014 |
[production] |
| 15:40 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: depool es1006 (duration: 00m 12s) |
[production] |
| 15:17 |
<hashar> |
stopping nodepool on labnodepool1001.eqiad.wmnet not ready yet |
[releng] |
| 15:17 |
<hashar> |
stopping nodepool on labnodepool1001.eqiad.wmnet not ready yet |
[production] |
| 15:15 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: es2 master switchover from es1006 to es1011 (eqiad) (duration: 00m 13s) |
[production] |
| 15:14 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: es2 master switchover from es1006 to es1011 (codfw) (duration: 00m 12s) |
[production] |
| 15:07 |
<hashar> |
deleting /mnt/jenkins-workspace/workspace/mediawiki-extensions-qunit on integration-slave-trusty-1016 ( https://phabricator.wikimedia.org/T111369 ) |
[releng] |
| 15:05 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 13s) |
[production] |
| 15:04 |
<demon@tin> |
Synchronized php-1.26wmf21/extensions/Translate/: (no message) (duration: 00m 15s) |
[production] |
| 15:03 |
<hashar> |
rebased operations/puppet on integration puppetmaster |
[releng] |
| 14:55 |
<hashar> |
restarting puppetmaster on integration-puppetmaster |
[releng] |
| 14:51 |
<jynus> |
performing es2 master switchover from es1006 to es1011 |
[production] |
| 14:33 |
<paravoid> |
rebooting msw1-eqiad |
[production] |
| 14:28 |
<twentyafterfour> |
restarted phd (phabricator daemon) to pick up new configuration |
[production] |
| 14:24 |
<paravoid> |
changing IPv6 RA interval/lifetime/virtual-router-only @ eqiad |
[production] |
| 14:21 |
<paravoid> |
rebooting msw1-codfw |
[production] |
| 13:17 |
<paravoid> |
upgrading mr1-esams and mr1-eqiad to newer junos |
[production] |
| 13:13 |
<godog> |
bounce carbon daemons on graphite1001 |
[production] |
| 12:42 |
<chasemp> |
unban elastic1001 and put back in service |
[production] |
| 12:24 |
<chasemp> |
move all shards off of elastic1001 |
[production] |
| 12:24 |
<chasemp> |
disable elastic1001 in lvs as we are gonig to try fw apply round #2 |
[production] |
| 11:01 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1028; increase the load of es1010, es1013 and es1017 (duration: 00m 12s) |
[production] |
| 10:45 |
<jynus> |
applying schema change for ContentTranslation on x1-master "wikishared" |
[production] |
| 10:02 |
<godog> |
reenable puppet on ms-be1* |
[production] |
| 09:16 |
<jynus> |
started profiling mysql queries at phabricator. Only a 1% overhead is expected. |
[production] |
| 09:12 |
<moritzm> |
updated rsyncd firewall rules (see https://gerrit.wikimedia.org/r/235425 for details) |
[production] |
| 09:12 |
<godog> |
stop puppet on ms-be1* after ferm rsync change |
[production] |
| 08:23 |
<godog> |
fixup current graphite retention T96662 |
[production] |
| 07:26 |
<moritzm> |
enabled ferm on dbstore* servers in codfw |
[production] |
| 07:09 |
<valhallasw`cloud> |
and just re-running puppet solves the issue. Sigh. |
[tools] |
| 07:09 |
<valhallasw`cloud> |
last message in puppet.log.1.gz is Error: /Stage[main]/Toollabs::Exec_environ/Package[fonts-ipafont-gothic]/ensure: change from 00303-5 to latest failed: Could not get latest version: Execution of '/usr/bin/apt-cache policy fonts-ipafont-gothic' returned 100: fonts-ipafont-gothic: (...) E: Cache is out of sync, can't x-ref a package file |
[tools] |
| 07:07 |
<valhallasw`cloud> |
err, is empty. |
[tools] |
| 07:07 |
<valhallasw`cloud> |
uppet failure on tools-exec-1215 is CRITICAL 66.67% of data above the critical threshold -- but /var/log/puppet.log doesn't exist?! |
[tools] |
| 06:29 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Sep 3 06:29:35 UTC 2015 (duration 29m 34s) |
[production] |
| 05:46 |
<ircnotifier> |
legoktm: Deployed 1564da8bd2a53f9899e93497f03ba13e4a6b734f Forrestbot → ReleaseTaggerBot wb2-irc |
[tools.wikibugs] |
| 05:40 |
<legoktm> |
switched over config to use "ReleaseTaggerBot" account |
[tools.forrestbot] |
| 03:09 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf21) at 2015-09-03 03:09:20+00:00 |
[production] |
| 03:06 |
<l10nupdate@tin> |
Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 32s) |
[production] |
| 02:45 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf20) at 2015-09-03 02:45:36+00:00 |
[production] |
| 02:39 |
<l10nupdate@tin> |
Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 41s) |
[production] |
| 02:30 |
<bd808> |
Added Jforrester as project member |
[tools.forrestbot] |
| 01:32 |
<krenair@tin> |
Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 12s) |
[production] |
| 00:51 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/235656 |
[releng] |
| 00:36 |
<ori@tin> |
Synchronized php-1.26wmf21/includes/parser/Preprocessor_Hash.php: Idd1acd903: Decline to cache preprocessor items larger than 1 Mb (duration: 00m 11s) |
[production] |
| 00:36 |
<ori@tin> |
Synchronized php-1.26wmf20/includes/parser/Preprocessor_Hash.php: Idd1acd903: Decline to cache preprocessor items larger than 1 Mb (duration: 00m 13s) |
[production] |
| 00:27 |
<RoanKattouw> |
Deployed patch for T111029 |
[production] |
| 00:08 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/235641 |
[releng] |