|
2013-03-14
§
|
| 13:15 |
<mark> |
Set OSPF cost to 10 on csw1-esams:ve7 to facilitate ip multipath |
[production] |
| 13:15 |
<mark> |
Fixed OSPF on cr2-knams:xe-0/0/0 - csw1-esams:e8/2 (earlier) |
[production] |
| 12:16 |
<hashar> |
gallium setting up /srv/ as a copy of integration/docroot.git |
[production] |
| 09:03 |
<apergos> |
restarted varnishncsa.vanadium on niobium, it was using 9gb memory or so |
[production] |
| 02:25 |
<LocalisationUpdate> |
completed (1.21wmf11) at Thu Mar 14 02:25:19 UTC 2013 |
[production] |
| 02:20 |
<hashar> |
jenkins: made Zuul to block changes on operations/debs/ircecho whenever pep8/pyflakes fails. |
[production] |
| 01:35 |
<mutante> |
running puppet on celsus to investigate puppet freshness reports from icinga - it started parsoid, but also celsus is a Wikimedia DECOMMISSIONED server (base::decommissioned). |
[production] |
| 01:10 |
<reedy> |
synchronized php-1.21wmf11/cache/l10n/ |
[production] |
| 00:59 |
<reedy> |
synchronized php-1.21wmf11/extensions/Scribunto |
[production] |
| 00:53 |
<reedy> |
synchronized wmf-config/ |
[production] |
|
2013-03-13
§
|
| 23:43 |
<mutante> |
installing package upgrades on zirconium |
[production] |
| 23:41 |
<demon> |
Finished syncing Wikimedia installation... : |
[production] |
| 23:38 |
<Ryan_Lane> |
deploying change 53464 to OpenStackManager on wikitech |
[production] |
| 23:23 |
<RobH> |
no one sync apaches, we wre doing crazy magic on them |
[production] |
| 23:17 |
<demon> |
Started syncing Wikimedia installation... : |
[production] |
| 23:05 |
<py> |
synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad. upgrades over.' |
[production] |
| 23:00 |
<py> |
synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad. upgrades over.' |
[production] |
| 22:53 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 22:44 |
<demon> |
synchronized php-1.21wmf11/extensions/Scribunto/ 'Updating Scribunto to master' |
[production] |
| 22:30 |
<robh> |
gracefulled all apaches |
[production] |
| 21:46 |
<awight> |
payments cluster updated to 61d3f1f |
[production] |
| 21:43 |
<pp-pdf1> |
upgraded mwlib to 0.15.3 |
[production] |
| 21:38 |
<Tim> |
on professor: restarted collector |
[production] |
| 21:21 |
<RobH> |
repooling all ssl servers (except broken ssl3004) in all sites. new unified certificate is now in use. |
[production] |
| 21:15 |
<RobH> |
depooling ssl servers 3 and 4 in each dc site |
[production] |
| 21:14 |
<RobH> |
repooling the first two ssl servers per site |
[production] |
| 21:08 |
<py> |
synchronized wmf-config/lucene-production.php 'temp moving all earch traffic to pmtpa for upgrades in eqiad' |
[production] |
| 21:05 |
<RobH> |
setting first 2 ssl servers per dc site to false in pybal |
[production] |
| 20:21 |
<Ryan_Lane> |
repooling ssl1001 |
[production] |
| 20:20 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Re-enable collection' |
[production] |
| 20:16 |
<Ryan_Lane> |
depooling ssl1001 |
[production] |
| 20:08 |
<LeslieCarr> |
cp1022 unresponsive, power cycling |
[production] |
| 19:44 |
<LeslieCarr> |
switched preferred path between esams and eqiad |
[production] |
| 19:41 |
<LeslieCarr> |
deploying a possibly mobile front end traffic effecting ipv6 apache request firewall filter |
[production] |
| 19:26 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Disable Collection extension' |
[production] |
| 19:08 |
<LeslieCarr> |
site was popedotted ! |
[production] |
| 18:47 |
<py> |
synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqaid' |
[production] |
| 18:31 |
<py> |
synchronized wmf-config/lucene-production.php 'temp moving all earch traffic to pmtpa for upgrades in eqiad' |
[production] |
| 18:15 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Enable scribunto EVERYWHERE' |
[production] |
| 18:09 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf11 |
[production] |
| 17:23 |
<pp-pdf1> |
downgrade to 0.14.3 |
[production] |
| 16:39 |
<pp-pdf1> |
upgraded mwlib to 0.15.1. |
[production] |
| 16:32 |
<apergos> |
cleaned up borked puppet agent on ms-be12 and re-enabled/restarted it |
[production] |
| 12:30 |
<anomie> |
synchronized php-1.21wmf11/extensions/Scribunto/common/Hooks.php 'Fix for bug 46031' |
[production] |
| 12:30 |
<anomie> |
synchronized php-1.21wmf11/extensions/Scribunto/common/Common.php 'Fix for bug 46031' |
[production] |
| 12:07 |
<Danny_B> |
reenabled mail delivery to wikibugs 30 mins ago (disabled automatically for excessive bounces) |
[production] |
| 11:42 |
<apergos> |
restarted morebots |
[production] |
| 01:10 |
<Tim> |
running updateCollation.php on all uca-* wikis in 6 screens on hume |
[production] |
| 00:44 |
<dzahn> |
gracefulled all apaches |
[production] |
| 00:39 |
<Ryan_Lane> |
depooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002 |
[production] |