|
2013-01-22
§
|
| 18:59 |
<mutante> |
restarted pdns on ns1, failed during update |
[production] |
| 18:58 |
<mutante> |
DNS update - adding ca./ja. planet to zirconium for testing new planet |
[production] |
| 18:51 |
<mutante> |
DNS update - deleting wikimania2006/2007 |
[production] |
| 18:14 |
<root> |
synchronized wmf-config/db-eqiad.php 'eqiad read-write!' |
[production] |
| 17:59 |
<notpeter> |
moving jobrunning to eqiad |
[production] |
| 17:54 |
<aaron> |
synchronized php-1.21wmf7/includes/Block.php 'deployed b0d5ff9bfb09f15ccbab6849564511d372ecd8d4' |
[production] |
| 17:46 |
<notpeter> |
authdns-update |
[production] |
| 17:42 |
<Jeff_Green> |
db1013 wiped, reimaged, powered off, and un-monitored |
[production] |
| 17:40 |
<asher> |
synchronized wmf-config/db-pmtpa.php 'setting pmtpa to readonly' |
[production] |
| 17:14 |
<notpeter> |
dns svn repo on sockpuppet has changes staged and checked in, not yet deployed on dobson |
[production] |
| 17:05 |
<notpeter> |
staging cname switch for pmtpa/eqiad dbs on sockpuppet |
[production] |
| 15:22 |
<mark> |
Changed AS14907->AS43821 routing |
[production] |
| 12:26 |
<^demon> |
restarted ircecho on manganese |
[production] |
| 10:30 |
<Tim> |
also replaced /etc/sudoers on fenari and hume. Hume needs a puppet change which will follow shortly |
[production] |
| 10:18 |
<hashar> |
gallium : restarted puppet |
[production] |
| 10:14 |
<hashar> |
jenkins updated all plugins and restarting |
[production] |
| 10:09 |
<Tim> |
on searchidx2: replaced /etc/sudoers with the distro default so that /etc/sudoers.d/* from puppet can take effect |
[production] |
| 06:48 |
<Tim> |
on eqiad upload varnishes: purged /wikipedia/it/b/bc/Wiki.png at user request. varnishhtcpd appears to be totally broken. |
[production] |
| 02:48 |
<LocalisationUpdate> |
completed (1.21wmf8) at Tue Jan 22 02:48:37 UTC 2013 |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.21wmf7) at Tue Jan 22 02:26:14 UTC 2013 |
[production] |
|
2013-01-21
§
|
| 23:11 |
<Tim> |
on sockpuppet: fixed puppet checkout, switching branch from mikepatch1 to production, and then did fetch&&rebase for good measure |
[production] |
| 22:52 |
<Ryan_Lane> |
adding virt9-11 entries in dns |
[production] |
| 22:49 |
<reedy> |
synchronized wmf-config/CommonSettings.php |
[production] |
| 20:22 |
<mark> |
Rerouted AS43821->AS14907 traffic |
[production] |
| 19:41 |
<mark> |
Restarting knsq* upload frontends manually |
[production] |
| 19:34 |
<paravoid> |
deploying squid config for upload's /monitoring/ |
[production] |
| 19:14 |
<mark> |
Restarting amssq* upload frontends in a slow loop |
[production] |
| 19:11 |
<mark> |
Restarted oversized frontend on amssq50 |
[production] |
| 18:55 |
<paravoid> |
starting knsq19 backend squid |
[production] |
| 18:50 |
<mark> |
Restarted knsq16 backend minus two disks |
[production] |
| 18:40 |
<mark> |
Started amssq56 squid instances |
[production] |
| 18:38 |
<mark> |
Started knsq16 minus one disk |
[production] |
| 18:37 |
<mark> |
Started knsq18 minus one disk |
[production] |
| 18:32 |
<mark> |
power cycled amssq56 |
[production] |
| 18:28 |
<mark> |
Rebooting knsq24 |
[production] |
| 18:05 |
<mark> |
Rerouted AS43821->AS14907 traffic |
[production] |
| 17:53 |
<paravoid> |
stopping knsq19 backend squid |
[production] |
| 17:46 |
<paravoid> |
restarting knsq19 backend squid |
[production] |
| 17:35 |
<paravoid> |
restarting pybal on lvs1002 |
[production] |
| 17:35 |
<cmjohnson1> |
db1038 swapping bad disk (slot 2) with new disk |
[production] |
| 17:27 |
<mark> |
Rerouted AS14907->AS43821 traffic |
[production] |
| 16:36 |
<paravoid> |
depooling, restarting and repooling ms-fe2/3/4 one by one |
[production] |
| 16:30 |
<paravoid> |
repooling ms-fe1 |
[production] |
| 16:29 |
<mark> |
Rerouted AS43821->AS14907 traffic |
[production] |
| 15:48 |
<reedy> |
synchronized php-1.21wmf8/includes/EditPage.php |
[production] |
| 15:45 |
<reedy> |
synchronized php-1.21wmf7/includes/EditPage.php |
[production] |
| 15:32 |
<paravoid> |
depooling ms-fe1 for testing |
[production] |
| 14:17 |
<hashar> |
gallium: manually installed pyflakes {{gerrit|44974}} |
[production] |
| 11:10 |
<apergos> |
nagios was dead over the weekend (config broken), fixed in puppet and on spence, now back in action |
[production] |
| 09:57 |
<hashar> |
relaying Ryan: he restarted ldap on virt0 (was hung after server restart). nscld was properly falling back to virt1000 but ldap was stuck there too. DNS got restarted. |
[production] |