|
2011-07-01
§
|
| 14:56 |
<RobH> |
pdns crash after rehup on ns1.wikimedia.org, restarting pdns |
[production] |
| 14:49 |
<RobH> |
dns updated to remove the mgmt ip addresses for srv302+ and replace them with mw1-mw74 |
[production] |
| 14:33 |
<RobH> |
updating dns, removing entries for non-existent srv302+ and replacing them with mw1-mw74 |
[production] |
| 11:18 |
<andrewS> |
motion sensors are being installed today in sdtpa that are going to trip the lights when there is movement. not sure how this will affect cameras. will update once their work is done later today. |
[production] |
| 07:33 |
<andrewS> |
the 3 new racks in pmtpa have power plugs running to them now. |
[production] |
| 05:51 |
<apergos> |
cron job set up on snapshot1 to dump central auth tables (in subdir by date just like all the private wikis) on weekly basis, runs as backup user |
[production] |
| 00:01 |
<RobH> |
disabled the notify requesters on resolve in the hw-decommission queue in RT |
[production] |
|
2011-06-30
§
|
| 20:43 |
<awjrichards> |
synchronized php-1.17/extensions/CentralNotice/special/SpecialBannerLoader.php 'r91218' |
[production] |
| 20:42 |
<awjrichards> |
synchronized php-1.17/extensions/CentralNotice/special/SpecialBannerController.php 'r91218' |
[production] |
| 18:39 |
<RoanKattouw> |
Running extensions/Vector/switchExperimentPrefs.php on enwiki to disable WikiLove for those with the "Exclude from experiments" preference |
[production] |
| 18:38 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove by default on enwiki' |
[production] |
| 18:28 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove on enwiki, non-default initially' |
[production] |
| 18:18 |
<catrope> |
synchronizing Wikimedia installation... Revision: 91190: |
[production] |
| 18:17 |
<RoanKattouw> |
Deploying updates for ArticleFeedback and WikiLove |
[production] |
| 18:09 |
<RobH> |
srv169 still offline, do not repool (set to false in pybal on lvs3) |
[production] |
| 18:00 |
<RobH> |
doing reboot testing on srv169 and puppet testing. server is depooled, disregard notices. |
[production] |
| 17:56 |
<RobH> |
depooled srv169, its not updating its /var/local/apache via puppet for some reason. |
[production] |
| 17:53 |
<catrope> |
synchronized php-1.17/wmf-config/CommonSettings.php 'Add code enabling WikiLove by default if $wmgWikiLoveDefault is true' |
[production] |
| 17:52 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add wmgWikiLoveDefault variable, set to false' |
[production] |
| 17:48 |
<RobH> |
blog.wikimedia.org updated to wp3.1.4 |
[production] |
| 16:48 |
<RobH> |
srv169 reinstalled and returned to service. (lvs3 is not repooling, citing a key error that has since been removed, perhaps pybal nees to be refreshed?) |
[production] |
| 16:25 |
<RobH> |
reinstalling srv169 |
[production] |
| 10:57 |
<mark> |
Changed preferred path for AS14907 (pmtpa/eqiad) - AS43821 (esams) from 14907 2828 6908 43821 to 14907 3257 1299 43821 (bidirectionally) |
[production] |
|
2011-06-29
§
|
| 23:39 |
<awjrichards> |
synchronizing Wikimedia installation... Revision: 91090: |
[production] |
| 23:38 |
<awjr> |
running scap to deploy CentralNotice changes (r91118) |
[production] |
| 23:25 |
<awjrichards> |
synchronized php-1.17/extensions/CentralNotice/centralnotice.css 'r91118' |
[production] |
| 23:24 |
<awjrichards> |
synchronized php-1.17/extensions/CentralNotice/centralnotice.js 'r91118' |
[production] |
| 23:03 |
<RobH> |
db1001-db1048 (minus a few hosts mentioned in rt#1105) are installed, puppetized, and ready to be placed into replication service. |
[production] |
| 22:53 |
<awjrichards> |
synchronized php-1.17/wmf-config/CommonSettings.php 'Making CentralNotice config changes for test wiki (page path and CentralDBname)' |
[production] |
| 21:03 |
<RobH> |
OS install running on db1033-db1048 |
[production] |
| 20:53 |
<RobH> |
db1033-db1048 raid setup, ready for OS installs |
[production] |
| 20:44 |
<mark> |
Started rsync of ms5 commons thumb dir '1' to ms1004 |
[production] |
| 20:23 |
<RobH> |
working on db1033-db1048 (setting up hardware raid) |
[production] |
| 20:10 |
<RobH> |
db1017-db1031 os installed, puppet updated, ready for replication setup (db1023 & db1032 need further troubleshooting) |
[production] |
| 19:54 |
<RobH> |
os install on db1017-db1031 done, running puppet updates |
[production] |
| 19:45 |
<mark> |
Started rsync of ms5 commons thumb dir '0' to ms1004 |
[production] |
| 19:36 |
<mark> |
Deployed ms1004 (eqiad) as caching thumbs server in a similar configuration as ms6.esams; requesting thumbs from ms5 and caching (and purging) them locally |
[production] |
| 19:35 |
<RobH> |
db1023 and db1032 drac unresponsive, rt# 1111 abd 1112 to troubleshoot next onsite |
[production] |
| 19:23 |
<RobH> |
rob is working installs on db1017-db1032 |
[production] |
| 19:08 |
<RobH> |
updated netboot.cfg for private1-b-eqiad broadcast |
[production] |
| 18:46 |
<RobH> |
raid10 setup on db1017-db1027 |
[production] |
| 18:30 |
<catrope> |
synchronizing Wikimedia installation... Revision: 91090: |
[production] |
| 18:27 |
<RoanKattouw> |
Deploying r91089 (jQuery in the head and position=top support) with scap |
[production] |
| 18:11 |
<catrope> |
synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove on testwiki' |
[production] |
| 17:58 |
<mark> |
OS-installed ms1004 |
[production] |
| 17:33 |
<RobH> |
os install running on db1017-db1021 |
[production] |
| 17:22 |
<mark> |
Deployed DNS entries for ms1001-1004 |
[production] |
| 17:20 |
<RobH> |
db1015 and db1016 os installed, puppet updated, ready to go into replication service |
[production] |
| 17:20 |
<RobH> |
corrected dns, pushed db1017 and upwards into eqiad-b-private ip range |
[production] |
| 17:18 |
<mark> |
Caught ms5 doing RAID10 resync check, stopped it using echo idle > /sys/block/md1/md/sync_action |
[production] |