|
2012-11-23
§
|
| 15:09 |
<mark> |
Dist-upgraded and rebooted all eqiad bits servers (new Varnish package) |
[production] |
| 15:04 |
<logmsgbot_> |
maxsem synchronized php-1.21wmf4/extensions/CategoryTree/ 'https://gerrit.wikimedia.org/r/#/c/34668/' |
[production] |
| 14:57 |
<mark> |
Dist-upgraded and rebooted all pmtpa bits servers (new Varnish package) |
[production] |
| 14:33 |
<mark> |
Dist-upgraded and rebooted all esams bits servers (new Varnish package) |
[production] |
| 14:05 |
<mark> |
Built new varnish 3.0.3plus~rc1-wm6 packages with fixed epoll deadlock, and inserted it into the precise-wikimedia APT repository |
[production] |
| 04:20 |
<Tim> |
on fenari: updated the "search" dsh node group based on nmap -sP and fixed the remaining search servers |
[production] |
| 04:04 |
<Tim> |
oh yeah, and I upgraded lucene to my version with the timeouts, deployed to pmtpa only via puppet |
[production] |
| 04:02 |
<Tim> |
many lucene search servers failed to bind to port 1099 when they were restarted by the upgrade, restarting manually |
[production] |
| 02:24 |
<logmsgbot_> |
LocalisationUpdate completed (1.21wmf4) at Fri Nov 23 02:24:34 UTC 2012 |
[production] |
|
2012-11-22
§
|
| 14:52 |
<apergos> |
msbe8 and msbe10 installed, not yet deployed |
[production] |
| 14:12 |
<hashar> |
jenkins: configure jobbuilder-bot user permissions |
[production] |
| 13:39 |
<hashar> |
Jenkins: updated zuul-bot user permission so it can trigger jobs. |
[production] |
| 12:17 |
<hashar> |
Installed plugins on Jenkins for Zuul deployment: notification, build-timeout |
[production] |
| 12:06 |
<hashar> |
restarted Jenkins to load in the new zuul jobs. |
[production] |
| 09:23 |
<apergos> |
fixed perms on srv284 php-1.*wmf* and wmf-config, test (were 700 instead of 777) |
[production] |
| 02:24 |
<logmsgbot_> |
LocalisationUpdate completed (1.21wmf4) at Thu Nov 22 02:24:22 UTC 2012 |
[production] |
| 01:29 |
<mutante> |
sync-apache srv284-only, start apache (was missing all.conf), repool |
[production] |
| 01:15 |
<mutante> |
rebooting srv193 (test.wp) for upgrade |
[production] |
| 00:56 |
<mutante> |
repooled mw58,mw59 (upgrades) srv284 (hw ticket was resolved, reinstalled) |
[production] |
| 00:07 |
<mutante> |
powercycling kaulen, this time no console output at all |
[production] |
|
2012-11-21
§
|
| 23:47 |
<mutante> |
tmp. depooling and rebooting a few mw5x servers for kernel upgrades, one by one |
[production] |
| 23:37 |
<logmsgbot_> |
preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess 'update post deploy' |
[production] |
| 23:34 |
<logmsgbot_> |
preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess 'update post deploy' |
[production] |
| 22:58 |
<mutante> |
disabled swap on kaulen per Tim's advice |
[production] |
| 22:55 |
<mutante> |
bugzilla back up |
[production] |
| 22:50 |
<mutante> |
powercycling kaulen |
[production] |
| 22:25 |
<logmsgbot_> |
asher synchronized wmf-config/throttle.php 'increased api limit for [[bugzilla:42319|bug 42319]]' |
[production] |
| 21:51 |
<mutante> |
installing package upgrades in the mw55-mw99 range |
[production] |
| 21:17 |
<logmsgbot_> |
preilly Finished syncing Wikimedia installation... : update zero rated mobile access |
[production] |
| 20:56 |
<Ryan_Lane> |
deploying Change If5f6bc33: ([[bugzilla:42334|bug 42334]]) on labsconsole |
[production] |
| 20:47 |
<Ryan_Lane> |
applying changes to fix no creds issue when setting preferences on labsconsole |
[production] |
| 20:47 |
<logmsgbot_> |
preilly Started syncing Wikimedia installation... : update zero rated mobile access |
[production] |
| 20:34 |
<logmsgbot_> |
preilly synchronized php-1.21wmf4/extensions/ZeroRatedMobileAccess 'update post deploy' |
[production] |
| 20:34 |
<logmsgbot_> |
preilly synchronized php-1.21wmf3/extensions/ZeroRatedMobileAccess 'update post deploy' |
[production] |
| 20:00 |
<logmsgbot_> |
reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rest of wikipedias to 1.21wmf4 |
[production] |
| 19:24 |
<mutante> |
installing package upgrades in the srv193-srv199 range |
[production] |
| 18:55 |
<mutante> |
re-signing srv284 on puppetmaster, package installs... |
[production] |
| 18:45 |
<AaronSchulz> |
Copying over all math/timeline files to nas1 that are not already there for DR |
[production] |
| 18:34 |
<AaronSchulz> |
prepared all wikivoyage file directories and containers |
[production] |
| 17:41 |
<Reedy> |
Running scap-recompile on all hosts in mediawiki-installation group |
[production] |
| 16:38 |
<mutante> |
powering on analytics1007 |
[production] |
| 16:29 |
<mutante> |
reinstalling srv284 "last lucid standing" |
[production] |
| 16:24 |
<mark> |
Takeback by nas1001-a completed, snapmirror relationships back in sync |
[production] |
| 16:15 |
<mark> |
Aborted synchronous snapmirror relationships between nas1-a and nas1001-a to initiate cf giveback |
[production] |
| 16:08 |
<mutante> |
powercycling srv284 |
[production] |
| 16:05 |
<mutante> |
remove known troublemakers srv266 and srv284 from dsh group "apaches" |
[production] |
| 15:50 |
<mark> |
Initiated takeover of nas1001-a to nas1001-b |
[production] |
| 15:49 |
<mark> |
nas1001-b back up and running |
[production] |
| 14:46 |
<mark> |
Halted nas1001-b for NIC upgrade |
[production] |
| 09:18 |
<notpeter> |
repooling mw20-mw34 |
[production] |