|
2014-04-17
§
|
| 20:41 |
<manybubbles> |
elastic1016 restarted and not freaking out any more. |
[production] |
| 20:37 |
<_joe_> |
restarting gitblit in order to prevent crippling due to the usual memory leak |
[production] |
| 20:28 |
<manybubbles> |
restarting elastic1016 - it is freaking out. If it happens again I'll dig deeper, but for now I consider it a fluke of the rolling restarts today.... |
[production] |
| 20:20 |
<RobH> |
sorry for the misc-web-lb issues folks, they should be resolved at this time (for now) |
[production] |
| 20:19 |
<paravoid> |
lvs1002/1005: commenting first resolv.conf entry until we have a more permanent fix, restarting pybal |
[production] |
| 20:18 |
<paravoid> |
disabling puppet on lvs1002/lvs1005 |
[production] |
| 19:57 |
<RobH> |
still working on issue |
[production] |
| 19:57 |
<RobH> |
both cp1043 and cp1044 seem online and serving nginx service, but pybal says they are down still working |
[production] |
| 19:46 |
<ottomata> |
power off emery |
[production] |
| 19:40 |
<RobH> |
replacing ticket.wikimedia.org cert/key, apache may hiccup |
[production] |
| 19:33 |
<RobH> |
blog.w.o cert replacement successful |
[production] |
| 19:30 |
<ottomata> |
disabling puppet on emery for decommission |
[production] |
| 19:29 |
<RobH> |
blog.w.o certificate swap (yes, again ;), apache may hiccup |
[production] |
| 19:10 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 19:09 |
<reedy> |
synchronized database lists files: I6fc44d3eb829d656d352dab652148dd327b06679 |
[production] |
| 19:04 |
<ottomata> |
reinstalling elastic1001 |
[production] |
| 18:59 |
<faidon> |
synchronized wmf-config/CommonSettings.php 'reenable CN CrossWiki Hiding' |
[production] |
| 18:58 |
<faidon> |
updated /a/common to {{Gerrit|Ie95165065}}: Reenable CentralNotice CrossWiki Hiding |
[production] |
| 18:58 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 18:34 |
<reedy> |
synchronized wmf-config/InitialiseSettings.php 'Touch for I0c36c65bb9f405e03b84d3f6c6b93acda522c5c9' |
[production] |
| 18:33 |
<reedy> |
synchronized database lists files: I0c36c65bb9f405e03b84d3f6c6b93acda522c5c9 |
[production] |
| 18:30 |
<ottomata> |
switching erbium udp2log instance from consuming multicast relay to unicast direct from varnishes |
[production] |
| 18:21 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf1 |
[production] |
| 18:10 |
<ottomata> |
stopping puppet on elastic1001 and elastic1002, reinstalling elastic1002 |
[production] |
| 18:02 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf22 |
[production] |
| 16:06 |
<reedy> |
Finished scap: testwiki to 1.24wmf1 and build l10n cache (duration: 26m 06s) |
[production] |
| 15:47 |
<anomie> |
synchronized php-1.23wmf22/extensions/VisualEditor 'SWAT: 126913 - backport to wmf22 of critical fixes for the Math extension's VisualEditor tool' |
[production] |
| 15:47 |
<anomie> |
synchronized php-1.23wmf22/extensions/Math 'SWAT: 126913 - backport to wmf22 of critical fixes for the Math extension's VisualEditor tool' |
[production] |
| 15:40 |
<reedy> |
Started scap: testwiki to 1.24wmf1 and build l10n cache |
[production] |
| 15:28 |
<faidon> |
synchronized wmf-config/CommonSettings.php 'disable CN CrossWiki Hiding again' |
[production] |
| 15:27 |
<faidon> |
updated /a/common to {{Gerrit|If74ba5a52}}: Revert "Enable CentralNotice CrossWiki Hiding" |
[production] |
| 15:17 |
<manybubbles> |
updgraded site plugins on Elasticsearch nodes |
[production] |
| 15:04 |
<ottomata> |
reinstalling elastic1016 |
[production] |
| 14:02 |
<reedy> |
updated /a/common to {{Gerrit|I290bd1ea6}}: Remove further pmtpa remnants |
[production] |
| 13:41 |
<manybubbles> |
synced experimental highlighter to elasticsearch nodes - they'll pick it up on restart |
[production] |
| 11:06 |
<reedy> |
synchronized wmf-config/ 'I290bd1ea628563646c02651041fa2cec4a320b56' |
[production] |
| 10:56 |
<mutante> |
lvs3,lvs4,lvs5,lvs6 - shutdown |
[production] |
| 10:42 |
<mutante> |
lvs1, lvs2 shutdown |
[production] |
| 10:16 |
<mutante> |
re-deleting unaccepted salt keys for virt2,5-11 |
[production] |
| 10:11 |
<mutante> |
lvs1-6 - disable puppet,salt,revoke certs,keys |
[production] |
| 08:28 |
<mutante> |
db35,db38 - shutdown |
[production] |
| 07:47 |
<mutante> |
db35,db38, stop puppet and salt, revoke certs,keys |
[production] |
| 07:45 |
<mutante> |
restarting gitblit |
[production] |
| 03:49 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Apr 17 03:49:19 UTC 2014 (duration 49m 18s) |
[production] |
| 03:06 |
<subbu> |
deployed Parsoid 0bccf02c (deploy SHA 5e25f3b05) @ 1:30 pm PST, Apr 16th, 2014 |
[production] |
| 03:02 |
<LocalisationUpdate> |
completed (1.23wmf22) at 2014-04-17 03:02:25+00:00 |
[production] |
| 02:33 |
<LocalisationUpdate> |
completed (1.23wmf21) at 2014-04-17 02:33:47+00:00 |
[production] |
| 02:03 |
<springle> |
stop mysqld on db35 (m1) for decom |
[production] |
| 01:42 |
<springle> |
xtrabackup db63 to db60 |
[production] |