|
2014-08-22
§
|
| 15:54 |
<springle> |
xtrabackup db1056 to db1053 |
[production] |
| 15:53 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1056 while cloning (duration: 00m 07s) |
[production] |
| 15:33 |
<^d> |
elastic1008: fixed /etc/hosts to point to actual IP instead of loopback |
[production] |
| 15:18 |
<springle> |
upgrade & restart db1053, fs check |
[production] |
| 15:08 |
<bd808> |
Still no apache2.log on fluorine or in logstash. Log seems to be available on fenari. |
[production] |
| 14:51 |
<springle> |
switched s1 sanitarium and labsdb replication to db1069:3311 mariadb 10 |
[production] |
| 14:39 |
<mark> |
Removed IPv6 subnets 2620:0:860:1::/64 (squid subnet) and 2620:0:860:3::/64 (sandbox subnet) from cr2-pmtpa configuration |
[production] |
| 04:11 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Aug 22 04:10:47 UTC 2014 (duration 10m 46s) |
[production] |
| 03:19 |
<LocalisationUpdate> |
completed (1.24wmf18) at 2014-08-22 03:18:44+00:00 |
[production] |
| 02:33 |
<LocalisationUpdate> |
completed (1.24wmf17) at 2014-08-22 02:32:11+00:00 |
[production] |
| 00:03 |
<ori> |
Finished scap: SWAT: d3de89777, 7abfe0d5e7, 8ec9853c32b, 476e9e90bd01 (duration: 06m 29s) |
[production] |
|
2014-08-21
§
|
| 23:57 |
<ori> |
Started scap: SWAT: d3de89777, 7abfe0d5e7, 8ec9853c32b, 476e9e90bd01 |
[production] |
| 21:58 |
<ori> |
Synchronized php-1.24wmf17/resources/src/mediawiki/mediawiki.js: I8d27442d1: Workaround for bug introduced by Icf6ede09b (duration: 00m 03s) |
[production] |
| 21:57 |
<manybubbles> |
performing elasticsearch upgrade on elastic1015 |
[production] |
| 21:02 |
<ori> |
Synchronized php-1.24wmf17/resources/src/mediawiki/mediawiki.util.js: Touch resources/src/mediawiki/mediawiki.util.js (duration: 00m 06s) |
[production] |
| 20:44 |
<godog> |
rolling restart of swift-proxy on ms-fe1* |
[production] |
| 20:11 |
<godog> |
restarted swift-proxy on ms-fe1001 |
[production] |
| 19:55 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 13s) |
[production] |
| 19:49 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf18 |
[production] |
| 19:46 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf17 |
[production] |
| 19:44 |
<reedy> |
Finished scap: testwiki to 1.24wmf18 (duration: 34m 01s) |
[production] |
| 19:31 |
<mutante> |
disabled mw1178 in pybal |
[production] |
| 19:27 |
<godog> |
restarted memcached on ms-fe1004 |
[production] |
| 19:23 |
<reedy|webirc> |
mw1178 returned [255]: ssh: connect to host mw1178 port 22: Connection timed out |
[production] |
| 19:23 |
<reedy|webirc> |
mw1019 returned [127]: bash: sync-common: command not found |
[production] |
| 19:10 |
<reedy> |
Started scap: testwiki to 1.24wmf18 |
[production] |
| 18:28 |
<manybubbles> |
*victim* |
[production] |
| 18:28 |
<manybubbles> |
trying to recover from weird Elasticsearch upgrade failure by redoing the upgrade on one node while also blowing away the data directory during the upgrade. elastic1005, you are my first victem. |
[production] |
| 17:29 |
<cmjohnson1> |
removing mw1130 from pybal |
[production] |
| 14:54 |
<hashar> |
Jenkins: updated PHP CodeSniffer MediaWiki standard on all slaves. |
[production] |
| 14:36 |
<hashar_> |
Jenkins: updating mediawiki code sniffer repo bf82117..bc4e590 |
[production] |
| 10:02 |
<hashar> |
Jenkins installed plugin [https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin Throttle Concurrent Builds]. |
[production] |
| 03:21 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Aug 21 03:20:47 UTC 2014 (duration 20m 46s) |
[production] |
| 02:36 |
<LocalisationUpdate> |
completed (1.24wmf17) at 2014-08-21 02:34:56+00:00 |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-21 02:19:26+00:00 |
[production] |
| 00:08 |
<MatmaRex> |
(manybubbles contd.) …a single node going down but I expect the cluster to stay "yellow" during the process- no alerts. |
[production] |
| 00:07 |
<manybubbles> |
bd808 needs to plan a logstash upgrade soon - let it be logged |
[production] |
| 00:00 |
<manybubbles> |
unattended rolling restart of Elasticsearch cluster is going just fine - adding the 30 minute sleep between servers and turning down the replication rate makes it pretty boring. |
[production] |
|
2014-08-20
§
|
| 23:07 |
<awight> |
stopping the Thank You job |
[production] |
| 22:51 |
<ori> |
disabled puppet on osmium to debug memory leak |
[production] |
| 21:46 |
<marktraceur> |
Synchronized php-1.24wmf17/extensions/MultimediaViewer/: Add disable-by-default option to MultimediaViewer (duration: 00m 07s) |
[production] |
| 21:10 |
<marktraceur> |
Synchronized wmf-config: Turn off Media Viewer for logged-in users at Commons. (duration: 00m 07s) |
[production] |
| 21:06 |
<marktraceur> |
updated /a/common to {{Gerrit|I226bd1468}}: Add item-redirect to OAuth permissions |
[production] |
| 19:50 |
<hashar> |
Restarting Zuul to prettify build results {{bug|66095}} |
[production] |
| 19:48 |
<awight> |
Synchronized php-1.24wmf17/extensions/CentralNotice: push CentralNotice updates, including new hide cookie format (duration: 00m 05s) |
[production] |
| 19:47 |
<awight> |
Synchronized php-1.24wmf16/extensions/CentralNotice: push CentralNotice updates, including new hide cookie format (duration: 00m 04s) |
[production] |
| 19:46 |
<awight> |
Synchronized php-1.24wmf16/extensions/CentralNotice: push CentralNotice updates, including new hide cookie format (duration: 00m 07s) |
[production] |
| 16:11 |
<manybubbles> |
elastic1001 upgrade went well - upgrading elastic1002 now |
[production] |
| 15:48 |
<hashar> |
dns: Jenkins will now complain whenever you attempt to send tabs in any file of operations/dns.git {{bug|69478}} |
[production] |
| 15:17 |
<manybubbles> |
manually lowered elasticsearch recovery speeds to stem off high load caused by healing the restart of elastic1001 - we were slowing down enough that we were filling the pool counter |
[production] |