|
2014-08-23
§
|
| 11:33 |
<mark> |
Manually removed IPv6 addresses from fenari |
[production] |
| 11:23 |
<mark> |
Deactivated IPv6 router-advertisement on cr2-pmtpa |
[production] |
| 11:21 |
<mark> |
Manually removed IPv6 address from mchenry |
[production] |
| 10:36 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1004. pool db1053. (duration: 00m 07s) |
[production] |
| 03:07 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Aug 23 03:06:13 UTC 2014 (duration 6m 12s) |
[production] |
| 02:23 |
<LocalisationUpdate> |
completed (1.24wmf18) at 2014-08-23 02:21:59+00:00 |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.24wmf17) at 2014-08-23 02:17:28+00:00 |
[production] |
| 01:32 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1056 (duration: 00m 06s) |
[production] |
| 00:29 |
<ori> |
disabled puppet on osmium again to debug a leak; please don't re-enable |
[production] |
|
2014-08-22
§
|
| 18:10 |
<ori> |
updated /a/common to {{Gerrit|I338d72a47}}: Do not define MEDIAWIKI before loading WebStart.php |
[production] |
| 17:43 |
<ottomata> |
moving sqstat udp2log filter from analytics1003 to analytics1026, reqstats might blip for a sec... |
[production] |
| 17:41 |
<ori> |
nuking /srv/deployment/rcstream on rcs1002 to verify trebuchet package provider reprovisions it |
[production] |
| 15:54 |
<springle> |
xtrabackup db1056 to db1053 |
[production] |
| 15:53 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1056 while cloning (duration: 00m 07s) |
[production] |
| 15:33 |
<^d> |
elastic1008: fixed /etc/hosts to point to actual IP instead of loopback |
[production] |
| 15:18 |
<springle> |
upgrade & restart db1053, fs check |
[production] |
| 15:08 |
<bd808> |
Still no apache2.log on fluorine or in logstash. Log seems to be available on fenari. |
[production] |
| 14:51 |
<springle> |
switched s1 sanitarium and labsdb replication to db1069:3311 mariadb 10 |
[production] |
| 14:39 |
<mark> |
Removed IPv6 subnets 2620:0:860:1::/64 (squid subnet) and 2620:0:860:3::/64 (sandbox subnet) from cr2-pmtpa configuration |
[production] |
| 04:11 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Aug 22 04:10:47 UTC 2014 (duration 10m 46s) |
[production] |
| 03:19 |
<LocalisationUpdate> |
completed (1.24wmf18) at 2014-08-22 03:18:44+00:00 |
[production] |
| 02:33 |
<LocalisationUpdate> |
completed (1.24wmf17) at 2014-08-22 02:32:11+00:00 |
[production] |
| 00:03 |
<ori> |
Finished scap: SWAT: d3de89777, 7abfe0d5e7, 8ec9853c32b, 476e9e90bd01 (duration: 06m 29s) |
[production] |
|
2014-08-21
§
|
| 23:57 |
<ori> |
Started scap: SWAT: d3de89777, 7abfe0d5e7, 8ec9853c32b, 476e9e90bd01 |
[production] |
| 21:58 |
<ori> |
Synchronized php-1.24wmf17/resources/src/mediawiki/mediawiki.js: I8d27442d1: Workaround for bug introduced by Icf6ede09b (duration: 00m 03s) |
[production] |
| 21:57 |
<manybubbles> |
performing elasticsearch upgrade on elastic1015 |
[production] |
| 21:02 |
<ori> |
Synchronized php-1.24wmf17/resources/src/mediawiki/mediawiki.util.js: Touch resources/src/mediawiki/mediawiki.util.js (duration: 00m 06s) |
[production] |
| 20:44 |
<godog> |
rolling restart of swift-proxy on ms-fe1* |
[production] |
| 20:11 |
<godog> |
restarted swift-proxy on ms-fe1001 |
[production] |
| 19:55 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 13s) |
[production] |
| 19:49 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf18 |
[production] |
| 19:46 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf17 |
[production] |
| 19:44 |
<reedy> |
Finished scap: testwiki to 1.24wmf18 (duration: 34m 01s) |
[production] |
| 19:31 |
<mutante> |
disabled mw1178 in pybal |
[production] |
| 19:27 |
<godog> |
restarted memcached on ms-fe1004 |
[production] |
| 19:23 |
<reedy|webirc> |
mw1178 returned [255]: ssh: connect to host mw1178 port 22: Connection timed out |
[production] |
| 19:23 |
<reedy|webirc> |
mw1019 returned [127]: bash: sync-common: command not found |
[production] |
| 19:10 |
<reedy> |
Started scap: testwiki to 1.24wmf18 |
[production] |
| 18:28 |
<manybubbles> |
*victim* |
[production] |
| 18:28 |
<manybubbles> |
trying to recover from weird Elasticsearch upgrade failure by redoing the upgrade on one node while also blowing away the data directory during the upgrade. elastic1005, you are my first victem. |
[production] |
| 17:29 |
<cmjohnson1> |
removing mw1130 from pybal |
[production] |
| 14:54 |
<hashar> |
Jenkins: updated PHP CodeSniffer MediaWiki standard on all slaves. |
[production] |
| 14:36 |
<hashar_> |
Jenkins: updating mediawiki code sniffer repo bf82117..bc4e590 |
[production] |
| 10:02 |
<hashar> |
Jenkins installed plugin [https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin Throttle Concurrent Builds]. |
[production] |
| 03:21 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Aug 21 03:20:47 UTC 2014 (duration 20m 46s) |
[production] |
| 02:36 |
<LocalisationUpdate> |
completed (1.24wmf17) at 2014-08-21 02:34:56+00:00 |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-21 02:19:26+00:00 |
[production] |
| 00:08 |
<MatmaRex> |
(manybubbles contd.) …a single node going down but I expect the cluster to stay "yellow" during the process- no alerts. |
[production] |
| 00:07 |
<manybubbles> |
bd808 needs to plan a logstash upgrade soon - let it be logged |
[production] |
| 00:00 |
<manybubbles> |
unattended rolling restart of Elasticsearch cluster is going just fine - adding the 30 minute sleep between servers and turning down the replication rate makes it pretty boring. |
[production] |