|
2014-09-30
§
|
| 21:23 |
<Nemo_bis> |
widespread reproducible 503 errors on wikidata and elsewhere |
[production] |
| 21:15 |
<bd808> |
local l10nupdate users on bastion, mediawiki01 and rsync01 |
[releng] |
| 21:06 |
<bd808> |
Local mwdeploy user on deployment-bastion making things sad |
[releng] |
| 20:55 |
<andrewbogott> |
powering down virt0, just to see what breaks |
[production] |
| 20:48 |
<andrewbogott> |
shutting down pdns on virt0 |
[production] |
| 20:48 |
<andrewbogott> |
shutting down opendj on virt (temporary, a preview of tomorrow) |
[production] |
| 20:36 |
<bd808> |
lots and lots of "file has vanished" errors from rsync. Not sure why |
[releng] |
| 20:35 |
<bd808> |
Initial puppet run with role::beta::rsync_slave applied on rsync02 failed spectacularly in /Stage[main]/Mediawiki::Scap/Exec[fetch_mediawiki] stage |
[releng] |
| 20:26 |
<bd808> |
Converted deployment-rsync02 to use local puppet & salt masters |
[releng] |
| 20:02 |
<bd808> |
Started building deployment-rsync02 to replace deployment-rsync01 |
[releng] |
| 19:59 |
<bd808|LUNCH> |
/srv partition on deployment-rsync01 full again. We need a new rsync server with more space |
[releng] |
| 18:50 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 16s) |
[production] |
| 18:49 |
<reedy> |
Synchronized database lists: (no message) (duration: 00m 15s) |
[production] |
| 18:41 |
<mutante> |
pc1001-1003 - can't generate tmp files for percona monitoring checks -> puppet fail |
[production] |
| 18:24 |
<mutante> |
killing silver from icinga and puppet |
[production] |
| 18:23 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 16s) |
[production] |
| 18:21 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf1 |
[production] |
| 18:05 |
<ori> |
Synchronized wmf-config/HHVMRequestInit.php: (no message) (duration: 00m 07s) |
[production] |
| 18:04 |
<K4-713> |
re-enabled all queue consumers |
[production] |
| 17:56 |
<ejegg> |
updated civicrm from e83c999f39e6ae847d9b48e38c8c825fc10d1635 to b6c350f620c8dc1f3410de179c19cbcbdeb62270 |
[production] |
| 17:44 |
<bd808> |
Updated scap to 064425b (Remove restart-nutcracker and restart-twemproxy scripts) |
[releng] |
| 17:19 |
<K4-713> |
disabled qc jobs and TY mail send for pending civi deploy |
[production] |
| 16:08 |
<bd808> |
Occasional memecached-serious errors in beta for something trying to connect to the default memcached port (11211) rather than the nutcracker port (11212). |
[releng] |
| 15:58 |
<bd808> |
scap happy again after fixing rogue group/user on rsync01 \\o/ Not sure why they were created but likely an ldap hiccup during a puppet run |
[releng] |
| 15:56 |
<bd808> |
removed local group/user mwdeploy on deployment-rsync01 |
[releng] |
| 15:54 |
<bd808> |
Local mwdeploy (gid=996) shadowing ldap group gid=603(mwdeploy) on deployment-rsync01 |
[releng] |
| 15:49 |
<bd808> |
apt-get dist-upgrade fixed hhvm on deployment-mediawiki03 |
[releng] |
| 15:45 |
<hashar> |
Updating our Jenkins job builder fork 686265a..ee80dbc (no job changed) |
[production] |
| 15:45 |
<hashar> |
Updating our Jenkins job builder fork 686265a..ee80dbc (no job changed) |
[releng] |
| 15:44 |
<bd808> |
scap failing in beta due to "Permission denied (publickey)" talking to deployment-rsync01.eqiad.wmflabs |
[releng] |
| 15:42 |
<bblack> |
rebooting mexia |
[production] |
| 15:39 |
<bd808> |
hhvm not starting after puppet run on deployment-mediawiki03. Investigating. |
[releng] |
| 15:36 |
<bd808> |
enabling puppet and forcing run on deployment-mediawiki03 |
[releng] |
| 15:36 |
<bd808> |
enabling puppet and forcing run on deployment-mediawiki03 |
[releng] |
| 15:34 |
<bd808> |
enabling puppet and forcing run on deployment-mediawiki02 |
[releng] |
| 15:34 |
<bd808> |
enabling puppet and forcing run on deployment-mediawiki02 |
[releng] |
| 15:33 |
<demon> |
Synchronized docroot/bits/favicon/wikipedia.ico: Favicons are my favorite icons, especially when they're only 18% of the size of the original (duration: 00m 04s) |
[production] |
| 15:29 |
<bd808> |
puppet showed no changes on mediawiki01‽ |
[releng] |
| 15:28 |
<bd808> |
enabling puppet and forcing run on deployment-mediawiki01 |
[releng] |
| 15:27 |
<bd808> |
enabling puppet and forcing run on deployment-mediawiki01 |
[releng] |
| 15:16 |
<demon> |
Synchronized php-1.25wmf1/extensions/Wikidata: (no message) (duration: 00m 11s) |
[production] |
| 15:14 |
<demon> |
Synchronized php-1.25wmf1/extensions/VisualEditor: (no message) (duration: 00m 08s) |
[production] |
| 15:13 |
<bd808> |
Fixed logstash by installing http://packages.elasticsearch.org/logstash/1.4/debian/pool/main/l/logstash-contrib/logstash-contrib_1.4.2-1-efd53ef_all.deb |
[releng] |
| 15:12 |
<akosiaris> |
merging https://gerrit.wikimedia.org/r/#/c/163735/1, changing the LDAP master from sanger to ldap-mirror for inbound mail |
[production] |
| 15:12 |
<andrewbogott> |
running sync-common on virt1000 |
[production] |
| 15:12 |
<demon> |
Synchronized visualeditor.dblist: (no message) (duration: 00m 04s) |
[production] |
| 15:12 |
<demon> |
Synchronized visualeditor-default.dblist: (no message) (duration: 00m 04s) |
[production] |
| 15:06 |
<demon> |
Synchronized wmf-config/Wikibase.php: (no message) (duration: 00m 04s) |
[production] |
| 15:06 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s) |
[production] |
| 15:02 |
<bd808> |
Logstash doesn't bundle the prune filter by default any more -- http://logstash.net/docs/1.4.2/filters/prune |
[releng] |