|
2016-02-27
§
|
| 03:36 |
<bd808> |
Backfilled missing missing https://wmflabs.org/sal/production data (2016-02-21T14:39 to 2016-02-24T16:53) |
[production] |
| 03:24 |
<jzerebecki> |
works again, but lost queued jobs |
[releng] |
| 03:20 |
<jzerebecki> |
that made it worse, restarting zuul |
[releng] |
| 03:18 |
<jzerebecki> |
tryping reload |
[releng] |
| 03:16 |
<jzerebecki> |
no luck. different problem. |
[releng] |
| 03:14 |
<urandom> |
bootstrap of restbase1008-a.eqiad.wmnet complete; begining `nodetool cleanup' of 1003, 1004-a, and 1008-b : T95253 |
[production] |
| 03:12 |
<jzerebecki> |
trying https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Gearman_deadlock |
[releng] |
| 02:31 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Feb 27 02:31:10 UTC 2016 (duration 7m 44s) |
[production] |
| 02:23 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.14) (duration: 10m 24s) |
[production] |
| 01:50 |
<twentyafterfour> |
updated phabricator/tools to https://phabricator.wikimedia.org/rPHTO1433a20886a4168d41b81b5ecf5e29687678f2e2 (applying fix for T127524 ) |
[production] |
| 00:52 |
<aaron@tin> |
Synchronized php-1.27.0-wmf.14/includes/filebackend/FileBackendMultiWrite.php: 7ca624b8b93 (duration: 00m 47s) |
[production] |
| 00:51 |
<jzerebecki> |
salt -v --show-timeout '*slave*' cmd.run "bash -c 'cd /srv/deployment/integration/slave-scripts; git pull'" T128191 |
[releng] |
|
2016-02-26
§
|
| 22:20 |
<mutante> |
powercycle mw1130 |
[production] |
| 20:42 |
<demon@tin> |
Synchronized wmf-config/InitialiseSettings.php: update wmf address in echo footer thingie (duration: 00m 59s) |
[production] |
| 20:29 |
<hashar> |
Restarting Jenkins |
[production] |
| 20:04 |
<urandom> |
issuing test repair on cerium (restbase staging), keyspace : T108611 |
[production] |
| 19:08 |
<bd808> |
Upgraded Elasticsearch on tools-elastic-0[123] to 1.7.5 |
[tools] |
| 18:41 |
<ori@tin> |
Synchronized php-1.27.0-wmf.14/includes/user/User.php: I43cde3a48: Prevent duplicate memcached lookups for user record (duration: 01m 02s) |
[production] |
| 16:46 |
<chasemp> |
labstore1001 'mdadm --manage /dev/md126 --add /dev/sdaf' |
[production] |
| 15:51 |
<jynus> |
shutting down mariadb on db2030 to clone from db1009 |
[production] |
| 15:24 |
<urandom> |
forcing puppet run on restbase1009.eqiad.wmnet |
[production] |
| 15:24 |
<urandom> |
re-enabling puppet on restbase1009.eqiad.wmnet |
[production] |
| 15:20 |
<jynus> |
performing backup of m5-master mysql data |
[production] |
| 15:17 |
<urandom> |
blocking CQL native port on restbase1009.eqiad.wmnet : https://phabricator.wikimedia.org/P2677 |
[production] |
| 15:14 |
<urandom> |
disabling puppet on restbase1009.eqiad to preserve local changes during a quick experiment |
[production] |
| 15:14 |
<jzerebecki> |
salt -v --show-timeout '*slave*' cmd.run "bash -c 'cd /srv/deployment/integration/slave-scripts; git pull'" T128191 |
[releng] |
| 15:14 |
<jzerebecki> |
salt -v --show-timeout '*slave*' cmd.run "bash -c 'cd /srv/deployment/integration/slave-scripts; git pull'" |
[releng] |
| 15:03 |
<hashar> |
Switched MediaWiki core npm test to Nodepool instance T119143 |
[production] |
| 14:44 |
<hashar> |
(since it started, dont be that scared!) |
[releng] |
| 14:44 |
<hashar> |
Nodepool has triggered 40 000 instances |
[releng] |
| 13:59 |
<krinkle@tin> |
Synchronized wmf-config/InitialiseSettings.php: T99096: Enable wmgUseWmfstatic on remaining wikis (duration: 00m 50s) |
[production] |
| 13:54 |
<moritzm> |
rebooting lithium for kernel update |
[production] |
| 13:26 |
<godog> |
launch swiftrepl continuous replication for unsharded containers on ms-fe1003 T128096 |
[production] |
| 12:31 |
<elukey> |
added mc1017/mc1018 back to the redis/memcached pools after maintenance |
[production] |
| 11:53 |
<hashar> |
Restarted memcached on deployment-memc02 T128177 |
[releng] |
| 11:53 |
<hashar> |
memcached process on deployment-memc02 seems to have a nice leak of socket usages (from lost) and plainly refuse connections (bunch of CLOSE_WAIT) T128177 |
[releng] |
| 11:53 |
<hashar> |
memcached process on deployment-memc02 seems to have a nice leak of socket usages (from lost) and plainly refuse connections (bunch of CLOSE_WAIT) |
[releng] |
| 11:42 |
<godog> |
run swiftrepl eqiad -> codfw for unsharded containers |
[production] |
| 11:40 |
<hashar> |
deployment-memc04 find /etc/apt -name '*proxy' -delete (prevented apt-get update) |
[releng] |
| 11:26 |
<hashar> |
beta: salt -v '*' cmd.run 'apt-get -y install ruby-msgpack' . I am tired of seeing puppet debug messages: "Debug: Failed to load library 'msgpack' for feature 'msgpack'" |
[releng] |
| 11:24 |
<hashar> |
puppet keep restarting nutcracker apparently T128177 |
[releng] |
| 11:20 |
<hashar> |
Memcached error for key "enwiki:flow_workflow%3Av2%3Apk:63dc3cf6a7184c32477496d63c173f9c:4.8" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY |
[releng] |
| 11:01 |
<elukey> |
removed mc1018/1017 from the redis memcached pools for maintenance |
[production] |
| 09:46 |
<elukey> |
mc1016.eqiad re-added to the memcached/redis pools after maintenance |
[production] |
| 08:12 |
<elukey> |
removed mc1016.eqiad from the redis/memcached pools for maintenance |
[production] |
| 08:01 |
<moritzm> |
blacklisting aufs kernel module |
[production] |
| 02:32 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Feb 26 02:32:19 UTC 2016 (duration 7m 42s) |
[production] |
| 02:24 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.14) (duration: 10m 34s) |
[production] |
| 01:53 |
<bd808> |
Setup initial wiki farm on am-01.authmanager.eqiad.wmflabs (T125320) |
[authmanager] |
| 01:06 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Lower survey rate again (duration: 01m 05s) |
[production] |