|
2016-02-02
§
|
| 12:15 |
<_joe_> |
stopped puppet on mira, added a big warning in the motd |
[production] |
| 12:15 |
<_joe_> |
stopped rsync, puppet, l10nupdate cronjob on tin |
[production] |
| 12:06 |
<_joe_> |
stopped rsync on tin to avoid problems |
[production] |
| 11:38 |
<moritzm> |
rolling reboot of aqs* (for kernel update) |
[production] |
| 11:24 |
<hashar_> |
Restarting Zuul. Stuck in a dependency loop :( |
[production] |
| 11:12 |
<jynus> |
restarting and reconfiguring mysql at db1063 |
[production] |
| 10:51 |
<_joe_> |
stopped jobrunner on mw1161 after failed sync-common |
[production] |
| 10:44 |
<jynus@mira> |
Synchronized wmf-config/db-eqiad.php: Depool db1063, repool db1036 (duration: 00m 21s) |
[production] |
| 10:00 |
<jynus> |
reconfigure and upgrade db1036 |
[production] |
| 09:51 |
<jynus@mira> |
Synchronized wmf-config/db-eqiad.php: Testing scap-reduce db1018 weight (duration: 00m 21s) |
[production] |
| 09:42 |
<jynus@mira> |
Synchronized wmf-config/db-eqiad.php: Depool db1036, repool db1021 (duration: 00m 22s) |
[production] |
| 09:38 |
<hashar> |
Jenkins is fully up and operational |
[releng] |
| 09:37 |
<hashar> |
Jenkins is fully up and operational |
[production] |
| 09:36 |
<jynus> |
armed keyholder on tin |
[production] |
| 09:33 |
<dcausse> |
elastic (codfw and eqiad): unfreezing indices |
[production] |
| 09:33 |
<moritzm> |
restarting gerrit on ytterbium for java security update |
[production] |
| 09:33 |
<_joe_> |
re-syncing tin homes |
[production] |
| 09:33 |
<hashar> |
restarting Jenkins |
[releng] |
| 09:32 |
<hashar> |
gallium: apt-get upgrade | Restarting Jenkins |
[production] |
| 09:12 |
<jynus@mira> |
Synchronized wmf-config/db-eqiad.php: Depool db1036, repool db1021 (duration: 00m 21s) |
[production] |
| 09:08 |
<dcausse> |
elastic (codfw and eqiad): freezing indices to stop titlesuggest maint scripts |
[production] |
| 09:03 |
<godog> |
repool restbase1007 via confctl |
[production] |
| 08:47 |
<hashar> |
pooling back integration-slave-precise1011 , puppet run got fixed ( https://phabricator.wikimedia.org/T125474 ) |
[releng] |
| 08:13 |
<jynus> |
restarting and upgrading db1021 |
[production] |
| 08:02 |
<jynus@mira> |
Synchronized wmf-config/db-eqiad.php: Pool db1018; Depool db1021 (duration: 00m 20s) |
[production] |
| 07:46 |
<jynus> |
https://phabricator.wikimedia.org/rOMWC2ea9167221d11eb1880e4d26eae64a85cb9b2697 and https://phabricator.wikimedia.org/rOMWCa55d2bf8cd3a2853fac35d5b8239b8e8c2fe6a0f merged but not deployed |
[production] |
| 06:58 |
<_joe_> |
reimaging tin.eqiad.wmnet |
[production] |
| 03:48 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/267828 |
[releng] |
| 03:29 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/266941 |
[releng] |
| 01:30 |
<ebernhardson@mira> |
Finished scap: Add Cookie statement link to footer of all WMF wikis per legal (duration: 19m 42s) |
[production] |
| 01:10 |
<ebernhardson@mira> |
Started scap: Add Cookie statement link to footer of all WMF wikis per legal |
[production] |
| 01:07 |
<ebernhardson@mira> |
scap failed: CalledProcessError Command '/srv/deployment/scap/scap/bin/refreshCdbJsonFiles --directory="/srv/mediawiki-staging/php-1.27.0-wmf.10/cache/l10n" --threads=10 ' returned non-zero exit status 255 (duration: 03m 31s) |
[production] |
| 01:03 |
<ebernhardson@mira> |
Started scap: Add Cookie statement link to footer of all WMF wikis per legal |
[production] |
| 00:42 |
<legoktm> |
due to T125474 |
[releng] |
| 00:42 |
<legoktm> |
marked integration-slave-precise-1011 as offline |
[releng] |
| 00:39 |
<legoktm> |
precise-1011 slave hasn't had a puppet run in 6 days |
[releng] |
| 00:31 |
<ebernhardson@mira> |
scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="cawikibooks" --outdir="/tmp/scap_l10n_1684485672" --threads=10 --quiet' returned non-zero exit status 255 (duration: 02m 35s) |
[production] |
| 00:30 |
<mobrovac> |
restbase deploy end of c3bd864 |
[production] |
| 00:29 |
<ebernhardson@mira> |
Started scap: Add Cookie statement link to footer of all WMF wikis per legal |
[production] |
| 00:26 |
<ebernhardson@mira> |
Synchronized wmf-config/logging.php: Revert "monolog: Ensure that context data added by WebProcessor is utf-8 safe" (duration: 01m 27s) |
[production] |
| 00:23 |
<ebernhardson@mira> |
Synchronized wmf-config/CirrusSearch-production.php: Move morelike query load back to eqiad to allow load testing on codfw (duration: 01m 38s) |
[production] |
|
2016-02-01
§
|
| 23:53 |
<bd808> |
Logstash working again; I applied a change to the default mapping template for Elasticsearch that ensures that fields named "timestamp" are indexed as plain strings |
[releng] |
| 23:51 |
<mobrovac> |
restbase deploy start of c3bd864 on canary rb1001 |
[production] |
| 23:46 |
<bd808> |
Elasticsearch index template for beta logstash cluster making crappy guesses about syslog events; dropped 2016-02-01 index; trying to fix default mappings |
[releng] |
| 23:08 |
<bd808> |
HHVM logs causing rejections during document parse when inserting in Elasticsearch from logstash. They contain a "timestamp" field that looks like "Feb 1 22:56:39" which is making the mapper in Elasticsearch sad. |
[releng] |
| 23:02 |
<bd808> |
Elasticsearch on deployment-logstash2 rejecting all documents with 400 status. Investigating |
[releng] |
| 22:50 |
<bd808> |
Copying deployment-logstash2.deployment-prep:/var/log/logstash/logstash.log to /srv for debugging later |
[releng] |
| 22:48 |
<bd808> |
deployment-logstash2.deployment-prep:/var/log/logstash/logstash.log is 11G of fail! |
[releng] |
| 22:46 |
<bd808> |
root partition on deployment-logstash2 full |
[releng] |
| 22:43 |
<bd808> |
No data in logstash since 2016-01-30T06:55:37.838Z; investigating |
[releng] |