|
2015-12-18
§
|
| 11:40 |
<hashar> |
logstash: reorganized list of dashboards per sections https://logstash.wikimedia.org/#/dashboard/elasticsearch/default |
[production] |
| 10:04 |
<hashar> |
rechecking mediawiki/core REL branches ( REL1_26 https://gerrit.wikimedia.org/r/#/c/247327/ ) ( REL1_25 https://gerrit.wikimedia.org/r/#/c/247337/ ) ( REL1_24 https://gerrit.wikimedia.org/r/#/c/179886/ ) ( https://gerrit.wikimedia.org/r/#/c/143591/ | REL1_23 ) |
[releng] |
| 09:43 |
<akosiaris> |
rebooting planet1001, memory exhaustion, OOM showed up |
[production] |
| 09:20 |
<hashar> |
Killed Zuul entirely, the queues were full / deadlocked. Patches need to be retriggered |
[production] |
| 09:18 |
<hashar> |
killing Zuul |
[releng] |
| 09:16 |
<hashar> |
mass cancelling jobs for changes that got force merged |
[releng] |
| 06:47 |
<gwicke> |
restbase1004: nodetool stop -- COMPACTION to avoid running out of disk space |
[production] |
| 03:07 |
<ori@tin> |
Synchronized php-1.27.0-wmf.9/includes/api/ApiStashEdit.php: ab32f4e740: Make ApiStashEdit use statsd metrics (duration: 00m 49s) |
[production] |
| 02:29 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri Dec 18 02:29:10 UTC 2015 (duration 6m 55s) |
[production] |
| 02:22 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.9) (duration: 08m 45s) |
[production] |
| 01:52 |
<ori> |
re-enabled puppet on rdb* / mc* |
[production] |
| 01:25 |
<ori> |
in preparation for Iaefb2d191e, disabling puppet on mc* and rdb* |
[production] |
| 01:21 |
<krinkle@tin> |
Synchronized docroot and w: (no message) (duration: 00m 32s) |
[production] |
| 00:53 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.9/extensions/Flow: Revert Nuke-Flow integration, doesn't work (duration: 00m 32s) |
[production] |
| 00:42 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.9/extensions/Flow: SWAT: Nuke support for Flow, part 3 (duration: 00m 32s) |
[production] |
| 00:34 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Add completion suggester to BetaFeatures whitelist (duration: 00m 30s) |
[production] |
| 00:26 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: grumble grumble touch InitialiseSettings grumble (duration: 00m 30s) |
[production] |
| 00:25 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.9/extensions/Flow: SWAT: Nuke support for Flow, part 2 (duration: 00m 32s) |
[production] |
| 00:23 |
<catrope@tin> |
Synchronized wmf-config/CirrusSearch-production.php: SWAT: enable completion suggester beta on all wikis except wikidata (duration: 00m 30s) |
[production] |
| 00:23 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: enable completion suggester beta on all wikis except wikidata (duration: 00m 29s) |
[production] |
| 00:20 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.9/extensions/Nuke/: SWAT: Nuke support in Flow, part 1 (duration: 00m 30s) |
[production] |
| 00:18 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.9/resources/src/mediawiki.messagePoster/mediawiki.messagePoster.factory.js: SWAT: fix error in messagePoster (duration: 00m 29s) |
[production] |
| 00:17 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.9/extensions/MobileFrontend: SWAT: Schema:MobileWebSectionUsage: always log the isTestA field (duration: 00m 31s) |
[production] |
| 00:08 |
<catrope@tin> |
Synchronized wmf-config/CommonSettings.php: SWAT: cleanup (duration: 00m 30s) |
[production] |
| 00:00 |
<ori> |
restarted mathoid on sca1001 |
[production] |
|
2015-12-17
§
|
| 23:42 |
<mobrovac> |
mathoid deploying 8d2295 |
[production] |
| 23:07 |
<legoktm> |
sudo salt --show-timeout '*slave*' cmd.run 'rm -fR /mnt/home/jenkins-deploy/tmpfs/jenkins-?/*' |
[releng] |
| 23:05 |
<legoktm> |
marked integration-slave-trusty-1012 as offline due to tmpfs issues |
[releng] |
| 22:58 |
<legoktm> |
marked integration-slave-trusty-1014 as offline due to tmpfs issues |
[releng] |
| 22:47 |
<hashar> |
ssh to tin is back https://gerrit.wikimedia.org/r/#/c/259876/ |
[production] |
| 22:39 |
<hashar> |
Only tin lost SSH user keys apparently due to https://gerrit.wikimedia.org/r/#/c/253465/ overriding the admin::groups to simply "eventlogging-admins" |
[production] |
| 22:34 |
<hashar> |
Ssh User keys are gone on deployment servers ( tin / mira ) |
[production] |
| 22:19 |
<hashar> |
Nodepool: all nodes are on ci-jessie-wikimedia-1450388384 |
[releng] |
| 22:04 |
<hashar> |
Nodepool: openstack image delete ci-jessie-wikimedia_old_20151210 |
[releng] |
| 21:49 |
<hashar> |
milestone, Nodepool has spawned 14500 instances so far |
[releng] |
| 21:48 |
<hashar> |
Image ci-jessie-wikimedia-1450388384 in wmflabs-eqiad is ready |
[releng] |
| 21:39 |
<hashar> |
refreshing nodepool snapshot to get the latest zuul right now ( https://wikitech.wikimedia.org/wiki/Nodepool#Manually_generate_a_new_snapshot ) |
[releng] |
| 21:33 |
<hashar> |
gallium: upgrading Zuul from 2.1.0-60-g1cc37f7-wmf2precise1 .. 2.1.0-60-g1cc37f7-wmf4precise1 . Should be noop, only change zuul-cloner which is not used there |
[production] |
| 21:29 |
<mutante> |
add zuul_2.1.0-60-g1cc37f7-wmf4jessie1 to jessie-wikimedia repo |
[production] |
| 21:28 |
<mutante> |
add zuul_2.1.0-60-g1cc37f7-wmf4trusty1 to trusty-wikimedia repo |
[production] |
| 21:22 |
<mutante> |
add zuul_2.1.0-60-g1cc37f7-wmf4precise1 to precise-wikimedia APT |
[production] |
| 19:42 |
<mobrovac> |
mathoid deploying a2187a6 |
[production] |
| 19:07 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.9 |
[production] |
| 19:05 |
<godog> |
disable puppet on graphite2001, brief testing cluster aggregations |
[production] |
| 19:00 |
<thcipriani> |
starting update of all wikis to 1.27.0-wmf.9 |
[production] |
| 18:11 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool x1-slave (db1031), increase db1041 load to 100% (duration: 00m 30s) |
[production] |
| 18:06 |
<gwicke> |
running `nodetool cleanup` on restbase1001 and restbase1005 |
[production] |
| 18:04 |
<robh> |
calcium is supposed to be down, reclaiming to spares, ignore any irc alerts (its in maint mode in icinga) |
[production] |
| 17:55 |
<gwicke> |
running `nodetool cleanup` on restbase1002 |
[production] |
| 17:18 |
<jynus> |
setting mysql db1031 as db2009's master |
[production] |