|
2015-08-28
§
|
| 20:15 |
<bd808> |
restored 3 cherry picks that were lost when rebuilding the ops/puppet git repo |
[releng] |
| 20:07 |
<bd808> |
deployment-puppetmaster has only one cherry-pick; looks like maybe dcausse dropped the prior stack when working on Icc95ac8 |
[releng] |
| 18:17 |
<bd808> |
Cleaned up some puppet groups for deployment-prep that no longer exist in ops/puppet |
[releng] |
| 18:03 |
<bd808> |
Building deployment-tmh01.deployment-prep.eqiad.wmflabs to replace deployment-videoscaler01 |
[releng] |
| 18:01 |
<bd808> |
Nope, I deleted deployment-videoscaler01 |
[releng] |
| 18:01 |
<bd808> |
Deleted deployment-urldownloader.deployment-prep.eqiad.wmflabs |
[releng] |
| 16:53 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/234569 |
[releng] |
| 11:39 |
<hashar> |
gallium: rm -fR /srv/org/wikimedia/integration/cover/mediawiki-core/master/php2 . This way https://integration.wikimedia.org/cover/mediawiki-core/ redirects to the coverage report (thanks Krinkle) |
[releng] |
| 11:37 |
<hashar> |
deleting https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-2 (same) |
[releng] |
| 10:43 |
<hashar> |
pooling back integration-slave-trusty-1016 Was once depooled for debugging purposes and repealed ( https://phabricator.wikimedia.org/T110054 ) but apparently Jenkins restart did not pool it back again :/ |
[releng] |
| 00:56 |
<thcipriani> |
sudo keyholder arm on deployment-bastion fixed beta-scap-eqiad |
[releng] |
|
2015-08-26
§
|
| 23:39 |
<bd808> |
Updated scap to a7ec319 (Use configured bin_dir to find refreshCdbJsonFiles) |
[releng] |
| 23:32 |
<Krenair> |
Re-armed keyholder on deployment-bastion |
[releng] |
| 21:51 |
<matt_flaschen> |
To fix https://gerrit.wikimedia.org/r/#/c/233952/1 on Beta, manually ran: while read line; do echo "Starting $line\n"; echo 'ALTER TABLE flow_wiki_ref DROP COLUMN ref_src_wiki;' | sql --write "$line"; echo "Finished $line\n"; done < /srv/mediawiki/all-labs.dblist |
[releng] |
| 16:39 |
<bd808> |
marked https://integration.wikimedia.org/ci/computer/integration-slave-precise-1014/ offline for git clone problems |
[releng] |
| 16:18 |
<marxarelli> |
deleted udp2log.log and restarted service. so far nothing out of `tail -fn0 udp2log.log` |
[releng] |
| 16:16 |
<marxarelli> |
stopping udp2log on deployment-flourine |
[releng] |
| 16:14 |
<marxarelli> |
udp2log is mostly "egrep: writing output: Broken pipe" |
[releng] |
| 16:10 |
<marxarelli> |
disk space at 97% on deployment-flourine, mainly due to 15G /var/log/udp2log/udp2log.log |
[releng] |
| 16:01 |
<bd808> |
sudo rm -rf integration-slave-precise-1014:/mnt/jenkins-workspace/workspace/mediawiki-core-phplint/.git |
[releng] |
| 09:57 |
<hashar> |
Bumping our JJB mirror a3aef64..f01628c Required for the Android Emulator plugin support ( https://phabricator.wikimedia.org/T110307 ) |
[releng] |
| 07:39 |
<hashar_> |
puppet is back in action on beta cluster |
[releng] |
| 07:38 |
<hashar_> |
enabling puppet agent on deployment-puppetmaster. It is disable with no reason given |
[releng] |
| 07:24 |
<hashar_> |
resetted beta cluster puppet master to origin/production . We have lost any cherry pick that might have existed |
[releng] |
| 07:16 |
<hashar_> |
started puppetmaster on deployment-puppetmaster |
[releng] |
| 07:11 |
<hashar> |
puppet fails on most beta cluster instances :-( |
[releng] |
|
2015-08-25
§
|
| 23:48 |
<thcipriani> |
stopping puppetmaster and disabling puppet runs on deployment-puppetmaster until we get a change to diagnose/rebuild (tomorrow) |
[releng] |
| 23:47 |
<thcipriani> |
deployment-puppetmaster showing signs of a corrupt disk "error: object file .git/objects/cc/026ba0cdc872490ef6a616b2bac4bb829639cd is empty" shutting it off for now. |
[releng] |
| 23:43 |
<thcipriani> |
reboot deployment-puppetmaster unreachable from other vms (labvirt1007 thing, probably) |
[releng] |
| 15:10 |
<hashar> |
unpooling and deleting integration-slave-trusty-1014 integration-slave-trusty-1017 and integration-slave-precise-1014 . They are most probably corrupted ( https://phabricator.wikimedia.org/T110052#1571184 ) |
[releng] |
| 15:04 |
<hashar> |
soft rebooting integration-slave-trusty-1014 (ssh dead) |
[releng] |
| 15:02 |
<hashar> |
tashing workspaces on integration-slave-trusty-1014 and integration-slave-trusty-1017 ( https://phabricator.wikimedia.org/T110052#1571184 ) |
[releng] |
| 14:55 |
<hashar> |
dropping all workspaces from integration-slave-precise-1014 . Some .git repos in workspaces might be corrupted |
[releng] |
| 11:47 |
<hashar> |
Upgraded a bunch of Jenkins plugins |
[releng] |
| 10:02 |
<hashar> |
deleted job https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-8-sauce/ Was disabled and no more in our JJB config |
[releng] |
| 08:52 |
<hashar> |
pooling back integration-slave-precise-1014 , integration-slave-trusty-1014 and integration-slave-trusty-1017 . labvirt1007 missed disk space ( https://phabricator.wikimedia.org/T110052 ) |
[releng] |
| 00:58 |
<bd808> |
Updated tin:/srv/deployment/integration/slave-scripts to b287e93 (Revert "Run mw-install-mysql.sh with statement tracing") and synced via trebuchet |
[releng] |