|
2016-08-07
§
|
| 16:22 |
<cwd|afk> |
disabled globalcollect recurring donations |
[production] |
| 16:13 |
<akosiaris> |
restarted apache2 on palladium for full depool to take place |
[production] |
| 12:47 |
<hashar> |
root cause of CI outage is T126552 |
[production] |
| 12:41 |
<hashar> |
CI fully back. Root cause was Jenkins that could not properly create slaves config due to : Could not create rootDir /var/lib/jenkins/config-history/xxxx . Deleting via find /var/lib/jenkins/config-history/nodes/ -path '*_deleted_*' -delete |
[production] |
| 12:12 |
<hashar> |
CI stuck spawning instances via Nodepool apparently due to : Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) --- Though there is only 8 instances ... |
[production] |
| 12:10 |
<hashar> |
CI stuck spawning instances via Nodepool apparently due to : Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) --- Though there is only 8 instances ... |
[production] |
| 12:01 |
<hashar> |
Nodepool: can't spawn instances due to: Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) |
[releng] |
| 12:01 |
<hashar> |
nodepool: deleted servers stuck in "used" states for roughly 4 hours (using: nodepool list , then nodepool delete <id>) |
[releng] |
| 11:54 |
<hashar> |
Nodepool: can't spawn instances due to: Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) |
[releng] |
| 11:54 |
<hashar> |
nodepool: deleted servers stuck in "used" states for roughly 4 hours (using: nodepool list , then nodepool delete <id>) |
[releng] |
| 02:24 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Aug 7 02:24:55 UTC 2016 (duration 5m 51s) |
[production] |
| 02:19 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.13) (duration: 08m 55s) |
[production] |
|
2016-08-06
§
|
| 23:09 |
<yuvipanda> |
cleaned and re-accepted salt-key for labvirt1014, minion back up now |
[production] |
| 22:49 |
<yuvipanda> |
run 'service mariadb start' on labsdb1003, puppet run didn't do anything |
[production] |
| 19:43 |
<andrewbogott> |
rebooting labvirt1012 for a kernel downgrade |
[production] |
| 19:12 |
<andrewbogott> |
rebooting labvirt1013 for kernel downgrade |
[production] |
| 12:31 |
<Amir1> |
restarting uwsgi-ores and celery-ores-worker in deployment-sca03 |
[releng] |
| 12:28 |
<Amir1> |
cherry-picked 303356/1 into the puppetmaster |
[releng] |
| 12:00 |
<Amir1> |
restarting uwsgi-ores and celery-ores-worker in deployment-sca03 |
[releng] |
| 09:36 |
<akosiaris> |
revert back to old backed up bayes database on mendelevium.eqiad.wmnet (OTRS) to get bayes training working again |
[production] |
| 02:26 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Aug 6 02:26:00 UTC 2016 (duration 5m 48s) |
[production] |
| 02:20 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.13) (duration: 08m 46s) |
[production] |
| 01:02 |
<andrewbogott> |
re-imaging labvirt1014 |
[production] |
|
2016-08-05
§
|
| 23:39 |
<tgr@tin> |
Synchronized php-1.28.0-wmf.13/includes/api/ApiLogin.php: temporarily re-add dropped API feature to unbreak Pywikibot T142155 (duration: 00m 48s) |
[production] |
| 22:52 |
<paladox> |
live testing gerrit-test3 changes things and disabling puppet for temp now |
[git] |
| 22:37 |
<andrewbogott> |
rebooting labvirt1014 as part of a protracted iptables/nova-compute investigation |
[production] |
| 21:03 |
<reedy@tin> |
Synchronized wmf-config/CommonSettings.php: Add transitionary timeline config primarily for beta (duration: 00m 57s) |
[production] |
| 19:29 |
<paladox> |
adding tom29739 to lolrrit-wm project |
[tools.lolrrit-wm] |
| 19:28 |
<paladox> |
adding tom29739 to lolrrit-wm project |
[tools] |
| 18:26 |
<andrewbogott> |
restarting rabbitmq-server on labcontrol1001 |
[production] |
| 17:54 |
<bd808> |
Cherry-picked https://gerrit.wikimedia.org/r/#/c/299825/3 for testing |
[releng] |
| 17:50 |
<bd808> |
Removed stale cherry-picks for https://gerrit.wikimedia.org/r/#/c/302303/ and https://gerrit.wikimedia.org/r/#/c/300458/ that were blocking git rebase |
[releng] |
| 17:44 |
<halfak> |
repooled ores-web-04 in hiera and ran puppet on ores-lb-02 |
[ores] |
| 17:27 |
<ejegg> |
rolled back SmashPig to 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3 |
[production] |
| 17:25 |
<ejegg> |
updated SmashPig from 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3 to 2e8a2f4c92840bd999a8742211e0a65d484fde00 |
[production] |
| 16:15 |
<halfak> |
creates ores-web-04 debian-8.5-jessie |
[ores] |
| 16:11 |
<halfak> |
terminated ores-web-04 (re-creating instance with medium image size) |
[ores] |
| 16:03 |
<akosiaris> |
T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-spa-arg_0.4.0~r64399-1+wmf1 |
[production] |
| 16:02 |
<akosiaris> |
T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-arg-cat_0.1.0~r64925-1+wmf1 |
[production] |
| 15:39 |
<joal> |
Restart oozie jobs for druid loading from production refinery instead of joal |
[analytics] |
| 15:16 |
<akosiaris> |
T135176 pool wtp1019-wtp1024 |
[production] |
| 15:13 |
<akosiaris@palladium> |
conftool action : set/pooled=yes; selector: wtp1024.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 15:13 |
<akosiaris@palladium> |
conftool action : set/pooled=yes; selector: wtp1023.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 15:13 |
<akosiaris@palladium> |
conftool action : set/pooled=yes; selector: wtp1022.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 15:13 |
<akosiaris@palladium> |
conftool action : set/pooled=yes; selector: wtp1021.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 15:13 |
<akosiaris@palladium> |
conftool action : set/pooled=yes; selector: wtp1020.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 15:13 |
<akosiaris@palladium> |
conftool action : set/pooled=yes; selector: wtp1019.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid']) |
[production] |
| 14:31 |
<joal> |
Retrying deploying refinery from scap |
[analytics] |
| 13:56 |
<akosiaris> |
strontium has issues, see https://phabricator.wikimedia.org/T142187 |
[production] |
| 13:53 |
<moritzm> |
uploaded gerrit 2.12.2-wmf2 for jessie-wikipedia to apt.wikimedia.org |
[production] |