|
2016-08-07
§
|
| 23:39 |
<reedy@tin> |
Synchronized wmf-config: last 2 to wfLoadExtension (duration: 00m 59s) |
[production] |
| 23:12 |
<reedy@tin> |
Synchronized wmf-config: Handful more extensions to wfLoadExtension (duration: 00m 49s) |
[production] |
| 22:57 |
<reedy@tin> |
Synchronized wmf-config: RestbaseUpdateJobs to wfLoadExtension (duration: 00m 51s) |
[production] |
| 22:55 |
<reedy@tin> |
Synchronized wmf-config/CommonSettings.php: Image Area to 100MP (duration: 00m 48s) |
[production] |
| 22:44 |
<reedy@tin> |
Synchronized wmf-config/extension-list: 2 more to extension.json (duration: 00m 48s) |
[production] |
| 22:35 |
<reedy@tin> |
Synchronized docroot/noc/conf: Cleanup! (duration: 00m 50s) |
[production] |
| 22:30 |
<reedy@tin> |
Synchronized docroot: trusted-xff symlink updates (duration: 00m 50s) |
[production] |
| 22:28 |
<reedy@tin> |
Synchronized wmf-config/: Swap trusted-xff from cdb to php (duration: 00m 51s) |
[production] |
| 20:49 |
<legoktm@tin> |
Synchronized php-1.28.0-wmf.13/extensions/GlobalBlocking/extension.json: Adding globalblock-exempt grant for OAuth - T142306 (duration: 00m 57s) |
[production] |
| 16:59 |
<paladox> |
disabling puppet on gerrit-test3, testing a change that will allow https to run on labs. |
[git] |
| 16:22 |
<cwd|afk> |
disabled globalcollect recurring donations |
[production] |
| 16:13 |
<akosiaris> |
restarted apache2 on palladium for full depool to take place |
[production] |
| 12:47 |
<hashar> |
root cause of CI outage is T126552 |
[production] |
| 12:41 |
<hashar> |
CI fully back. Root cause was Jenkins that could not properly create slaves config due to : Could not create rootDir /var/lib/jenkins/config-history/xxxx . Deleting via find /var/lib/jenkins/config-history/nodes/ -path '*_deleted_*' -delete |
[production] |
| 12:12 |
<hashar> |
CI stuck spawning instances via Nodepool apparently due to : Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) --- Though there is only 8 instances ... |
[production] |
| 12:10 |
<hashar> |
CI stuck spawning instances via Nodepool apparently due to : Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) --- Though there is only 8 instances ... |
[production] |
| 12:01 |
<hashar> |
Nodepool: can't spawn instances due to: Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) |
[releng] |
| 12:01 |
<hashar> |
nodepool: deleted servers stuck in "used" states for roughly 4 hours (using: nodepool list , then nodepool delete <id>) |
[releng] |
| 11:54 |
<hashar> |
Nodepool: can't spawn instances due to: Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) |
[releng] |
| 11:54 |
<hashar> |
nodepool: deleted servers stuck in "used" states for roughly 4 hours (using: nodepool list , then nodepool delete <id>) |
[releng] |
| 02:24 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Aug 7 02:24:55 UTC 2016 (duration 5m 51s) |
[production] |
| 02:19 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.13) (duration: 08m 55s) |
[production] |
|
2016-08-06
§
|
| 23:09 |
<yuvipanda> |
cleaned and re-accepted salt-key for labvirt1014, minion back up now |
[production] |
| 22:49 |
<yuvipanda> |
run 'service mariadb start' on labsdb1003, puppet run didn't do anything |
[production] |
| 19:43 |
<andrewbogott> |
rebooting labvirt1012 for a kernel downgrade |
[production] |
| 19:12 |
<andrewbogott> |
rebooting labvirt1013 for kernel downgrade |
[production] |
| 12:31 |
<Amir1> |
restarting uwsgi-ores and celery-ores-worker in deployment-sca03 |
[releng] |
| 12:28 |
<Amir1> |
cherry-picked 303356/1 into the puppetmaster |
[releng] |
| 12:00 |
<Amir1> |
restarting uwsgi-ores and celery-ores-worker in deployment-sca03 |
[releng] |
| 09:36 |
<akosiaris> |
revert back to old backed up bayes database on mendelevium.eqiad.wmnet (OTRS) to get bayes training working again |
[production] |
| 02:26 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Aug 6 02:26:00 UTC 2016 (duration 5m 48s) |
[production] |
| 02:20 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.13) (duration: 08m 46s) |
[production] |
| 01:02 |
<andrewbogott> |
re-imaging labvirt1014 |
[production] |
|
2016-08-05
§
|
| 23:39 |
<tgr@tin> |
Synchronized php-1.28.0-wmf.13/includes/api/ApiLogin.php: temporarily re-add dropped API feature to unbreak Pywikibot T142155 (duration: 00m 48s) |
[production] |
| 22:52 |
<paladox> |
live testing gerrit-test3 changes things and disabling puppet for temp now |
[git] |
| 22:37 |
<andrewbogott> |
rebooting labvirt1014 as part of a protracted iptables/nova-compute investigation |
[production] |
| 21:03 |
<reedy@tin> |
Synchronized wmf-config/CommonSettings.php: Add transitionary timeline config primarily for beta (duration: 00m 57s) |
[production] |
| 19:29 |
<paladox> |
adding tom29739 to lolrrit-wm project |
[tools.lolrrit-wm] |
| 19:28 |
<paladox> |
adding tom29739 to lolrrit-wm project |
[tools] |
| 18:26 |
<andrewbogott> |
restarting rabbitmq-server on labcontrol1001 |
[production] |
| 17:54 |
<bd808> |
Cherry-picked https://gerrit.wikimedia.org/r/#/c/299825/3 for testing |
[releng] |
| 17:50 |
<bd808> |
Removed stale cherry-picks for https://gerrit.wikimedia.org/r/#/c/302303/ and https://gerrit.wikimedia.org/r/#/c/300458/ that were blocking git rebase |
[releng] |
| 17:44 |
<halfak> |
repooled ores-web-04 in hiera and ran puppet on ores-lb-02 |
[ores] |
| 17:27 |
<ejegg> |
rolled back SmashPig to 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3 |
[production] |
| 17:25 |
<ejegg> |
updated SmashPig from 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3 to 2e8a2f4c92840bd999a8742211e0a65d484fde00 |
[production] |