|
2014-09-16
§
|
| 12:35 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool s2 db1054, s3 db1027, s4 db1056, s5 db1037 (duration: 00m 10s) |
[production] |
| 12:26 |
<godog> |
reboot ms-be1014, xfs issues |
[production] |
| 12:22 |
<godog> |
temporarily chgrp wikidev /var/log/hhvm/error.log on mw1018 |
[production] |
| 12:21 |
<reedy> |
Synchronized php-1.24wmf20/LocalSettings.php: Fix path to be /srv based (duration: 00m 32s) |
[production] |
| 11:25 |
<reedy> |
Synchronized docroot and w: (no message) (duration: 00m 35s) |
[production] |
| 11:12 |
<reedy> |
Purged l10n cache for 1.24wmf19 |
[production] |
| 11:12 |
<reedy> |
Purged l10n cache for 1.24wmf18 |
[production] |
| 11:10 |
<reedy> |
Purged l10n cache for 1.24wmf15 |
[production] |
| 09:21 |
<_joe_> |
reimaging mw1018 and mw1021 w HAT: removing from pybal, etc. |
[production] |
| 06:29 |
<springle> |
xtrabackup clone db1037 to db2023 |
[production] |
| 05:31 |
<springle> |
xtrabackup clone db1056 to db2019 |
[production] |
| 04:01 |
<jeremyb> |
deployment-mediawiki02: salt was broken with a msgpack exception. mv -v /var/cache/salt{,.old} && service salt-minion restart fixed it. also did salt-call saltutil.sync_all |
[releng] |
| 04:01 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Sep 16 04:01:05 UTC 2014 (duration 1m 4s) |
[production] |
| 04:00 |
<jeremyb> |
deployment-mediawiki03: (/run was 99%) |
[releng] |
| 03:59 |
<jeremyb> |
deployment-mediawiki03: rm -rv /run/hhvm/cache && service hhvm restart |
[releng] |
| 03:11 |
<springle> |
xtrabackup clone db1027 to db2018 |
[production] |
| 03:04 |
<LocalisationUpdate> |
completed (1.24wmf21) at 2014-09-16 03:04:46+00:00 |
[production] |
| 02:53 |
<springle> |
xtrabackup clone db1054 to db2017 |
[production] |
| 02:50 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool s2 db1054, s3 db1027, s4 db1056, s5 db1037 for codfw cloning (duration: 01m 12s) |
[production] |
| 02:46 |
<cscott> |
updated OCG to version 188a3c221d927bd0601ef5e1b0c0f4a9d1cdbd31 |
[releng] |
| 02:39 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1036, depool db1002 (duration: 00m 07s) |
[production] |
| 02:31 |
<LocalisationUpdate> |
completed (1.24wmf20) at 2014-09-16 02:31:16+00:00 |
[production] |
| 00:51 |
<jeremyb> |
deployment-pdf01 removed base::firewall (ldap via wikitech) |
[releng] |
|
2014-09-15
§
|
| 23:32 |
<maxsem> |
Synchronized php-1.24wmf21/resources/: SWAT: https://gerrit.wikimedia.org/r/#/c/160488/1 https://gerrit.wikimedia.org/r/#/c/160543/ (duration: 00m 06s) |
[production] |
| 23:26 |
<bblack> |
restarting lvs1001 for HT disable + kernel upgrade |
[production] |
| 23:19 |
<maxsem> |
Synchronized php-1.24wmf21/extensions/VisualEditor/: SWAT: https://gerrit.wikimedia.org/r/#/c/160554/ (duration: 00m 07s) |
[production] |
| 23:12 |
<bblack> |
restarting lvs1002 for HT disable + kernel upgrade |
[production] |
| 23:07 |
<greg-g> |
Running sample job on integration-slave1006 and warming up npmjs.org cache |
[production] |
| 22:56 |
<Krinkle> |
Running sample job on integration-slave1008 and warming up npmjs.org cache |
[production] |
| 22:53 |
<jeremyb> |
deployment-pdf01: pkill -f grain-ensure |
[releng] |
| 22:49 |
<Krinkle> |
Running sample job on integration-slave1007 and warming up npmjs.org cache |
[production] |
| 22:48 |
<Krinkle> |
Pooling the newly setup Trusty-based Jenkins slaves (integration-slave1006, integration-slave1007 and integration-slave1008) |
[production] |
| 22:42 |
<bblack> |
dropping static routes for 2620:0:861:ed1a::[d,f,10,11] -> lvs1005 from cr[12]-eqiad (only 11 is of any consequence, misc-web-lb, and they're advertised by bgp and this is preventing failover to lvs1002) |
[production] |
| 21:44 |
<andrewbogott> |
migrating deployment-videoscaler01 to virt1002 |
[releng] |
| 21:41 |
<andrewbogott> |
migrating deployment-sentry2 to virt1002 |
[releng] |
| 21:40 |
<cscott> |
*skipped* deploy of OCG, due to deployment-salt issues |
[releng] |
| 21:36 |
<bd808> |
Trying to fix salt with `salt '*' service.restart salt-minion` |
[releng] |
| 21:32 |
<bd808> |
only hosts responding to salt in beta are deployment-mathoid, deployment-pdf01 and deployment-stream |
[releng] |
| 21:29 |
<bd808> |
salt calls failing in beta with errors like "This master address: 'salt' was previously resolvable but now fails to resolve!" |
[releng] |
| 21:28 |
<cscott> |
updated OCG to version 188a3c221d927bd0601ef5e1b0c0f4a9d1cdbd31 |
[production] |
| 21:19 |
<bd808> |
Added Matanya to under_NDA sudoers group (bug 70864) |
[releng] |
| 20:46 |
<subbu> |
deployed Parsoid version b845bff9 |
[production] |
| 20:18 |
<hashar> |
restarted salt-master |
[releng] |
| 19:50 |
<hashar> |
killed on deployment-bastion a bunch of <tt>python /usr/local/sbin/grain-ensure contains ... </tt> and <tt>/usr/bin/python /usr/bin/salt-call --out=json grains.append deployment_target scap</tt> commands |
[releng] |
| 18:57 |
<hashar> |
scap breakage due to ferm is logged as https://bugzilla.wikimedia.org/show_bug.cgi?id=70858 |
[releng] |
| 18:49 |
<ejegg> |
Synchronized php-1.24wmf20/extensions/CentralNotice/: Update CentralNotice to remove jquery.json dependency (duration: 00m 23s) |
[production] |
| 18:48 |
<hashar> |
https://gerrit.wikimedia.org/r/#/c/160485/ tweaked a default ferm configuration file which caused puppet to reload ferm. It ends up having rules that prevent ssh from other host thus breaking rsync \\O/ |
[releng] |
| 18:46 |
<hoo> |
Sync to tmh100[12] failed, according to awight |
[production] |
| 18:44 |
<ejegg> |
Synchronized php-1.24wmf21/extensions/CentralNotice/: Update CentralNotice to remove jquery.json dependency (duration: 00m 09s) |
[production] |
| 18:43 |
<manybubbles> |
performance tests show cirrus should handle jawiki with no problem but if load spirals out of control and I'm not around then revert https://gerrit.wikimedia.org/r/#/c/160465/ |
[production] |