|
2015-10-12
§
|
| 12:10 |
<yurik> |
deployed kartotherian & tilerator to maps-test200{1-4} |
[production] |
| 08:54 |
<hashar> |
zuul-merger process leaked file descriptors and ended up unable to open any more files. Fixed by restarting the service on gallium. https://phabricator.wikimedia.org/T115243 |
[releng] |
| 08:54 |
<hashar> |
zuul-merger process leaked file descriptors and ended up unable to open any more files. Fixed by restarting the service on gallium. https://phabricator.wikimedia.org/T115243 |
[production] |
| 08:44 |
<hashar> |
Zuul CI in trouble. zuul-merger can't not apply patches anymore https://phabricator.wikimedia.org/T115243 |
[releng] |
| 08:44 |
<hashar> |
Zuul CI in trouble. zuul-merger can't not apply patches anymore https://phabricator.wikimedia.org/T115243 |
[production] |
| 02:43 |
<Krenair> |
fixed puppet on deployment-conf03 by running several manual apt-get commands |
[releng] |
| 02:33 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Oct 12 02:33:13 UTC 2015 (duration 33m 12s) |
[production] |
| 02:33 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-12 02:33:13+00:00 |
[production] |
| 02:29 |
<l10nupdate@tin> |
Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 50s) |
[production] |
| 00:38 |
<bd808> |
fixed puppet on deploymnet-restbase01 by running several manual apt-get and dpkg commands; had to downgrade zsh from 5.1.1-1 (unstable) to 5.0.7-5 (stable) |
[releng] |
| 00:27 |
<bd808> |
puppet failing on deployment-restbase01 due to corrupt apt config state |
[releng] |
|
2015-10-09
§
|
| 22:47 |
<yuvipanda> |
kill NFS on tools-puppetmaster-01 with https://wikitech.wikimedia.org/wiki/Hiera:Tools/host/tools-puppetmaster-01 |
[tools] |
| 21:57 |
<greg-g> |
21:51 < ori> !log deployment-prep Accidentally clobbered /etc/init.d/mysql on deployment-db1, causing deployment-prep failures. Restored now |
[releng] |
| 21:55 |
<twentyafterfour> |
deployment-db1 has a running mysqld again, shinken reports recovery. |
[releng] |
| 21:51 |
<ori> |
Accidentally clobbered /etc/init.d/mysql on deployment-db1, causing deployment-prep failures. Restored now. |
[deployment-prep] |
| 21:41 |
<twentyafterfour> |
ori broke mariadb on deployment-db1 :-P |
[releng] |
| 20:28 |
<hashar> |
beta cluster parsoid now runs from /parsoid.git && npm install (was from /deploy.git previously). In case of troubles poke subbu and see revert instructions on https://phabricator.wikimedia.org/T92871 |
[releng] |
| 20:16 |
<hashar> |
Parsoid on beta is broken. Busy installing npm dependencies |
[releng] |
| 20:09 |
<hashar> |
switching Parsoid on beta to install dependencies with npm (instead of /deploy) https://phabricator.wikimedia.org/T92871 for subbu |
[releng] |
| 19:33 |
<ori@tin> |
Synchronized multiversion/MWWikiversions.php: I9d4cbd3d67: Provide a smooth migration path of dblist files to dblists/ (duration: 01m 13s) |
[production] |
| 19:26 |
<ottomata> |
releasing refinery 0.20 |
[analytics] |
| 18:13 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.2/extensions/OpenStackManager/nova/OpenStackNovaProject.php: https://gerrit.wikimedia.org/r/#/c/244707/ (duration: 01m 13s) |
[production] |
| 17:45 |
<mutante> |
added johnflewis |
[icinga] |
| 16:04 |
<godog> |
rolling restart cassandra test cluster T95253 |
[production] |
| 15:35 |
<urandom> |
performing Cassandra cleanup on restbase-test2003.codfw |
[production] |
| 15:19 |
<ottomata> |
moved camus property files out of refinery repository and into puppet. Camus properties now live on an27 at /etc/camus.d, and camus log files are in /var/log/camus |
[analytics] |
| 15:15 |
<urandom> |
bouncing Cassandra on restbase-test2001-a |
[production] |
| 15:06 |
<urandom> |
starting nodetool cleanup on restbase-test2002 |
[production] |
| 14:54 |
<hashar> |
added Geodata as a dependency to the wikibase jobs ( https://gerrit.wikimedia.org/r/#/c/244489/ ) |
[releng] |
| 14:54 |
<joal> |
Cassandra restarted on aqs1003 |
[analytics] |
| 14:37 |
<Coren> |
Beginning rotation of execution nodes to apply fix for T106170 |
[tools] |
| 14:37 |
<jynus> |
more configuration testing (with puppet disabled) and several mysql restarts on db1022 |
[production] |
| 13:53 |
<hashar> |
Restarted Zuul, had a deadlocked job |
[production] |
| 11:50 |
<godog> |
bounce mathoid on sca100[12], stray instance found not running firejail |
[production] |
| 11:22 |
<mobrovac> |
restbase deploying aaee7c31 |
[production] |
| 11:16 |
<godog> |
force-run puppet on restbase after merging https://gerrit.wikimedia.org/r/#/c/244656/ |
[production] |
| 11:00 |
<jynus> |
restarting db1022's mysql (depooled) for configuration testing |
[production] |
| 09:17 |
<jynus> |
deployed visual glitch fix to dbtree |
[production] |
| 09:15 |
<joal> |
Restart cassandra on aqs1002 |
[analytics] |
| 09:11 |
<akosiaris> |
poweroff rhodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing. |
[production] |
| 09:11 |
<akosiaris> |
poweroff sodium, remove salt key, remove puppet storedconfigs in preparation for reinstall reinstall as a VM for temporary puppetmaster testing. |
[production] |
| 08:20 |
<ori> |
Purged graphite[12]001:/var/lib/carbon/whisper/servers/*/TcpConnStatesCollector and graphite[12]001:/var/lib/carbon/whisper/servers/*/network/work; cleaning up after https://gerrit.wikimedia.org/r/#/c/244637/ |
[production] |