|
2014-12-18
§
|
| 14:01 |
<qchris> |
EventLogging: deployed 937d804 & restarted EventLogging |
[production] |
| 13:54 |
<akosiaris> |
purged unpuppetized rrdcached from hafnium. It was segfaulting when started via the init script, which led to the package being unconfigured which led to dpkg alerts on icinga |
[production] |
| 13:42 |
<_joe|lunch> |
restarting hhvm on a few servers |
[production] |
| 13:11 |
<_joe_> |
restarted hhvm on mw1242, stuck in getrusage() |
[production] |
| 13:03 |
<_joe_> |
restarted hhvm on mw1191, load at 200 |
[production] |
| 13:00 |
<paravoid> |
salt-cleaning up /etc/sudoers.d/50_* (old naming scheme) |
[production] |
| 12:04 |
<godog> |
upload carbon-c-relay 0.36+git20141218-1 to trusty-wikimedia |
[production] |
| 09:13 |
<hashar> |
enabled MediaWiki core 'structure' PHPUnit tests for all extensions. Will require folks to fix their incorrect AutoLoader and RessourceLoader entries. {{gerrit|180496}} {{bug|T78798}} |
[production] |
| 06:23 |
<_joe|justawake> |
restarted the puppetmaster on palladium |
[production] |
| 04:35 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1055 (duration: 04m 19s) |
[production] |
| 01:42 |
<ori> |
Synchronized php-1.25wmf12/resources/src/mediawiki.action/mediawiki.action.edit.stash.js: Ibb29a825c: mediawiki.action.edit.stash: set timeout to 4 seconds (duration: 00m 05s) |
[production] |
| 01:31 |
<awight> |
update crm from f1e558592ee98ff8fc84d19ff2c0435619e11242 to ae7b2381667dd65d68812c58f61e3ea66fa9fa6f |
[production] |
| 01:25 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1072 (duration: 00m 08s) |
[production] |
| 01:22 |
<springle> |
mw1191 restarted hhvm, apparently stuck in futex |
[production] |
| 00:38 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1066, warm up (duration: 00m 05s) |
[production] |
| 00:24 |
<hashar> |
Restarting Jenkins to remove a deadlock on deployment-bastion slave |
[production] |
| 00:19 |
<maxsem> |
Synchronized php-1.25wmf13/extensions/Flow/: https://gerrit.wikimedia.org/r/#/c/180677/ (duration: 00m 08s) |
[production] |
| 00:01 |
<maxsem> |
Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/180376/ - no-op for prod (duration: 00m 06s) |
[production] |
|
2014-12-17
§
|
| 23:34 |
<hashar> |
Restarted Jenkins and Zuul again to have a clean start while I am crashing to bed. |
[production] |
| 23:22 |
<demon> |
Synchronized wmf-config/StartProfiler.php: xhprof on all hhvm hosts in eqiad (duration: 00m 05s) |
[production] |
| 22:46 |
<hashar> |
restarting Jenkins |
[production] |
| 21:45 |
<hashar> |
killing Jenkins |
[production] |
| 21:41 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf13 |
[production] |
| 21:40 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf12 |
[production] |
| 21:38 |
<reedy> |
Finished scap: testwiki to 1.25wmf13 and build l10n cache (duration: 12m 26s) |
[production] |
| 21:25 |
<reedy> |
Started scap: testwiki to 1.25wmf13 and build l10n cache |
[production] |
| 21:24 |
<Reedy> |
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ for mw1152 |
[production] |
| 20:36 |
<hashar> |
Jenkins/Zuul had some deadlock. Disconnected/reconnected slaves but that did not fix it. Finally had to disconnect/reconnect thegearman client in Jenkins and it is processing again. |
[production] |
| 20:36 |
<reedy> |
Started scap: testwiki to 1.25wmf13 and build l10n cache |
[production] |
| 20:12 |
<hashar> |
Jenkins some slaves are no more properly registered. Unpooling / Repooling them |
[production] |
| 16:42 |
<demon> |
Synchronized php-1.25wmf12/extensions/TextExtracts/: (no message) (duration: 00m 05s) |
[production] |
| 16:27 |
<demon> |
Synchronized wmf-config/CirrusSearch-labs.php: for completeness (duration: 00m 05s) |
[production] |
| 16:20 |
<^d> |
mw1190: manually ran sync-common since it was yelling about my key earlier |
[production] |
| 16:14 |
<demon> |
Synchronized php-1.25wmf12/includes/specials/SpecialSearch.php: (no message) (duration: 00m 06s) |
[production] |
| 16:10 |
<demon> |
Synchronized php-1.25wmf12/extensions/Wikidata/: (no message) (duration: 00m 12s) |
[production] |
| 14:52 |
<akosiaris> |
uploaded apertium-nno-nob_1.0.0+svn~57977-1 to apt.wikimedia.org |
[production] |
| 14:43 |
<anomie> |
Merged and fetched [[gerrit:180477]], so undeployed bad extension changes from [[gerrit:180229]] are no longer a danger |
[production] |
| 13:47 |
<akosiaris> |
uploaded apertium-nob_0.1.0+svn~58076-1 and apertium-nno_0.1.0+svn~58076-1 to apt.wikimedia.org |
[production] |
| 11:59 |
<_joe_> |
removing some core dumps from appservers, so that we don't run out of space by tomorrow |
[production] |
| 11:52 |
<Nemo_bis> |
Don't sync extensions, undeployed unintentional reverts https://wikitech.wikimedia.org/?diff=138472&oldid=138399 |
[production] |
| 10:54 |
<hashar> |
Jenkins deleting legacy 'mwext*testextension' jobs (now suffixed with '-zend') and restarting Jenkins. |
[production] |
| 10:40 |
<springle> |
Synchronized wmf-config/db-eqiad.php: repool db1064, warm up (duration: 00m 05s) |
[production] |
| 10:30 |
<andrewbogott> |
virt1010 and 1011 are up but with puppet and nova-compute disabled pending firewall issues |
[production] |
| 10:29 |
<hashar> |
mw1152 is a jobrunner being rebuild |
[production] |
| 10:27 |
<hashar> |
mw1152 has a wrong host key in /etc/ssh/ssh_known_hosts:2480 causing scap to spurts a remote identification error. |
[production] |
| 10:26 |
<hashar> |
Synchronized wmf-config/throttle.php: {{gerrit|180429}} - Throttle rule for University of Haifa event (duration: 00m 06s) |
[production] |
| 10:25 |
<hashar> |
Synchronized wmf-config/throttle.php: {{gerrit|180429}} - Throttle rule for University of Haifa event (duration: 00m 06s) |
[production] |
| 09:57 |
<_joe_> |
jobrunner started on mw1152 |
[production] |
| 08:43 |
<_joe_> |
depooling mw1152, reimaging as an HAT jobrunner |
[production] |
| 07:52 |
<godog> |
increase minimum raid reconstruction speed on virt1005 and virt1009 |
[production] |