|
2014-12-13
§
|
| 17:51 |
<bd808> |
Running chown -R apache:apache on /data/project/upload7 from deployment-mediawiki02 |
[releng] |
| 17:11 |
<bd808> |
Labs DNS seems to be flaking out badly and causing random scap and puppet failures |
[releng] |
| 17:10 |
<bd808> |
Many strange puppet and scap failures in beta that look to be related to DNS failures |
[releng] |
| 16:58 |
<bd808> |
restarted puppetmaster on deployment-salt |
[releng] |
| 16:31 |
<bd808> |
apache user renumbered on deployment-mediawiki03 |
[releng] |
| 16:23 |
<bd808> |
apache and hhvm restarted on beta app servers following apache user renumber |
[releng] |
| 16:09 |
<bd808> |
apache and hhvm stopped on beta app server tier. All requests expected to return 503 from varnish |
[releng] |
| 16:03 |
<bd808> |
Starting work on [[phab:T78076]] to renumber apache users in beta |
[releng] |
| 16:03 |
<bd808> |
Starting work on [[phab:T78076]] to renumber apache users in beta |
[releng] |
| 12:20 |
<andrewbogott> |
graceful'd apache2 on virt1000; puppet master was acting up. |
[production] |
| 09:51 |
<hashar> |
Restarting Jenkins to get rid of some deadlocks that occurred yesterday |
[production] |
| 08:21 |
<YuviPanda|zzz> |
forcing puppet run on all deployment-prep hosts |
[releng] |
| 03:29 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Dec 13 03:29:31 UTC 2014 (duration 29m 30s) |
[production] |
| 02:14 |
<LocalisationUpdate> |
completed (1.25wmf12) at 2014-12-13 02:14:00+00:00 |
[production] |
| 02:14 |
<l10nupdate> |
Synchronized php-1.25wmf12/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
| 02:09 |
<LocalisationUpdate> |
completed (1.25wmf11) at 2014-12-13 02:09:28+00:00 |
[production] |
| 02:09 |
<l10nupdate> |
Synchronized php-1.25wmf11/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
| 02:04 |
<krinkle> |
Synchronized w/robots.php: 54746fdef3402 (duration: 00m 05s) |
[production] |
| 00:58 |
<krinkle> |
Synchronized w/robots.php: 611892c62349d09c9758 (duration: 00m 06s) |
[production] |
|
2014-12-12
§
|
| 22:38 |
<bd808> |
Fixed scap by deleting /srv/mediawiki/~tmp~ on deployment-rsync01 |
[releng] |
| 22:27 |
<hashar> |
Creating 1300 Jenkins jobs to run extensions PHPUnit tests under either HHVM or Zend PHP flavors. |
[releng] |
| 22:06 |
<ori> |
Synchronized wmf-config/StartProfiler.php: (no message) (duration: 00m 06s) |
[production] |
| 19:47 |
<ottomata1> |
initiating kafka preferred-replica-election to bring analytics1021 back in to leadership :/ need to figure this out, or replace this node soon. |
[production] |
| 18:35 |
<bd808> |
Added puppet config to record !log messages in logstash |
[releng] |
| 18:15 |
<YuviPanda> |
ran sudo logrotate -f /etc/logrotate.d/dumpwikidatajson on snapshot1003 forhoo |
[production] |
| 18:13 |
<ori> |
Synchronized wmf-config/StartProfiler.php: (no message) (duration: 00m 08s) |
[production] |
| 17:32 |
<bd808> |
forcing puppet runs on deployment-mediawiki0[12]; hiera settings specific to beta were not applied on the hosts leading to all kinds of problems |
[releng] |
| 17:13 |
<bd808> |
restarted hhvm on deployment-mediawiki0[12] and purged hhbc database |
[releng] |
| 17:00 |
<bd808> |
restarted apache2 on deployment-mediawiki01 |
[releng] |
| 16:59 |
<bd808> |
restarted apache2 on deployment-mediawiki02 |
[releng] |
| 16:42 |
<akosiaris> |
uploaded apertium-sv-da, apertium-en-ca to apt.wikimedia.org |
[production] |
| 15:13 |
<hashar> |
Zuul Reverting Zuul back to wmf-deploy-20141030-4 . I previously reverted it to another change which was wrong. |
[production] |
| 14:59 |
<hashar> |
Zuul status page is no more. https://phabricator.wikimedia.org/T78400 |
[production] |
| 14:50 |
<hashar> |
upgrading python-statsd on Zuul server and restarting service. |
[production] |
| 14:37 |
<godog> |
upload python-statsd 3.0.1-1 to precise-wikimedia |
[production] |
| 14:13 |
<godog> |
upload python-statsd 3.0.1-1 to trusty-wikimedia |
[production] |
| 11:43 |
<YuviPanda> |
force puppet run on all labs hosts via salt |
[production] |
| 09:33 |
<ori> |
restarted mwprof on tungsten |
[production] |
| 09:20 |
<ori> |
Synchronized wmf-config/StartProfiler.php: I63864cc79: xenon log: collate stack samples and fold into single lines (duration: 00m 06s) |
[production] |
| 06:05 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1055 (duration: 00m 05s) |
[production] |
| 03:50 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Dec 12 03:50:10 UTC 2014 (duration 50m 9s) |
[production] |
| 03:28 |
<yurik> |
Synchronized php-1.25wmf12/extensions/ZeroPortal/: updatidng ZeroPortal to master - urgent bugfix - retry (duration: 00m 10s) |
[production] |
| 03:27 |
<yurik> |
Synchronized php-1.25wmf12/extensions/ZeroPortal/: updatidng ZeroPortal to master - urgent bugfix (duration: 00m 05s) |
[production] |
| 03:25 |
<ori> |
Synchronized wmf-config: I1d218c2d6: Log xenon-captured traces via wfDebugLog (duration: 00m 06s) |
[production] |
| 02:55 |
<andrewbogott> |
rebooted mw1041 from mgmt |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.25wmf12) at 2014-12-12 02:20:37+00:00 |
[production] |
| 02:20 |
<l10nupdate> |
Synchronized php-1.25wmf12/cache/l10n: (no message) (duration: 00m 03s) |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.25wmf11) at 2014-12-12 02:15:34+00:00 |
[production] |
| 02:15 |
<l10nupdate> |
Synchronized php-1.25wmf11/cache/l10n: (no message) (duration: 00m 03s) |
[production] |
| 00:41 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/179360 (duration: 00m 06s) |
[production] |