|
2016-07-03
§
|
| 17:09 |
<yuvipanda> |
run qstat -u '*' | grep 'dr ' | awk '{ print $1;}' | xargs -L1 qdel -f to clean out jobs stuck in dr state |
[tools] |
| 16:58 |
<yuvipanda> |
migrate tools-web-static-02 to labvirt1011 to provide more breathing room |
[tools] |
| 16:56 |
<yuvipanda> |
delete temp-test-trusty-package to provide more breathing room on labvirt1010 |
[tools] |
| 16:50 |
<yuvipanda> |
migrating visualeditor-test to labvirt1011, krenair reports it broken |
[visualeditor] |
| 15:16 |
<JeanFred> |
Deployed latest from Git: 76383f9 |
[tools.heritage] |
| 15:02 |
<yuvipanda> |
migrating toolsbeta-valhallasw-puppet-compiler to labvirt1011 to ease pressure on labvirt1010 |
[toolsbeta] |
| 14:59 |
<yuvipanda> |
restart nova-compute process on labvirt1010 |
[production] |
| 14:59 |
<yuvipanda> |
restart nova-compute process on labvirt10101 |
[production] |
| 14:44 |
<JeanFred> |
Deployed latest from Git: 1fdeff6 |
[tools.heritage] |
| 14:09 |
<yuvipanda> |
migrate social-tools2 to labvirt1011 |
[social-tools] |
| 13:49 |
<yuvipanda> |
reboot tools-exec-1219 |
[tools] |
| 13:37 |
<yuvipanda> |
migrating tools-exec-1216 to labvirt1011 |
[tools] |
| 13:10 |
<paladox> |
phabricator Update phab-01 and phab-05 (phab-02) and phab-03 to fix a security bug in phabricator (Did the update last night but forgot to log it) |
[releng] |
| 13:07 |
<yuvipanda> |
delete tools-bastion-01 which was shut down anyway |
[tools] |
| 13:04 |
<yuvipanda> |
attempt to reboot tools-exec-1212 |
[tools] |
| 13:01 |
<JeanFred> |
Deployed latest from Git: 0154a31, 9a4d05b, 4b6c343, 39c5409 (T138633) |
[tools.heritage] |
| 12:04 |
<jzerebecki> |
reloading zuul for 7e6a2e2..13ea50f |
[releng] |
| 09:06 |
<jynus> |
removing old logs from pc2004 |
[production] |
| 07:42 |
<legoktm@tin> |
Synchronized static/images/project-logos/: Put high-res enwiktionary logos in the right place - T139255 (duration: 00m 38s) |
[production] |
| 02:27 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Jul 3 02:27:13 UTC 2016 (duration 5m 38s) |
[production] |
| 02:21 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.8) (duration: 09m 13s) |
[production] |
|
2016-07-01
§
|
| 22:23 |
<krinkle@tin> |
Synchronized php-1.28.0-wmf.8/extensions/WikimediaEvents/extension.json: T128115 (duration: 00m 37s) |
[production] |
| 22:22 |
<krinkle@tin> |
Synchronized php-1.28.0-wmf.8/extensions/WikimediaEvents/modules/: T128115 (duration: 00m 30s) |
[production] |
| 21:04 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: I7a95c0f4: Bump $wgResourceLoaderMaxQueryLength to 5,000 (duration: 00m 32s) |
[production] |
| 20:08 |
<ori@tin> |
Synchronized wmf-config/CommonSettings.php: I6eb0ae67: Bump $wgResourceLoaderMaxQueryLength to 4,000 (duration: 00m 26s) |
[production] |
| 19:17 |
<ori> |
restarted coal on graphite1001 stopped receiving messages from EL 0mq publisher |
[production] |
| 19:16 |
<ori> |
restarted navtiming on hafnium; stopped receiving messages from EL 0mq publisher |
[production] |
| 18:34 |
<mutante> |
mw1259 - powercycling |
[production] |
| 18:32 |
<krinkle@tin> |
Synchronized docroot/default/: (no message) (duration: 00m 31s) |
[production] |
| 18:31 |
<krinkle@tin> |
Synchronized errorpages/: (no message) (duration: 01m 06s) |
[production] |
| 17:47 |
<ebernhardson> |
restart elasticsearch on elastic1017 to attempt to clear up a continuous backlog of relocating shards |
[production] |
| 15:53 |
<godog> |
temporarily run 3x statsdlb instances on graphite1001 to minimise drops - T101141 |
[production] |
| 14:57 |
<dcausse> |
upgraded and restarted elastic on nobelium@eqiad |
[production] |
| 14:21 |
<godog> |
enable another statsdlb instance temporarily on graphite1001 to investigate drops |
[production] |
| 14:15 |
<moritzm> |
rearmed keyholder on mira after reboot |
[production] |
| 13:55 |
<moritzm> |
rebooting codfw poolcounters for kernel update |
[production] |
| 13:47 |
<moritzm> |
rebooting osmium for kernel update |
[production] |
| 13:28 |
<cmjohnson1> |
mw1145 swapped eth0 cable |
[production] |
| 13:04 |
<moritzm> |
rebooting mira for kernel update |
[production] |
| 12:59 |
<moritzm> |
rebooting francium for kernel update |
[production] |
| 11:23 |
<godog> |
bounce statsdlb on graphite1001, drops are back after yesterday's reboot T101141 |
[production] |
| 11:15 |
<moritzm> |
removed two obsolete, older kernel packages from wtp1002 (had flagged an icinga warning on diskspace on /boot) |
[production] |
| 09:38 |
<elukey> |
rebooted eventlog2001.codfw.wmnet for kernel upgrades |
[production] |
| 09:35 |
<moritzm> |
rolling reboot of swift backends in codfw |
[production] |
| 09:15 |
<moritzm> |
powercycling ms-fe2003, stuck after reboot |
[production] |
| 09:03 |
<moritzm> |
powercycling ms-fe2002, stuck after reboot |
[production] |