|
2016-12-19
§
|
| 13:15 |
<_joe_> |
restarted hhvm, apache on mw1260, raised the apache timeout to 1 day, restarted the jobrunner, disabled puppet |
[production] |
| 13:06 |
<yuvipanda> |
run /usr/local/bin/deploy-master http://tools-docker-builder-03.tools.eqiad.wmflabs v1.3.3wmf1 on tools-k8s-master-01 |
[tools] |
| 12:53 |
<yuvipanda> |
cleaned out pbuilder from tools-docker-builder-01 to clean up |
[tools] |
| 11:47 |
<_joe_> |
disabling puppet, reconfiguring timeout on apache, restarting HHVM on mw1259 |
[production] |
| 10:16 |
<elukey> |
reimaging mw1168 and mw1169 to Trusty - T153488 |
[production] |
| 09:38 |
<elukey> |
stopping jobrunner/jobchron daemons on mw116[89] as prep step for repurpose to videoscalers - T153488 |
[production] |
| 09:35 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/328145 |
[releng] |
| 09:23 |
<marostegui> |
Stop mysql db2048 (depooled) for maintenance - T149553 |
[production] |
| 09:20 |
<elukey> |
killing irc-echo |
[production] |
| 09:04 |
<ariel@tin> |
Finished deploy [dumps/dumps@c8fb9a1]: table jobs to yaml config; stop dumping private tables completely (duration: 00m 01s) |
[production] |
| 09:04 |
<ariel@tin> |
Starting deploy [dumps/dumps@c8fb9a1]: table jobs to yaml config; stop dumping private tables completely |
[production] |
| 08:00 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/288819 https://gerrit.wikimedia.org/r/276065 https://gerrit.wikimedia.org/r/328136 |
[releng] |
| 06:44 |
<marostegui> |
Deploy innodb compression dbstore2001 on dewiki and wikidatawiki - T151552 |
[production] |
| 02:25 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/327692 |
[releng] |
| 02:23 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon Dec 19 02:23:18 UTC 2016 (duration 4m 23s) |
[production] |
| 02:18 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.6) (duration: 06m 39s) |
[production] |
| 00:33 |
<mobrovac> |
starting back cassandra on restbase1011 |
[production] |
|
2016-12-18
§
|
| 22:34 |
<ariel@tin> |
Finished deploy [dumps/dumps@92946f0]: make monitoring more robust (duration: 00m 01s) |
[production] |
| 22:34 |
<ariel@tin> |
Starting deploy [dumps/dumps@92946f0]: make monitoring more robust |
[production] |
| 22:17 |
<ariel@tin> |
Finished deploy [dumps/dumps@2a35e23]: fix checkpoint prefetch jobs (duration: 00m 02s) |
[production] |
| 22:17 |
<ariel@tin> |
Starting deploy [dumps/dumps@2a35e23]: fix checkpoint prefetch jobs |
[production] |
| 18:32 |
<WMFlabs> |
Testing |
[production] |
| 16:45 |
<elukey> |
starting cassandra instances on restbase1009, restbase1011 and restbase1013 (one at the time) - T153588 |
[production] |
| 12:38 |
<mobrovac> |
started back cassandra restbase1009-a |
[production] |
| 12:27 |
<mobrovac> |
started back cassandra restbase1011-c |
[production] |
| 12:17 |
<mobrovac> |
started back cassandra restbase1013-c |
[production] |
| 12:08 |
<mobrovac> |
disabling puppet on restbase1009, restbase1011 and restbase1013 due to cassandra OOMs |
[production] |
| 08:57 |
<elukey> |
forced restart of cassandra-c on restbase1011 |
[production] |
| 08:51 |
<elukey> |
forced restart of cassandra-b/c on restbase1013 (b not really needed, my error) |
[production] |
| 08:49 |
<elukey> |
forced restart for cassandra-a on restbase1009 (still OOMs) |
[production] |
| 08:43 |
<elukey> |
forced puppet on restbase1009 to bring up cassandra-a (stopped due to OOM issues) |
[production] |
| 07:07 |
<godog> |
force git-fat pull for twcs on restbase1* to restore twcs jar |
[production] |
| 02:23 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Dec 18 02:23:11 UTC 2016 (duration 4m 20s) |
[production] |
| 02:18 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.6) (duration: 06m 39s) |
[production] |
|
2016-12-17
§
|
| 22:20 |
<Krenair> |
restarting pod, seems to be having ping handling issues? |
[tools.lolrrit-wm] |
| 22:03 |
<multichill> |
Added .lighttpd.conf and webservice restart so that logs are now send as "text/plain;charset=UTF-8" |
[tools.noclaims] |
| 20:55 |
<multichill> |
Moved the two jobs here (one in the morning and one in the evening) and updated https://www.wikidata.org/wiki/User:NoclaimsBot |
[tools.noclaims] |
| 20:22 |
<Zppix> |
restarted web service to clear cache |
[tools.zppixbot] |
| 20:11 |
<multichill> |
Set up the bot with a clone of https://github.com/multichill/toollabs and a symlinked pywikibot (git clone is broken see phab:T151351 ) |
[tools.noclaims] |
| 09:38 |
<elukey> |
ran apt-get clean and removed some /tmp files on stat1002 to free some space |
[production] |
| 09:24 |
<elukey> |
restarted stuck hhvm on mw1168 (forgot to run hhvm-dump-debug) |
[production] |
| 04:50 |
<yuvipanda> |
kill process running on tools-login, was using up all NFS bandwidth |
[tools.gpsexif] |
| 04:49 |
<yuvipanda> |
turned on lookupcache again for bastions |
[tools] |
| 02:37 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Dec 17 02:37:21 UTC 2016 (duration 4m 30s) |
[production] |
| 02:32 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.6) (duration: 13m 14s) |
[production] |
| 02:01 |
<legoktm> |
grrrit-wm is currently running in legoktm's mosh session on tools-login |
[tools.lolrrit-wm] |
| 01:31 |
<Zppix> |
grrrit-wm-test is grouped to grrrit-wm account |
[tools.lolrrit-wm] |