|
2016-12-18
§
|
| 22:17 |
<ariel@tin> |
Starting deploy [dumps/dumps@2a35e23]: fix checkpoint prefetch jobs |
[production] |
| 18:32 |
<WMFlabs> |
Testing |
[production] |
| 16:45 |
<elukey> |
starting cassandra instances on restbase1009, restbase1011 and restbase1013 (one at the time) - T153588 |
[production] |
| 12:38 |
<mobrovac> |
started back cassandra restbase1009-a |
[production] |
| 12:27 |
<mobrovac> |
started back cassandra restbase1011-c |
[production] |
| 12:17 |
<mobrovac> |
started back cassandra restbase1013-c |
[production] |
| 12:08 |
<mobrovac> |
disabling puppet on restbase1009, restbase1011 and restbase1013 due to cassandra OOMs |
[production] |
| 08:57 |
<elukey> |
forced restart of cassandra-c on restbase1011 |
[production] |
| 08:51 |
<elukey> |
forced restart of cassandra-b/c on restbase1013 (b not really needed, my error) |
[production] |
| 08:49 |
<elukey> |
forced restart for cassandra-a on restbase1009 (still OOMs) |
[production] |
| 08:43 |
<elukey> |
forced puppet on restbase1009 to bring up cassandra-a (stopped due to OOM issues) |
[production] |
| 07:07 |
<godog> |
force git-fat pull for twcs on restbase1* to restore twcs jar |
[production] |
| 02:23 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Dec 18 02:23:11 UTC 2016 (duration 4m 20s) |
[production] |
| 02:18 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.6) (duration: 06m 39s) |
[production] |
|
2016-12-17
§
|
| 22:20 |
<Krenair> |
restarting pod, seems to be having ping handling issues? |
[tools.lolrrit-wm] |
| 22:03 |
<multichill> |
Added .lighttpd.conf and webservice restart so that logs are now send as "text/plain;charset=UTF-8" |
[tools.noclaims] |
| 20:55 |
<multichill> |
Moved the two jobs here (one in the morning and one in the evening) and updated https://www.wikidata.org/wiki/User:NoclaimsBot |
[tools.noclaims] |
| 20:22 |
<Zppix> |
restarted web service to clear cache |
[tools.zppixbot] |
| 20:11 |
<multichill> |
Set up the bot with a clone of https://github.com/multichill/toollabs and a symlinked pywikibot (git clone is broken see phab:T151351 ) |
[tools.noclaims] |
| 09:38 |
<elukey> |
ran apt-get clean and removed some /tmp files on stat1002 to free some space |
[production] |
| 09:24 |
<elukey> |
restarted stuck hhvm on mw1168 (forgot to run hhvm-dump-debug) |
[production] |
| 04:50 |
<yuvipanda> |
kill process running on tools-login, was using up all NFS bandwidth |
[tools.gpsexif] |
| 04:49 |
<yuvipanda> |
turned on lookupcache again for bastions |
[tools] |
| 02:37 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sat Dec 17 02:37:21 UTC 2016 (duration 4m 30s) |
[production] |
| 02:32 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.6) (duration: 13m 14s) |
[production] |
| 02:01 |
<legoktm> |
grrrit-wm is currently running in legoktm's mosh session on tools-login |
[tools.lolrrit-wm] |
| 01:31 |
<Zppix> |
grrrit-wm-test is grouped to grrrit-wm account |
[tools.lolrrit-wm] |
|
2016-12-16
§
|
| 23:53 |
<mutante> |
same fix for other broken 'mw-canary' mw1261 - 1268 - killed dpkg, dpkg --configure -a , apt-get install php5 (after upgrade to 5.6.29 in combination with php-pear hangs at postinst) |
[production] |
| 23:45 |
<mutante> |
mw1262 - killed dpkg, dpkg --configure -a , apt-get install php5 |
[production] |
| 23:41 |
<mutante> |
tungsten - fixed hanging dpkg install, killed, dpkg-reconfigure libapache2-mod-php5 |
[production] |
| 23:19 |
<mutante> |
upgrading php5 to 5.6.29 on mw canary (DSA-3737-1) |
[production] |
| 23:11 |
<bd808> |
Restarted bot to change config for #wikimedia-releng channel |
[tools.stashbot] |
| 22:39 |
<eevans@tin> |
Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 05s) |
[production] |
| 22:39 |
<eevans@tin> |
Starting deploy [cassandra/twcs@0b0c838]: (no message) |
[production] |
| 22:38 |
<mobrovac> |
restbase deployed the latest code and pooled restbase1018 |
[production] |
| 22:38 |
<eevans@tin> |
Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 04s) |
[production] |
| 22:38 |
<eevans@tin> |
Starting deploy [cassandra/twcs@0b0c838]: (no message) |
[production] |
| 22:37 |
<eevans@tin> |
Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 04s) |
[production] |
| 22:37 |
<eevans@tin> |
Starting deploy [cassandra/twcs@0b0c838]: (no message) |
[production] |
| 22:36 |
<eevans@tin> |
Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 10s) |
[production] |
| 22:36 |
<eevans@tin> |
Starting deploy [cassandra/twcs@0b0c838]: (no message) |
[production] |
| 22:34 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/327202 |
[releng] |
| 22:29 |
<mutante> |
salt master: deleting unacceptd keys for decom'ed hosts neon and palladium, accepting key for restbase1018 |
[production] |
| 22:24 |
<mutante> |
restbase1018 - signing puppet cert, initial run |
[production] |
| 21:35 |
<mutante> |
labmon1001 - upgrade apache, gnupg, host, openssh-*, openssl |
[production] |
| 20:31 |
<mutante> |
eventlog2001 - upgraded scap | bohrium - upgraded salt | multatuli - upgraded snimpy (all one-offs from servermon list) |
[production] |
| 20:17 |
<mutante> |
silver (wikitech) - upgraded openssh-sftp-server, login, firejail |
[production] |
| 20:14 |
<mutante> |
stat1002 - /etc/sudoers is puppetized but package upgrades of sudo want to override it and suggest to put local modifications in /etc/sudoers.d/, keeping installed version |
[production] |
| 20:09 |
<mutante> |
stat1002 - install various package upgrades |
[production] |
| 20:05 |
<mutante> |
helium (backups) - upgrade apache2, openssl, python, dpkg .. |
[production] |