|
2016-12-18
§
|
| 12:27 |
<mobrovac> |
started back cassandra restbase1011-c |
[production] |
| 12:17 |
<mobrovac> |
started back cassandra restbase1013-c |
[production] |
| 12:08 |
<mobrovac> |
disabling puppet on restbase1009, restbase1011 and restbase1013 due to cassandra OOMs |
[production] |
| 08:57 |
<elukey> |
forced restart of cassandra-c on restbase1011 |
[production] |
| 08:51 |
<elukey> |
forced restart of cassandra-b/c on restbase1013 (b not really needed, my error) |
[production] |
| 08:49 |
<elukey> |
forced restart for cassandra-a on restbase1009 (still OOMs) |
[production] |
| 08:43 |
<elukey> |
forced puppet on restbase1009 to bring up cassandra-a (stopped due to OOM issues) |
[production] |
| 07:07 |
<godog> |
force git-fat pull for twcs on restbase1* to restore twcs jar |
[production] |
| 02:23 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Dec 18 02:23:11 UTC 2016 (duration 4m 20s) |
[production] |
| 02:18 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.6) (duration: 06m 39s) |
[production] |
|
2016-12-16
§
|
| 23:53 |
<mutante> |
same fix for other broken 'mw-canary' mw1261 - 1268 - killed dpkg, dpkg --configure -a , apt-get install php5 (after upgrade to 5.6.29 in combination with php-pear hangs at postinst) |
[production] |
| 23:45 |
<mutante> |
mw1262 - killed dpkg, dpkg --configure -a , apt-get install php5 |
[production] |
| 23:41 |
<mutante> |
tungsten - fixed hanging dpkg install, killed, dpkg-reconfigure libapache2-mod-php5 |
[production] |
| 23:19 |
<mutante> |
upgrading php5 to 5.6.29 on mw canary (DSA-3737-1) |
[production] |
| 22:39 |
<eevans@tin> |
Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 05s) |
[production] |
| 22:39 |
<eevans@tin> |
Starting deploy [cassandra/twcs@0b0c838]: (no message) |
[production] |
| 22:38 |
<mobrovac> |
restbase deployed the latest code and pooled restbase1018 |
[production] |
| 22:38 |
<eevans@tin> |
Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 04s) |
[production] |
| 22:38 |
<eevans@tin> |
Starting deploy [cassandra/twcs@0b0c838]: (no message) |
[production] |
| 22:37 |
<eevans@tin> |
Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 04s) |
[production] |
| 22:37 |
<eevans@tin> |
Starting deploy [cassandra/twcs@0b0c838]: (no message) |
[production] |
| 22:36 |
<eevans@tin> |
Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 10s) |
[production] |
| 22:36 |
<eevans@tin> |
Starting deploy [cassandra/twcs@0b0c838]: (no message) |
[production] |
| 22:29 |
<mutante> |
salt master: deleting unacceptd keys for decom'ed hosts neon and palladium, accepting key for restbase1018 |
[production] |
| 22:24 |
<mutante> |
restbase1018 - signing puppet cert, initial run |
[production] |
| 21:35 |
<mutante> |
labmon1001 - upgrade apache, gnupg, host, openssh-*, openssl |
[production] |
| 20:31 |
<mutante> |
eventlog2001 - upgraded scap | bohrium - upgraded salt | multatuli - upgraded snimpy (all one-offs from servermon list) |
[production] |
| 20:17 |
<mutante> |
silver (wikitech) - upgraded openssh-sftp-server, login, firejail |
[production] |
| 20:14 |
<mutante> |
stat1002 - /etc/sudoers is puppetized but package upgrades of sudo want to override it and suggest to put local modifications in /etc/sudoers.d/, keeping installed version |
[production] |
| 20:09 |
<mutante> |
stat1002 - install various package upgrades |
[production] |
| 20:05 |
<mutante> |
helium (backups) - upgrade apache2, openssl, python, dpkg .. |
[production] |
| 20:02 |
<mutante> |
iridium (phabricator) - upgrade openssl, openssh-server, sftp, python2.7 |
[production] |
| 19:58 |
<mutante> |
carbon (install) apt-get upgrade |
[production] |
| 19:27 |
<mutante> |
rolling out php upgrades on mw-codfw |
[production] |
| 19:24 |
<mutante> |
contint1001, tungsten, hafnium - upgrade php5-cli/-common, libapache-mod-php5,.. |
[production] |
| 19:21 |
<mutante> |
planet[12]001, rutherfordium (people.wm.org), ununpentium (RT) - upgrade php5-cli/-common, libapache-mod-php5,.. |
[production] |
| 18:49 |
<mobrovac@tin> |
Finished deploy [citoid/deploy@d05c8bb]: (no message) (duration: 00m 39s) |
[production] |
| 18:49 |
<mobrovac@tin> |
Starting deploy [citoid/deploy@d05c8bb]: (no message) |
[production] |
| 18:08 |
<Zppix> |
Attention: At 7PM CST UTC-6, grrrit-wm will be going under maintaience and will be down for at most 30 minutes if you have any questions please direct them to me, for more info please go to E424 on Phab! |
[production] |
| 17:35 |
<gehel> |
restart elastic2006.codfw.wmnet |
[production] |
| 17:16 |
<gehel> |
unbanned elastic2006 to complete repooling |
[production] |
| 17:12 |
<mutante> |
mw-canary - upgraded php5 packages |
[production] |
| 17:05 |
<mutante> |
silver (wikitech) - upgrade apache2, apparmor, python |
[production] |
| 17:04 |
<mutante> |
silver (wikitech) - upgrade openssh, openssl |
[production] |
| 16:47 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=elastic2006.codfw.wmnet |
[production] |
| 16:46 |
<gehel> |
repooling elastic2006.codfw.wmnet |
[production] |