|
2016-12-18
§
|
| 22:34 |
<ariel@tin> |
Finished deploy [dumps/dumps@92946f0]: make monitoring more robust (duration: 00m 01s) |
[production] |
| 22:34 |
<ariel@tin> |
Starting deploy [dumps/dumps@92946f0]: make monitoring more robust |
[production] |
| 22:17 |
<ariel@tin> |
Finished deploy [dumps/dumps@2a35e23]: fix checkpoint prefetch jobs (duration: 00m 02s) |
[production] |
| 22:17 |
<ariel@tin> |
Starting deploy [dumps/dumps@2a35e23]: fix checkpoint prefetch jobs |
[production] |
| 18:32 |
<WMFlabs> |
Testing |
[production] |
| 16:45 |
<elukey> |
starting cassandra instances on restbase1009, restbase1011 and restbase1013 (one at the time) - T153588 |
[production] |
| 12:38 |
<mobrovac> |
started back cassandra restbase1009-a |
[production] |
| 12:27 |
<mobrovac> |
started back cassandra restbase1011-c |
[production] |
| 12:17 |
<mobrovac> |
started back cassandra restbase1013-c |
[production] |
| 12:08 |
<mobrovac> |
disabling puppet on restbase1009, restbase1011 and restbase1013 due to cassandra OOMs |
[production] |
| 08:57 |
<elukey> |
forced restart of cassandra-c on restbase1011 |
[production] |
| 08:51 |
<elukey> |
forced restart of cassandra-b/c on restbase1013 (b not really needed, my error) |
[production] |
| 08:49 |
<elukey> |
forced restart for cassandra-a on restbase1009 (still OOMs) |
[production] |
| 08:43 |
<elukey> |
forced puppet on restbase1009 to bring up cassandra-a (stopped due to OOM issues) |
[production] |
| 07:07 |
<godog> |
force git-fat pull for twcs on restbase1* to restore twcs jar |
[production] |
| 02:23 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Sun Dec 18 02:23:11 UTC 2016 (duration 4m 20s) |
[production] |
| 02:18 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.6) (duration: 06m 39s) |
[production] |
|
2016-12-16
§
|
| 23:53 |
<mutante> |
same fix for other broken 'mw-canary' mw1261 - 1268 - killed dpkg, dpkg --configure -a , apt-get install php5 (after upgrade to 5.6.29 in combination with php-pear hangs at postinst) |
[production] |
| 23:45 |
<mutante> |
mw1262 - killed dpkg, dpkg --configure -a , apt-get install php5 |
[production] |
| 23:41 |
<mutante> |
tungsten - fixed hanging dpkg install, killed, dpkg-reconfigure libapache2-mod-php5 |
[production] |
| 23:19 |
<mutante> |
upgrading php5 to 5.6.29 on mw canary (DSA-3737-1) |
[production] |
| 22:39 |
<eevans@tin> |
Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 05s) |
[production] |
| 22:39 |
<eevans@tin> |
Starting deploy [cassandra/twcs@0b0c838]: (no message) |
[production] |
| 22:38 |
<mobrovac> |
restbase deployed the latest code and pooled restbase1018 |
[production] |
| 22:38 |
<eevans@tin> |
Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 04s) |
[production] |
| 22:38 |
<eevans@tin> |
Starting deploy [cassandra/twcs@0b0c838]: (no message) |
[production] |
| 22:37 |
<eevans@tin> |
Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 04s) |
[production] |
| 22:37 |
<eevans@tin> |
Starting deploy [cassandra/twcs@0b0c838]: (no message) |
[production] |
| 22:36 |
<eevans@tin> |
Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 10s) |
[production] |
| 22:36 |
<eevans@tin> |
Starting deploy [cassandra/twcs@0b0c838]: (no message) |
[production] |
| 22:29 |
<mutante> |
salt master: deleting unacceptd keys for decom'ed hosts neon and palladium, accepting key for restbase1018 |
[production] |
| 22:24 |
<mutante> |
restbase1018 - signing puppet cert, initial run |
[production] |
| 21:35 |
<mutante> |
labmon1001 - upgrade apache, gnupg, host, openssh-*, openssl |
[production] |
| 20:31 |
<mutante> |
eventlog2001 - upgraded scap | bohrium - upgraded salt | multatuli - upgraded snimpy (all one-offs from servermon list) |
[production] |
| 20:17 |
<mutante> |
silver (wikitech) - upgraded openssh-sftp-server, login, firejail |
[production] |
| 20:14 |
<mutante> |
stat1002 - /etc/sudoers is puppetized but package upgrades of sudo want to override it and suggest to put local modifications in /etc/sudoers.d/, keeping installed version |
[production] |
| 20:09 |
<mutante> |
stat1002 - install various package upgrades |
[production] |
| 20:05 |
<mutante> |
helium (backups) - upgrade apache2, openssl, python, dpkg .. |
[production] |
| 20:02 |
<mutante> |
iridium (phabricator) - upgrade openssl, openssh-server, sftp, python2.7 |
[production] |
| 19:58 |
<mutante> |
carbon (install) apt-get upgrade |
[production] |
| 19:27 |
<mutante> |
rolling out php upgrades on mw-codfw |
[production] |
| 19:24 |
<mutante> |
contint1001, tungsten, hafnium - upgrade php5-cli/-common, libapache-mod-php5,.. |
[production] |
| 19:21 |
<mutante> |
planet[12]001, rutherfordium (people.wm.org), ununpentium (RT) - upgrade php5-cli/-common, libapache-mod-php5,.. |
[production] |
| 18:49 |
<mobrovac@tin> |
Finished deploy [citoid/deploy@d05c8bb]: (no message) (duration: 00m 39s) |
[production] |