151-200 of 10000 results (37ms)
2016-12-18 §
08:57 <elukey> forced restart of cassandra-c on restbase1011 [production]
08:51 <elukey> forced restart of cassandra-b/c on restbase1013 (b not really needed, my error) [production]
08:49 <elukey> forced restart for cassandra-a on restbase1009 (still OOMs) [production]
08:43 <elukey> forced puppet on restbase1009 to bring up cassandra-a (stopped due to OOM issues) [production]
07:07 <godog> force git-fat pull for twcs on restbase1* to restore twcs jar [production]
02:23 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Dec 18 02:23:11 UTC 2016 (duration 4m 20s) [production]
02:18 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.6) (duration: 06m 39s) [production]
2016-12-17 §
09:38 <elukey> ran apt-get clean and removed some /tmp files on stat1002 to free some space [production]
09:24 <elukey> restarted stuck hhvm on mw1168 (forgot to run hhvm-dump-debug) [production]
02:37 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Dec 17 02:37:21 UTC 2016 (duration 4m 30s) [production]
02:32 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.6) (duration: 13m 14s) [production]
2016-12-16 §
23:53 <mutante> same fix for other broken 'mw-canary' mw1261 - 1268 - killed dpkg, dpkg --configure -a , apt-get install php5 (after upgrade to 5.6.29 in combination with php-pear hangs at postinst) [production]
23:45 <mutante> mw1262 - killed dpkg, dpkg --configure -a , apt-get install php5 [production]
23:41 <mutante> tungsten - fixed hanging dpkg install, killed, dpkg-reconfigure libapache2-mod-php5 [production]
23:19 <mutante> upgrading php5 to 5.6.29 on mw canary (DSA-3737-1) [production]
22:39 <eevans@tin> Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 05s) [production]
22:39 <eevans@tin> Starting deploy [cassandra/twcs@0b0c838]: (no message) [production]
22:38 <mobrovac> restbase deployed the latest code and pooled restbase1018 [production]
22:38 <eevans@tin> Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 04s) [production]
22:38 <eevans@tin> Starting deploy [cassandra/twcs@0b0c838]: (no message) [production]
22:37 <eevans@tin> Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 04s) [production]
22:37 <eevans@tin> Starting deploy [cassandra/twcs@0b0c838]: (no message) [production]
22:36 <eevans@tin> Finished deploy [cassandra/twcs@0b0c838]: (no message) (duration: 00m 10s) [production]
22:36 <eevans@tin> Starting deploy [cassandra/twcs@0b0c838]: (no message) [production]
22:29 <mutante> salt master: deleting unacceptd keys for decom'ed hosts neon and palladium, accepting key for restbase1018 [production]
22:24 <mutante> restbase1018 - signing puppet cert, initial run [production]
21:35 <mutante> labmon1001 - upgrade apache, gnupg, host, openssh-*, openssl [production]
20:31 <mutante> eventlog2001 - upgraded scap | bohrium - upgraded salt | multatuli - upgraded snimpy (all one-offs from servermon list) [production]
20:17 <mutante> silver (wikitech) - upgraded openssh-sftp-server, login, firejail [production]
20:14 <mutante> stat1002 - /etc/sudoers is puppetized but package upgrades of sudo want to override it and suggest to put local modifications in /etc/sudoers.d/, keeping installed version [production]
20:09 <mutante> stat1002 - install various package upgrades [production]
20:05 <mutante> helium (backups) - upgrade apache2, openssl, python, dpkg .. [production]
20:02 <mutante> iridium (phabricator) - upgrade openssl, openssh-server, sftp, python2.7 [production]
19:58 <mutante> carbon (install) apt-get upgrade [production]
19:27 <mutante> rolling out php upgrades on mw-codfw [production]
19:24 <mutante> contint1001, tungsten, hafnium - upgrade php5-cli/-common, libapache-mod-php5,.. [production]
19:21 <mutante> planet[12]001, rutherfordium (people.wm.org), ununpentium (RT) - upgrade php5-cli/-common, libapache-mod-php5,.. [production]
18:49 <mobrovac@tin> Finished deploy [citoid/deploy@d05c8bb]: (no message) (duration: 00m 39s) [production]
18:49 <mobrovac@tin> Starting deploy [citoid/deploy@d05c8bb]: (no message) [production]
18:08 <Zppix> Attention: At 7PM CST UTC-6, grrrit-wm will be going under maintaience and will be down for at most 30 minutes if you have any questions please direct them to me, for more info please go to E424 on Phab! [production]
17:35 <gehel> restart elastic2006.codfw.wmnet [production]
17:16 <gehel> unbanned elastic2006 to complete repooling [production]
17:12 <mutante> mw-canary - upgraded php5 packages [production]
17:05 <mutante> silver (wikitech) - upgrade apache2, apparmor, python [production]
17:04 <mutante> silver (wikitech) - upgrade openssh, openssl [production]
16:47 <gehel@puppetmaster1001> conftool action : set/pooled=yes; selector: name=elastic2006.codfw.wmnet [production]
16:46 <gehel> repooling elastic2006.codfw.wmnet [production]
16:46 <mutante> silver - upgraded php5-fss | copper - upgraded php5-dev | einsteinium/tegmen - upgraded php5-gd, php5* [production]
16:42 <mutante> krypton, mira, wasat, phab2001, bohrium, notebook[12]001 - upgrade php5 packages [production]
16:36 <legoktm@tin> Synchronized php-1.29.0-wmf.6/resources/Resources.php: Revert confirmation button and apply other fixes - T153389 T153438 (duration: 00m 39s) [production]