|
2014-09-29
§
|
| 23:32 |
<andrewbogott> |
stopped apache, nova-scheduler, keystone, puppetmaster on virt0 |
[production] |
| 23:31 |
<bd808> |
/var/lib/jenkins-slave/tmpfs 100% full on lanthanum.eqiad.wmnet |
[production] |
| 23:26 |
<andrewbogott> |
disabling puppet on virt0 so I can kill off services one by one... |
[production] |
| 23:23 |
<maxsem> |
Started scap: SWATting a bunch of stuff |
[production] |
| 23:17 |
<maxsem> |
Synchronized docroot/: <mutante> he killed the dolphin (duration: 00m 06s) |
[production] |
| 23:13 |
<Reedy> |
dist-upgraded logstash1001 and reboot |
[production] |
| 22:47 |
<Reedy> |
dist-upgrade logstash1002 and reboot |
[production] |
| 22:36 |
<Reedy> |
dist-upgrade on logstash1003 and rebooting |
[production] |
| 22:34 |
<Reedy> |
restarted elasticsearch on logstash1003 post java upgrades |
[production] |
| 22:30 |
<Reedy> |
packages upgraded on logstash1002 |
[production] |
| 22:28 |
<mutante> |
silver - shutting down, wait with wiping it for a few days, just incase |
[production] |
| 22:28 |
<Reedy> |
packages upgraded on logstash1001 |
[production] |
| 22:24 |
<Reedy> |
elasticsearch upgradeed to 1.3.2 on logstash1003 |
[production] |
| 22:18 |
<andrewbogott> |
renaming labs-ns1 to labs-ns0 and labs-ns2 to labs-ns1 |
[production] |
| 22:02 |
<Reedy> |
elasticsearch upgradeed to 1.3.2 on logstash1002 |
[production] |
| 22:01 |
<mutante> |
silver - revoke puppet cert, salt-key, stopping services, disable monitoring |
[production] |
| 21:58 |
<mutante> |
stopping udp2log-vumi on silver - not needed anymore per Yuvipanda |
[production] |
| 21:12 |
<Reedy> |
elasticsearch upgradeed to 1.3.2 on logstash1001 |
[production] |
| 20:50 |
<bd808> |
Ran sync-common on tmh1002.eqiad.wmnet for cscott's failed sync-dir there |
[production] |
| 20:49 |
<bd808> |
Ran sync-common on tmh1001.eqiad.wmnet for cscott's failed sync-dir there |
[production] |
| 20:29 |
<cscott> |
Synchronized wmf-config: Switch default PDF renderer to OCG (duration: 00m 15s) |
[production] |
| 20:04 |
<subbu> |
deployed Parsoid version deed30b2 |
[production] |
| 19:41 |
<ottomata> |
restarted varnishkafka on cp3019 to troubleshoot drerrs |
[production] |
| 19:26 |
<Reedy> |
doing rolling upgrade of elasticsearch on logstash100[1-3] |
[production] |
| 17:59 |
<cscott> |
updated OCG to version 89d8f29a24295b05d0643abe976fea83b56575c9 |
[production] |
| 17:58 |
<ori> |
Synchronized php-1.24wmf22/includes/password/Pbkdf2Password.php: I3b0a1de69: Test for string in Pbkdf2Password::crypt() (duration: 00m 05s) |
[production] |
| 17:47 |
<bblack> |
stopped powerdns and disabled puppet on virt1000 to prevent further cache pollution w/ bad data in public caches |
[production] |
| 15:57 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT enable collection extension svwikiversity (duration: 00m 06s) |
[production] |
| 15:53 |
<hashar> |
Zuul jobs reregistered |
[production] |
| 15:46 |
<hashar> |
Zuul lost all Jenkins jobs :( |
[production] |
| 15:24 |
<manybubbles> |
Synchronized php-1.24wmf22/extensions/UploadWizard/: SWAT update UploadWizard (duration: 00m 05s) |
[production] |
| 15:17 |
<manybubbles> |
Synchronized php-1.25wmf1/extensions/Wikidata/: SWAT update wikidata to fix hhvm issues. (duration: 00m 14s) |
[production] |
| 15:05 |
<manybubbles> |
Synchronized wmf-config/wikitech.php: SWAT sync wikitech file - is a noop I believe (duration: 00m 05s) |
[production] |
| 15:02 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT fix config type of flow. (duration: 00m 06s) |
[production] |
| 13:32 |
<hashar> |
Restarted Zuul |
[production] |
| 13:27 |
<hashar> |
Zuul: tweaking configuration files {{gerrit|162584}} |
[production] |
| 09:31 |
<godog> |
deployed new swift ring to eqiad-prod |
[production] |
| 08:21 |
<hashar> |
Restarting Jenkins to have a plugin installed/loaded properly |
[production] |
| 03:25 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Sep 29 03:25:11 UTC 2014 (duration 25m 10s) |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-09-29 02:26:50+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf22) at 2014-09-29 02:15:34+00:00 |
[production] |
| 02:14 |
<bblack> |
restarting squid on carbon (webproxy) |
[production] |