|
2014-09-30
§
|
| 14:22 |
<oblivian> |
gracefulled all apaches |
[production] |
| 12:10 |
<mark> |
Stopped exim daemon on mchenry |
[production] |
| 09:41 |
<godog> |
removed obsolete /etc/puppet/hiera from strontium and palladium, /etc/puppet/hieradata is the new location |
[production] |
| 09:24 |
<godog> |
reboot ms-be2001 as a test |
[production] |
| 04:18 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Tue Sep 30 04:18:48 UTC 2014 (duration 18m 47s) |
[production] |
| 03:17 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-09-30 03:17:15+00:00 |
[production] |
| 02:41 |
<ori> |
Synchronized 503.html: Ia88b306ef: Make the 503 error page consistent with other 5xx error pages (duration: 00m 08s) |
[production] |
| 02:34 |
<LocalisationUpdate> |
completed (1.24wmf22) at 2014-09-30 02:34:07+00:00 |
[production] |
| 01:00 |
<Krinkle> |
Jenkins connection seemed in order with integration-slave1007 and 8, but disconnecting and relaunching the slave agents immediately resulted in them getting jobs assigned. Cause unknown, problem resolved for now. |
[production] |
| 00:58 |
<Krinkle> |
integration-slave1007 and integration-slave1008 have not gotten any jobs in the past 24h. integration-slave1006 however has gotten loads of action. Investigating load balancing issue. |
[production] |
| 00:24 |
<mutante> |
linne - shutting down, revoking puppet cert, salt key, puppet/icinga ... |
[production] |
| 00:12 |
<maxsem> |
Synchronized w/skins-1.5: (no message) (duration: 00m 03s) |
[production] |
| 00:12 |
<MaxSem> |
https://gerrit.wikimedia.org/r/#/c/162520/ broke stuff, reverted |
[production] |
| 00:10 |
<maxsem> |
Synchronized live-1.5: (no message) (duration: 00m 03s) |
[production] |
|
2014-09-29
§
|
| 23:59 |
<maxsem> |
Synchronized w: https://gerrit.wikimedia.org/r/#/c/162520/ (duration: 00m 03s) |
[production] |
| 23:58 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/163773/ (duration: 00m 03s) |
[production] |
| 23:50 |
<maxsem> |
Synchronized php-1.25wmf1/extensions/MultimediaViewer/: second try... (duration: 00m 04s) |
[production] |
| 23:42 |
<maxsem> |
Finished scap: SWATting a bunch of stuff (duration: 18m 44s) |
[production] |
| 23:32 |
<andrewbogott> |
stopped apache, nova-scheduler, keystone, puppetmaster on virt0 |
[production] |
| 23:31 |
<bd808> |
/var/lib/jenkins-slave/tmpfs 100% full on lanthanum.eqiad.wmnet |
[production] |
| 23:26 |
<andrewbogott> |
disabling puppet on virt0 so I can kill off services one by one... |
[production] |
| 23:23 |
<maxsem> |
Started scap: SWATting a bunch of stuff |
[production] |
| 23:17 |
<maxsem> |
Synchronized docroot/: <mutante> he killed the dolphin (duration: 00m 06s) |
[production] |
| 23:13 |
<Reedy> |
dist-upgraded logstash1001 and reboot |
[production] |
| 22:47 |
<Reedy> |
dist-upgrade logstash1002 and reboot |
[production] |
| 22:36 |
<Reedy> |
dist-upgrade on logstash1003 and rebooting |
[production] |
| 22:34 |
<Reedy> |
restarted elasticsearch on logstash1003 post java upgrades |
[production] |
| 22:30 |
<Reedy> |
packages upgraded on logstash1002 |
[production] |
| 22:28 |
<mutante> |
silver - shutting down, wait with wiping it for a few days, just incase |
[production] |
| 22:28 |
<Reedy> |
packages upgraded on logstash1001 |
[production] |
| 22:24 |
<Reedy> |
elasticsearch upgradeed to 1.3.2 on logstash1003 |
[production] |
| 22:18 |
<andrewbogott> |
renaming labs-ns1 to labs-ns0 and labs-ns2 to labs-ns1 |
[production] |
| 22:02 |
<Reedy> |
elasticsearch upgradeed to 1.3.2 on logstash1002 |
[production] |
| 22:01 |
<mutante> |
silver - revoke puppet cert, salt-key, stopping services, disable monitoring |
[production] |
| 21:58 |
<mutante> |
stopping udp2log-vumi on silver - not needed anymore per Yuvipanda |
[production] |
| 21:12 |
<Reedy> |
elasticsearch upgradeed to 1.3.2 on logstash1001 |
[production] |
| 20:50 |
<bd808> |
Ran sync-common on tmh1002.eqiad.wmnet for cscott's failed sync-dir there |
[production] |
| 20:49 |
<bd808> |
Ran sync-common on tmh1001.eqiad.wmnet for cscott's failed sync-dir there |
[production] |
| 20:29 |
<cscott> |
Synchronized wmf-config: Switch default PDF renderer to OCG (duration: 00m 15s) |
[production] |
| 20:04 |
<subbu> |
deployed Parsoid version deed30b2 |
[production] |
| 19:41 |
<ottomata> |
restarted varnishkafka on cp3019 to troubleshoot drerrs |
[production] |
| 19:26 |
<Reedy> |
doing rolling upgrade of elasticsearch on logstash100[1-3] |
[production] |
| 17:59 |
<cscott> |
updated OCG to version 89d8f29a24295b05d0643abe976fea83b56575c9 |
[production] |
| 17:58 |
<ori> |
Synchronized php-1.24wmf22/includes/password/Pbkdf2Password.php: I3b0a1de69: Test for string in Pbkdf2Password::crypt() (duration: 00m 05s) |
[production] |
| 17:47 |
<bblack> |
stopped powerdns and disabled puppet on virt1000 to prevent further cache pollution w/ bad data in public caches |
[production] |
| 15:57 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT enable collection extension svwikiversity (duration: 00m 06s) |
[production] |
| 15:53 |
<hashar> |
Zuul jobs reregistered |
[production] |
| 15:46 |
<hashar> |
Zuul lost all Jenkins jobs :( |
[production] |
| 15:24 |
<manybubbles> |
Synchronized php-1.24wmf22/extensions/UploadWizard/: SWAT update UploadWizard (duration: 00m 05s) |
[production] |
| 15:17 |
<manybubbles> |
Synchronized php-1.25wmf1/extensions/Wikidata/: SWAT update wikidata to fix hhvm issues. (duration: 00m 14s) |
[production] |