|
2014-09-29
§
|
| 22:47 |
<Reedy> |
dist-upgrade logstash1002 and reboot |
[production] |
| 22:36 |
<Reedy> |
dist-upgrade on logstash1003 and rebooting |
[production] |
| 22:34 |
<Reedy> |
restarted elasticsearch on logstash1003 post java upgrades |
[production] |
| 22:30 |
<Reedy> |
packages upgraded on logstash1002 |
[production] |
| 22:28 |
<mutante> |
silver - shutting down, wait with wiping it for a few days, just incase |
[production] |
| 22:28 |
<Reedy> |
packages upgraded on logstash1001 |
[production] |
| 22:24 |
<Reedy> |
elasticsearch upgradeed to 1.3.2 on logstash1003 |
[production] |
| 22:18 |
<andrewbogott> |
renaming labs-ns1 to labs-ns0 and labs-ns2 to labs-ns1 |
[production] |
| 22:02 |
<Reedy> |
elasticsearch upgradeed to 1.3.2 on logstash1002 |
[production] |
| 22:01 |
<mutante> |
silver - revoke puppet cert, salt-key, stopping services, disable monitoring |
[production] |
| 21:58 |
<mutante> |
stopping udp2log-vumi on silver - not needed anymore per Yuvipanda |
[production] |
| 21:12 |
<Reedy> |
elasticsearch upgradeed to 1.3.2 on logstash1001 |
[production] |
| 20:50 |
<bd808> |
Ran sync-common on tmh1002.eqiad.wmnet for cscott's failed sync-dir there |
[production] |
| 20:49 |
<bd808> |
Ran sync-common on tmh1001.eqiad.wmnet for cscott's failed sync-dir there |
[production] |
| 20:29 |
<cscott> |
Synchronized wmf-config: Switch default PDF renderer to OCG (duration: 00m 15s) |
[production] |
| 20:04 |
<subbu> |
deployed Parsoid version deed30b2 |
[production] |
| 19:41 |
<ottomata> |
restarted varnishkafka on cp3019 to troubleshoot drerrs |
[production] |
| 19:26 |
<Reedy> |
doing rolling upgrade of elasticsearch on logstash100[1-3] |
[production] |
| 17:59 |
<cscott> |
updated OCG to version 89d8f29a24295b05d0643abe976fea83b56575c9 |
[production] |
| 17:58 |
<ori> |
Synchronized php-1.24wmf22/includes/password/Pbkdf2Password.php: I3b0a1de69: Test for string in Pbkdf2Password::crypt() (duration: 00m 05s) |
[production] |
| 17:47 |
<bblack> |
stopped powerdns and disabled puppet on virt1000 to prevent further cache pollution w/ bad data in public caches |
[production] |
| 15:57 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT enable collection extension svwikiversity (duration: 00m 06s) |
[production] |
| 15:53 |
<hashar> |
Zuul jobs reregistered |
[production] |
| 15:46 |
<hashar> |
Zuul lost all Jenkins jobs :( |
[production] |
| 15:24 |
<manybubbles> |
Synchronized php-1.24wmf22/extensions/UploadWizard/: SWAT update UploadWizard (duration: 00m 05s) |
[production] |
| 15:17 |
<manybubbles> |
Synchronized php-1.25wmf1/extensions/Wikidata/: SWAT update wikidata to fix hhvm issues. (duration: 00m 14s) |
[production] |
| 15:05 |
<manybubbles> |
Synchronized wmf-config/wikitech.php: SWAT sync wikitech file - is a noop I believe (duration: 00m 05s) |
[production] |
| 15:02 |
<manybubbles> |
Synchronized wmf-config/InitialiseSettings.php: SWAT fix config type of flow. (duration: 00m 06s) |
[production] |
| 13:32 |
<hashar> |
Restarted Zuul |
[production] |
| 13:27 |
<hashar> |
Zuul: tweaking configuration files {{gerrit|162584}} |
[production] |
| 09:31 |
<godog> |
deployed new swift ring to eqiad-prod |
[production] |
| 08:21 |
<hashar> |
Restarting Jenkins to have a plugin installed/loaded properly |
[production] |
| 03:25 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Mon Sep 29 03:25:11 UTC 2014 (duration 25m 10s) |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-09-29 02:26:50+00:00 |
[production] |
| 02:15 |
<LocalisationUpdate> |
completed (1.24wmf22) at 2014-09-29 02:15:34+00:00 |
[production] |
| 02:14 |
<bblack> |
restarting squid on carbon (webproxy) |
[production] |
|
2014-09-28
§
|
| 23:27 |
<springle> |
Synchronized wmf-config/db-eqiad.php: switch db1042 load groups to db1056 (duration: 00m 06s) |
[production] |
| 23:17 |
<springle> |
powercycle db1042 |
[production] |
| 23:15 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1042, locked up (duration: 00m 07s) |
[production] |
| 23:12 |
<bblack> |
restarted apache on mw1123 + mw1196 |
[production] |
| 23:11 |
<bblack> |
test |
[production] |
| 23:11 |
<bblack> |
restarted apache on mw1123 + mw1196 |
[production] |
| 20:28 |
<ori> |
Puppet failures appear to be caused by apt-get timeouts |
[production] |
| 10:09 |
<_joe_> |
updated bash (again) across the whole cluster |
[production] |
| 03:24 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sun Sep 28 03:24:20 UTC 2014 (duration 24m 19s) |
[production] |
| 02:28 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-09-28 02:28:14+00:00 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.24wmf22) at 2014-09-28 02:17:01+00:00 |
[production] |