|
2014-09-28
§
|
| 23:27 |
<springle> |
Synchronized wmf-config/db-eqiad.php: switch db1042 load groups to db1056 (duration: 00m 06s) |
[production] |
| 23:17 |
<springle> |
powercycle db1042 |
[production] |
| 23:15 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1042, locked up (duration: 00m 07s) |
[production] |
| 23:12 |
<bblack> |
restarted apache on mw1123 + mw1196 |
[production] |
| 23:11 |
<bblack> |
test |
[production] |
| 23:11 |
<bblack> |
restarted apache on mw1123 + mw1196 |
[production] |
| 20:28 |
<ori> |
Puppet failures appear to be caused by apt-get timeouts |
[production] |
| 10:09 |
<_joe_> |
updated bash (again) across the whole cluster |
[production] |
| 03:24 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sun Sep 28 03:24:20 UTC 2014 (duration 24m 19s) |
[production] |
| 02:28 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-09-28 02:28:14+00:00 |
[production] |
| 02:17 |
<LocalisationUpdate> |
completed (1.24wmf22) at 2014-09-28 02:17:01+00:00 |
[production] |
|
2014-09-26
§
|
| 23:26 |
<mutante> |
switched noc.wikimedia.org to terbium, behind misc-web |
[production] |
| 22:01 |
<bd808> |
sudo apache2ctl graceful on logstash100[123] for ldap revert |
[production] |
| 22:00 |
<bd808> |
running puppet on logstash100[123] to revert ldap change |
[production] |
| 21:56 |
<bd808> |
sudo apache2ctl graceful on logstash100[123] for ldap change |
[production] |
| 21:35 |
<andrewbogott> |
gracefulled apache on neon |
[production] |
| 21:21 |
<mutante> |
graceful'ed apache on neon |
[production] |
| 20:45 |
<ori> |
Synchronized wmf-config/CommonSettings.php: Disable HHVM beta-feature on wikidatawiki (duration: 00m 06s) |
[production] |
| 19:58 |
<awight> |
update CRM from 25159fcfc29921b08de86f12121fb292139be09d to 3e42bac8cb7f58f5e504946f4944c69ca5553e60 |
[production] |
| 19:42 |
<mutante> |
removing root's public_html from fenari - backup kept just in case |
[production] |
| 19:15 |
<AaronS> |
Deployed security patches to CentralAuth |
[production] |
| 19:09 |
<Krinkle> |
git-deploy: Deploying integration/slave-scripts 08147c42ea42e1a5eca1d29 |
[production] |
| 19:09 |
<aaron> |
Synchronized php-1.25wmf1/extensions/CentralAuth: (no message) (duration: 00m 07s) |
[production] |
| 19:06 |
<aaron> |
Synchronized php-1.24wmf22/extensions/CentralAuth: (no message) (duration: 00m 08s) |
[production] |
| 18:15 |
<Nemo_bis> |
untruncated: andrewbogott> ldap is broken for gerrit, should be working elsewhere |
[production] |
| 18:14 |
<legoktm> |
ldap is broken |
[production] |
| 18:10 |
<K4-713> |
re-enabled donations queue consumer |
[production] |
| 17:50 |
<awight> |
CRM queue consumer disabled |
[production] |
| 17:43 |
<andrewbogott> |
upgraded libgnutls26 on ytterbium |
[production] |
| 17:35 |
<andrewbogott> |
"git reset --hard origin" to remove that terrible hotfix on palladium and strontium. |
[production] |
| 17:28 |
<awight> |
CRM jobs reenabled |
[production] |
| 17:22 |
<hoo> |
Manually ran rebuildEntityPerPage.php for Wikidata |
[production] |
| 17:16 |
<andrewbogott> |
hotfixing /var/lib/git/operations/puppet in hopes of fixing gerrit so I don't have to hotfix no more |
[production] |
| 17:08 |
<awight> |
updated crm from 06c9546f9b68f6ecbaaf510944418aa52f9ed0fb to 25159fcfc29921b08de86f12121fb292139be09d |
[production] |
| 17:02 |
<awight> |
disabling CRM jobs for deployment... |
[production] |
| 15:29 |
<andrewbogott> |
puppet is now moving all labs instances to new ldap servers: ldap-eqiad and ldap-codfw |
[production] |
| 15:02 |
<cscott> |
documented what I'm going to clear the OCG queues at https://wikitech.wikimedia.org/wiki/OCG#Pruning_the_queue |
[production] |
| 14:36 |
<bblack> |
address for ns1 switched in our local dns data - https://gerrit.wikimedia.org/r/163164 |
[production] |
| 13:57 |
<hoo> |
Manually declared the global rename Secretary-> VlsergeyBot done after it twice timed out on pages moves on ruwiki |
[production] |
| 13:39 |
<akosiaris> |
moved mathoid to low-traffic lvs servers@eqiad |
[production] |
| 12:49 |
<cscott> |
cleared OCG caches again when I woke up to buy me more time to investigate the issue properly. |
[production] |
| 08:44 |
<awight> |
rollback: revision for civicrm locked to 06c9546f9b68f6ecbaaf510944418aa52f9ed0fb |
[production] |
| 08:30 |
<_joe_> |
updated hhvm on mw1053, kicked the jr a couple of times, working again now |
[production] |
| 08:29 |
<awight> |
large_donation schema migration 7000 |
[production] |