6451-6500 of 7932 results (26ms)
2014-07-18 §
00:30 <bd808> removed local l10nupdate user from deployment-jobrunner01 and deployment-videoscaler01 [releng]
00:22 <bd808> Killed stuck beta-update-databases-eqiad job ( stuck for over 60m waiting for executor; deadlock?) [releng]
00:21 <ori> beta broke due to I433826423. app servers load prod apache confs from /etc/apache2/wikimedia. temp fix: locally hack apache2.conf to load /usr/local/apache2/conf/all.conf; disable puppet. [releng]
2014-07-17 §
23:18 <bd808> Puppet broken for deployment-bastion by labs specific logic in misc::deployment::vars. [releng]
19:01 <mwalker> possibly breaking labs by cherry picking an apparmor patch that affects mysql https://gerrit.wikimedia.org/r/#/c/147027/ [releng]
2014-07-16 §
19:15 <mwalker> updated puppet about 20 minutes ago for new ocg variables (now officially in production puppet instead of just cherry picked) [releng]
2014-07-15 §
18:26 <bd808> Removed local mwdeploy user from /etc/passwd on deployment-videoscaler01 and deployment-jobrunner01 [releng]
16:59 <bd808> scap failing to deploymnet-videoscaler01 and deploymnet-jobrunner01 due to other random failures now. Lots of strange permissions errors during rsync [releng]
16:37 <bd808> scap failing to deploymnet-videoscaler01 and deploymnet-jobrunner01 due to ssh auth failures; likely a puppet config problem [releng]
2014-07-10 §
22:37 <bd808> Added Gergő Tisza and Yuvipanda as project admins [releng]
2014-07-08 §
23:37 <bd808> Updated Kibana to 0afda49 (latest upstream head) [releng]
17:03 <greg-g> Added John F. Lewis to the project after his NDA was signed by Mark (RT 7722) [releng]
2014-07-07 §
20:55 <bd808> Killed stuck `apt-get update` job on deployment-jobrunner01 started on Jun17 [releng]
20:20 <bd808> Fixed puppet on deployment-analytics01 with manual apt-get commands. [releng]
20:08 <bd808> Ran `apt-get dist-upgrade` on deployment-analytics01 to upgrade hadoop, hive, pig, etc which were failing to update via puppet. [releng]
2014-07-04 §
02:28 <RoanKattouw> Unbroke replication on deployment-db2, it's catching up now [releng]
2014-07-03 §
18:59 <legoktm> manually created centralauth.renameuser_status table [releng]
16:04 <bd808> Updated scap to ff04431 [releng]
09:24 <hashar> Reindexed ElasticSearch index for cawiki/eswiki with: mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki {cawiki,eswiki} --batch-size=50 [releng]
09:10 <hashar> used addwiki.php to create the wiki. manually triggered the Jenkins job that update the databases https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/2319/ [releng]
09:06 <hashar> Adding cawiki and eswiki for cxserver testing {{gerrit|Ibbcbd43d43f878099788518d9e952fa7b20d8ca3}} [releng]
2014-07-02 §
07:49 <hashar> cxserver being configured! {{gerrit|140723}} by Kartik and Niklas \\O/ [releng]
2014-07-01 §
15:46 <bd808> Fixed git rebase conflict in operations/puppet on deployment-salt [releng]
13:29 <manybubbles> rebuilding Cirrus search index in beta to pick up new configuration and cache warmers [releng]
11:20 <hashar> Added Filippo Giunchedi to the project as an admin (WMF ops) [releng]
2014-06-30 §
20:47 <bd808> The state of puppet for beta is badly broken. I have hacked things to get puppet to apply on deployment-apache0[12] but puppet won't apply on deployment-bastion in part due to the same hacks. [releng]
18:48 <bd808> Created symlink /apache -> /usr/local/apache on deployment-apache0[12] to fix docroot symlinks [releng]
18:09 <bd808> Beta apaches are broken with latest puppet config applied. Working to correct. [releng]
18:08 <bd808> Manually added symlink for /etc/apache/wmf on deployment-apache0[12] [releng]
2014-06-26 §
12:48 <YuviPanda> cherry picked https://gerrit.wikimedia.org/r/#/c/142228/ to puppetmaster, sending events to charcoal.wmflabs.org now with projectname \\o/ [releng]
09:46 <YuviPanda> cherry-picked https://gerrit.wikimedia.org/r/#/c/142210/ on to puppetmaster [releng]
09:38 <hashar> Granting sudo to YuviPanda [releng]
2014-06-25 §
20:58 <bd808> Fixed rebase conflict in operations/puppet.git on deployment-salt caused by cherry-picked vcl patch left over from varnish submodule usage [releng]
2014-06-24 §
19:29 <bd808> Manually updated operations/puppet checkout on deployment-salt to deal with varnish submodule change [releng]
2014-06-19 §
22:47 <bd808> Updated scap to 792a572 [releng]
22:46 <bd808> Trebuchet runs on deployment-videoscaler01 are succeeding but not showing up in the `git deploy report` output [releng]
22:40 <bd808> Deleted /var/log/diamond/diamond.log on deployment-jobrunner01 because /var was full [releng]
2014-06-18 §
16:55 <bd808> Setup hourly cron as user bd808 on deployment-salt to test automatic update of puppet repo using ~bd808/git-sync-upstream script [releng]
2014-06-17 §
20:36 <bd808> Upgraded elasticsearch to version 1.2.1 on deployment-logstash1 [releng]
2014-06-16 §
21:16 <bd808> Jenkins beta-scap-eqiad job broken because of missing puppet config on deployment-jobrunner01; needs role::beta::scap_target [releng]
20:36 <bd808> Enabled puppet on deployment-jobrunner01 and forced a run [releng]
20:34 <bd808> Puppet disabled on deployment-jobrunner01 since 2014-06-03; No SAL logs explaining why [releng]
20:19 <bd808> Updated scap to 5adce72; trebuchet reported i-00000237 (deployment-videoscaler01) as not updating, but manual check shows it did sync properly [releng]
20:00 <bd808> Deleted /var/lib/puppet/state/agent_catalog_run.lock on deployment-bastion after verifying that no puppet processes were running [releng]
19:55 <bd808> Truncated /var/log/diamond/diamond.log and restarted diamond on deployment-bastion [releng]
19:36 <bd808> /var/log/diamond is 787M of 1.2G total logs [releng]
19:29 <bd808> /var 0% free on deployment-bastion; looking for things to clean-up [releng]
2014-06-09 §
15:19 <andrewbogott> doing a 'rebase origin' on deployment-salt, because it needs it. [releng]
15:10 <andrewbogott> updating all instances to puppet 3 via a cherry-pick�� of https://gerrit.wikimedia.org/r/#/c/137898/ on deployment-salt [releng]
2014-06-07 §
02:44 <bd808> Restarted logstash on deployment-logstash1; last even logged at 2014-06-06T22:11:04 [releng]