|
2014-07-30
§
|
| 16:00 |
<bd808> |
Puppet runs on videoscaler01 and jobrunner01 failing for "Could not find dependency Ferm::Rule[bastion-ssh] for Ferm::Rule[deployment-bastion-scap-ssh]" |
[releng] |
| 16:00 |
<bd808> |
Puppet seems manually disabled on apache0[12]. |
[releng] |
| 15:59 |
<bd808> |
Can't ssh to apache0[12], videoscaler01 and jobrunner01. Puppet not running on any of them. libnss-ldapd unattended update has broken /etc/nslcd.conf |
[releng] |
| 15:23 |
<bd808> |
Removed cherry-pick for Iac547efa83cf059a1276b6e279c3ebd4c7224b2c and updated cherry-pick for I5afba2c6b0fbf90ff8495cc4a82f5c7851893b52 to latest patch set. |
[releng] |
| 15:21 |
<hoo> |
Synchronized php-1.24wmf15/extensions/Wikidata/extensions/Wikibase/lib/resources/wikibase.js: touch (duration: 00m 20s) |
[production] |
| 15:17 |
<hashar> |
upgrading php5 on jenkins slaves |
[production] |
| 15:07 |
<cmjohnson1> |
shutting down neon |
[production] |
| 15:05 |
<bd808> |
Two cherry-picks in puppet conflicting with merged production changes: I5afba2c6b0fbf90ff8495cc4a82f5c7851893b52 and Iac547efa83cf059a1276b6e279c3ebd4c7224b2c (ori, twentyafterfour) |
[releng] |
| 14:49 |
<bd808> |
Started apache2 service on deployment-mediawiki01 |
[releng] |
| 14:46 |
<demon> |
Synchronized wmf-config/CirrusSearch-production.php: (no message) (duration: 00m 04s) |
[production] |
| 14:35 |
<demon> |
Synchronized wmf-config/PrivateSettings.php: Swift config for Cirrus (duration: 00m 08s) |
[production] |
| 14:30 |
<godog> |
rolling restart of ms-fe* to pick up search backup user |
[production] |
| 14:17 |
<bblack> |
rebooting neon again, trying to fix the disk situation |
[production] |
| 14:16 |
<hashar> |
rebooting hhvm |
[releng] |
| 14:11 |
<Coren> |
reinstalling labmon1001 -> change disk partitioning scheme |
[production] |
| 13:50 |
<springle> |
neon read-only fs. fsck + reboot |
[production] |
| 13:17 |
<manybubbles> |
rebuiding Cirrus index for commons to pick up weighted all field |
[production] |
| 11:17 |
<_joe_> |
enabling puppet on all mw* servers |
[production] |
| 11:15 |
<_joe_> |
re-enabling puppet on mw1019, last bunch of tests, then re-enabling globally |
[production] |
| 10:58 |
<_joe_> |
re-enabling puppet on mw1018, testwiki upgraded to the new config and looks fine |
[production] |
| 09:42 |
<hashar> |
bastion had broken puppet because deployment_server and zuul both declare the same python packages {{gerrit|150501}} |
[releng] |
| 09:40 |
<hashar> |
restoring on puppetmaster modules/mediawiki/templates/apache/apache2.conf.erb which got deleted somehow |
[releng] |
| 09:29 |
<hashar> |
Rebooting apache01/02 to see whether it fix the ssh connection issue |
[releng] |
| 09:27 |
<hashar> |
manually started hhvm on mediawiki01 |
[releng] |
| 09:25 |
<godog> |
set weight for ms-be1014 and ms-be1015 to 2300 |
[production] |
| 09:25 |
<hashar> |
rebooting deployment-mediawiki01 hhvm process went zombie |
[releng] |
| 09:23 |
<hashar> |
restarting hhvm on mediawiki 01/02 |
[releng] |
| 09:05 |
<hashar_> |
Beta scap script broken since 6:30am UTC https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ |
[releng] |
| 08:58 |
<_joe_> |
stopping puppet on the appservers, in preparation for releasing change 148099 |
[production] |
| 08:30 |
<_joe_> |
powercycling neon, doesn't respond to requests, ssh hangs, console dark |
[production] |
| 06:41 |
<springle> |
labsdb1001 work in progress; it may misbehave. see labs-l for updates |
[production] |
| 04:29 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Jul 30 04:27:56 UTC 2014 (duration 27m 55s) |
[production] |
| 03:39 |
<LocalisationUpdate> |
completed (1.24wmf15) at 2014-07-30 03:38:28+00:00 |
[production] |
| 02:51 |
<LocalisationUpdate> |
completed (1.24wmf14) at 2014-07-30 02:50:14+00:00 |
[production] |
| 01:47 |
<bblack> |
ip addr del for cp4017's ip6_mapped addr on cp4018 (no idea why it was there...) |
[production] |
|
2014-07-29
§
|
| 23:37 |
<catrope> |
Finished scap: SWAT updates for wmf15, I'm lazy (duration: 07m 02s) |
[production] |
| 23:30 |
<AaronSchulz> |
Updated /srv/jobrunner to d2298139ea22bf8e48de066a73f28024b140ea33 |
[production] |
| 23:30 |
<catrope> |
Started scap: SWAT updates for wmf15, I'm lazy |
[production] |
| 23:28 |
<catrope> |
Synchronized php-1.24wmf14/extensions/VisualEditor: (no message) (duration: 00m 05s) |
[production] |
| 23:28 |
<catrope> |
Synchronized php-1.24wmf14/extensions/MobileFrontend: (no message) (duration: 00m 05s) |
[production] |
| 23:18 |
<catrope> |
Synchronized wmf-config/: Do not put OCG in sidebar (duration: 00m 04s) |
[production] |
| 23:11 |
<catrope> |
Synchronized wmf-config/: Enable TemplateData GUI on nlwiki (duration: 00m 05s) |
[production] |
| 23:10 |
<bblack> |
took OCG service IP out of downtime in icinga, it's live |
[production] |
| 23:06 |
<mwalker> |
Synchronized wmf-config: Enabling OCG in production (duration: 00m 04s) |
[production] |
| 23:05 |
<aaron> |
Synchronized rpc: 0df032d957155aa475d99e2b887ba98b9a4c32fd (duration: 00m 07s) |
[production] |
| 23:04 |
<cscott> |
Synchronized wmf-config: (no message) (duration: 00m 12s) |
[production] |
| 23:03 |
<cscott> |
updated /a/common to {{Gerrit|Iae1ac79d5}}: Enable OCG in production |
[production] |
| 22:56 |
<cscott> |
updated OCG to version aeb8623d6ebe41ae7c7e36c57844bd9ea8e6d595 |
[releng] |
| 22:55 |
<cscott> |
updated OCG to version aeb8623d6ebe41ae7c7e36c57844bd9ea8e6d595 |
[production] |
| 22:50 |
<RoanKattouw> |
Fixed ownership of slot0/cache on wikitech (virt1000), was root:root but should have been www-data:www-data |
[production] |