|
2014-01-23
§
|
| 23:39 |
<hashar> |
Zuul fixed up by killing a stuck job. That somehow prevented Zuul from triggering more jobs :/ |
[production] |
| 23:33 |
<hashar> |
killed job https://integration.wikimedia.org/ci/job/pywikibot-core-tests/412/ |
[production] |
| 23:32 |
<hashar> |
jenkins / zuul broken somehow with jobs triggered but not being run in jenkins. Looking |
[production] |
| 22:20 |
<paravoid> |
testing ulsfo redundancy, ignore possible ulsfo alerts |
[production] |
| 20:45 |
<paravoid> |
setting new ulsfo-eqiad transport |
[production] |
| 20:45 |
<paravoid> |
setting up new transit for ulsfo |
[production] |
| 20:20 |
<RobH> |
further tweaks on gallium, had to fully disable all 443 use, should all be ok now |
[production] |
| 19:51 |
<RobH> |
gallium has had a few patchsets applied recently for removing https support on it, apache restarted, etc. |
[production] |
| 18:30 |
<hashar> |
Gerrit: granted jenkins-mwext-sync user the ability to CR+2, V+2 and submit changes on mediawiki/extensions.git repository |
[production] |
| 18:08 |
<cmjohnson1> |
lutetium replacing failing disk at slot 8 |
[production] |
| 18:08 |
<cmjohnson1> |
lutetium replacing failing disk at slot 8 |
[production] |
| 18:07 |
<cmjohnson1> |
ms-be1008 replacing failing disk slot 11 |
[production] |
| 14:14 |
<mutante> |
fixing cron spam from planet, was caused by wrong permissions on logs, did somebody run update as root? fixed, but please use sudo -u planet |
[production] |
| 09:27 |
<ori> |
restarting carbon on tungsten |
[production] |
| 05:33 |
<reedy> |
updated /a/common to {{Gerrit|Iab1639fd8}}: Add meta charset |
[production] |
| 05:33 |
<reedy> |
synchronized docroot/noc/conf/index.php |
[production] |
| 05:30 |
<reedy> |
synchronized docroot/noc/conf/index.php |
[production] |
| 05:30 |
<reedy> |
updated /a/common to {{Gerrit|Ie8d6d2266}}: Fix broken a tags |
[production] |
| 04:10 |
<reedy> |
synchronized docroot/noc/conf/index.php |
[production] |
| 04:07 |
<reedy> |
synchronized multiversion/ |
[production] |
| 04:05 |
<reedy> |
synchronized docroot and w |
[production] |
| 04:04 |
<reedy> |
updated /a/common to {{Gerrit|I06964b159}}: dirname( __FILE__ ) to __DIR__ |
[production] |
| 02:18 |
<reedy> |
synchronized php-1.23wmf11/extensions/EducationProgram/ |
[production] |
| 02:14 |
<ori> |
synchronized wmf-config/InitialiseSettings.php 'I0dbeff07d: : true everywhere' |
[production] |
| 02:13 |
<ori> |
updated /a/common to {{Gerrit|I0dbeff07d}}: $wgResourceLoaderStorageEnabled: true everywhere |
[production] |
| 02:10 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-23 02:09:57+00:00 |
[production] |
| 02:03 |
<LocalisationUpdate> |
completed (1.23wmf11) at 2014-01-23 02:03:47+00:00 |
[production] |
| 02:02 |
<ori> |
synchronized wmf-config/InitialiseSettings.php 'I16738b6ef: : true everywhere except enwiki' |
[production] |
| 02:02 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-23 02:02:42+00:00 |
[production] |
| 02:02 |
<ori> |
updated /a/common to {{Gerrit|I16738b6ef}}: $wgResourceLoaderStorageEnabled: true everywhere except enwiki |
[production] |
| 01:08 |
<ori> |
synchronized wmf-config/InitialiseSettings.php 'I321ce06ae: Enable module storage on it, de, es and nlwikis' |
[production] |
| 01:08 |
<ori> |
updated /a/common to {{Gerrit|I321ce06ae}}: Enable module storage on it, de, es and nlwikis |
[production] |
| 00:56 |
<hasharMeeting> |
integration.wikimedia.org and doc.wikimedia.org have been successfully migrated behind the misc varnish |
[production] |
| 00:53 |
<ori> |
doing staggered graceful restarts on bits app servers for I9e311c98f |
[production] |
| 00:38 |
<hasharMeeting> |
migrating (with RobH) integration.wikimedia.org being misc varnish |
[production] |
| 00:24 |
<hasharMeeting> |
restarted zuul again |
[production] |
| 00:18 |
<hasharMeeting> |
upgrading Zuul (lightning deploy of doom) |
[production] |
|
2014-01-22
§
|
| 23:41 |
<hasharMeeting> |
Going to upgrade Zuul, need a patch to make it send Cache-Control: no-cache so we can migrate Zuul service behind misc varnish (being done with RobH) |
[production] |
| 22:58 |
<ottomata> |
restarted icinga (a while ago) and puppet freshness checks are all boogery(?) will check up on this tomorrow |
[production] |
| 22:51 |
<RobH> |
doc.wikimedia.org updates complete, once dns re-propagates it should be again reachable over https |
[production] |
| 22:43 |
<RobH> |
doc.wikimedia.org and doc.mediawiki.org relocating behind misc-web-lb. dns ttl was set to 5 minutes yesterday, so this change should result in less than 5 minutes of unreachability by anyone |
[production] |
| 22:09 |
<reedy> |
updated /a/common to {{Gerrit|Ie95fdf8e9}}: Adjust linkpurge and renderfile limits |
[production] |
| 21:26 |
<matanya> |
git.wikimedia.org is back up |
[production] |
| 20:44 |
<RobH> |
all plugins updated except caching cuz i hate it and i plan to move it behind misc-web-lb |
[production] |
| 20:42 |
<RobH> |
blog updated |
[production] |
| 19:40 |
<matanya> |
git.wikimedia.org is down |
[production] |
| 19:32 |
<reedy> |
synchronized wmf-config/ |
[production] |
| 17:40 |
<RobH> |
virt0 puppet agent re-enabled and resumed normal service |
[production] |
| 17:05 |
<RobH> |
issues with change, don't re-enable puppet agent on virt0 yet |
[production] |
| 16:59 |
<RobH> |
rolling certificate changes to virt1000 and virt0. virt0 puppet agent disabled for moment |
[production] |