|
2014-08-02
§
|
| 15:28 |
<godog> |
reboot ms-be1008, stuck on xfs errors and most processes in D state |
[production] |
| 14:10 |
<Krinkle> |
Restarting Zuul |
[production] |
| 14:09 |
<hashar> |
Jenkins / Zuul stuck {{bug|69045}} |
[production] |
| 14:00 |
<Krinkle> |
Restarting Jenkins in attempt to unstuck the clogged Zuul pipeline for gallium |
[production] |
| 08:53 |
<swtaarrs> |
rebuilt and restarted hhvm on deployment-mediawiki02 with potential fix |
[releng] |
| 05:17 |
<swtaarrs> |
restarted hhvm on deployment-mediawiki0{1,2} to unwedge them |
[releng] |
| 04:21 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Aug 2 04:20:45 UTC 2014 (duration 20m 44s) |
[production] |
| 02:33 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-02 02:32:36+00:00 |
[production] |
| 02:21 |
<LocalisationUpdate> |
completed (1.24wmf15) at 2014-08-02 02:20:02+00:00 |
[production] |
| 01:49 |
<springle> |
Synchronized wmf-config/db-eqiad.php: depool db1018, replag (duration: 00m 06s) |
[production] |
| 00:43 |
<Krinkle> |
Restarting Jenkins on gallium because the pipeline is clogged |
[production] |
|
2014-08-01
§
|
| 20:25 |
<andrewbogott> |
shorted the logrotate interval on vanadium; disk space critical should resolve soon |
[production] |
| 18:11 |
<csteipp> |
Synchronized php-1.24wmf16/extensions/CentralAuth: Fix for bug 69007 - logins failing for old style hashes (duration: 00m 06s) |
[production] |
| 17:32 |
<AaronSchulz> |
Restarted maintenance/populateBacklinkNamespace.php on enwiki |
[production] |
| 17:31 |
<aaron> |
Synchronized php-1.24wmf15/maintenance/populateBacklinkNamespace.php: e1cea29342f964cd9a720310185b09ca41eb1a4a (duration: 00m 04s) |
[production] |
| 17:16 |
<akosiaris> |
upgraded etherpad-lite on zirconium to 1.4.0-2. Uploaded etherpad-lite_1.4.0-2 on apt.wikimedia.org |
[production] |
| 17:12 |
<aaron> |
Synchronized php-1.24wmf15/includes: d218d86dff90a5f0110353c492bd2e8ddaf35497 (duration: 00m 08s) |
[production] |
| 17:09 |
<aaron> |
Synchronized php-1.24wmf16/includes: f1a8ff7f802b57cc9f452d47c4c762a185ed93c2 (duration: 00m 06s) |
[production] |
| 15:48 |
<reedy> |
Synchronized php-1.24wmf16/includes/specials/SpecialRecentchangeslinked.php: (no message) (duration: 00m 14s) |
[production] |
| 15:03 |
<bd808> |
Updated cherry-pick of Iceb8f43 |
[releng] |
| 15:02 |
<bd808> |
Cleaned up puppet repo on deployment-salt; merge conflicts with local Ia463120 hack; reapplied depool of deployment-mediawiki01 |
[releng] |
| 14:50 |
<bd808> |
Restarted stuck hhvm on deployment-mediawiki02; apache had 89 children waiting for a response |
[releng] |
| 13:27 |
<godog> |
changed inplace bt-hhvm on deployment-mediawiki01/02 to also copy the binary |
[releng] |
| 12:07 |
<apergos> |
powercycled dataset1001, inaccessible via mgmt console, only visible message was 'mnt.nfs failed' |
[production] |
| 09:10 |
<_joe_> |
apache mediawiki::web train finished its run. re-enabling puppet on all appservers |
[production] |
| 07:47 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Aug 1 07:46:04 UTC 2014 (duration 46m 3s) |
[production] |
| 07:24 |
<_joe_> |
stopping puppet on appservers to deploy a potentially dangerous case |
[production] |
| 05:32 |
<ori> |
depooled deployment-mediawiki02 to investigate HHVM lock-up by cherry-picking I7df8c5310 on beta. |
[releng] |
| 05:16 |
<springle> |
Synchronized wmf-config/db-eqiad.php: Move enwiki api traffic away from lagging slaves (duration: 00m 07s) |
[production] |
| 03:12 |
<LocalisationUpdate> |
completed (1.24wmf16) at 2014-08-01 03:11:14+00:00 |
[production] |
| 02:40 |
<LocalisationUpdate> |
completed (1.24wmf15) at 2014-08-01 02:38:56+00:00 |
[production] |
| 00:52 |
<catrope> |
Synchronized php-1.24wmf16/extensions/VisualEditor/lib/ve/modules/ve/ui/inspectors/ve.ui.CommentInspector.js: Fix typo in class name (duration: 00m 10s) |
[production] |
| 00:40 |
<ori> |
disabled puppet on deployment-mediawiki{01,02} and enabled verbose apache logging |
[releng] |
|
2014-07-31
§
|
| 23:23 |
<mwalker> |
Synchronized php-1.24wmf16: Updating core and Flow for SWAT (duration: 00m 53s) |
[production] |
| 23:05 |
<mwalker> |
Synchronized wmf-config: Updating configuration for {{gerrit|150145}} (duration: 00m 05s) |
[production] |
| 22:41 |
<bd808> |
Restarted hhvm on -mediawiki{01,02}. Brett looked at 01 before I did and said "it's the same as before" |
[releng] |
| 21:17 |
<RobH> |
blog.wikimedia.org cname changed to migrate over to wp servers |
[production] |
| 20:22 |
<AaronSchulz> |
Started populateBacklinkNamespace.php on s1-s3,s5-s7 (commons already running) |
[production] |
| 20:13 |
<cscott> |
updated OCG to version d2919c59eb09e09fc87777696411a070620aef45 |
[production] |
| 20:09 |
<cscott> |
updated OCG to version d2919c59eb09e09fc87777696411a070620aef45 |
[releng] |
| 19:59 |
<hashar> |
Granted sudo right to cscott (under NDA). Will let him reboot OCG service |
[releng] |
| 19:40 |
<hashar> |
Jenkins build its first hhvm extension \\O/ https://integration.wikimedia.org/ci/job/php-FastStringSearch-hhvm-build/2/console |
[production] |
| 19:24 |
<Coren_away> |
labsdb1005 had to blow away the postgres slave: was using all the space on / because DB at wrong spot (should have been /srv/postgres) |
[production] |
| 18:58 |
<ori> |
re-enabled puppet on deployment-mediawiki{01,02} |
[releng] |