|
2015-05-28
§
|
| 15:00 |
<bblack> |
merged up https://gerrit.wikimedia.org/r/214345 - look here if IPv6 problems! |
[production] |
| 14:37 |
<cmjohnson1> |
powering down dataset1001 to add disk array |
[production] |
| 14:17 |
<bblack> |
deploying https://gerrit.wikimedia.org/r/214341 - keep in mind if ipv6-related issues arise! |
[production] |
| 13:50 |
<akosiaris> |
started ircecho (icinga-wm) on neon |
[production] |
| 13:46 |
<hashar> |
upgrading Jenkins git plugin from 1.4.6+wmf1 to 1.7.1 {{bug|T100655}} and restarting Jenkins |
[production] |
| 13:25 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: repool pc1003 (not to confuse with db1003) after warmup (duration: 00m 15s) |
[production] |
| 13:11 |
<akosiaris> |
killed ircecho service on neon |
[production] |
| 09:48 |
<_joe_> |
depooling the HHVM appserver. 503s reduced slightly but still non-irrelevant |
[production] |
| 09:37 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: Depool pc1003 (duration: 00m 15s) |
[production] |
| 09:35 |
<_joe_> |
pooling mw1152 into the imagescalers pool after fixes made in Lyon |
[production] |
| 06:11 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu May 28 06:09:56 UTC 2015 (duration 9m 55s) |
[production] |
| 04:22 |
<springle> |
reload dbstore1002 s7 |
[production] |
| 02:41 |
<LocalisationUpdate> |
completed (1.26wmf6) at 2015-05-28 02:40:00+00:00 |
[production] |
| 02:36 |
<l10nupdate> |
Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 46s) |
[production] |
| 02:20 |
<springle> |
set global read_only=0 on pc1001 pc1002. this config broke in the recent upgrade |
[production] |
| 00:59 |
<legoktm> |
Synchronized php-1.26wmf8/resources/: Revert "Convert mediawiki.toc and mediawiki.user to using mw.cookie" (duration: 00m 17s) |
[production] |
| 00:58 |
<legoktm> |
Synchronized php-1.26wmf7/resources/: Revert "Convert mediawiki.toc and mediawiki.user to using mw.cookie" (duration: 00m 13s) |
[production] |
| 00:07 |
<twentyafterfour> |
Synchronized rpc/RunJobs.php: deploy I98b8a4ddbcdd58d1f2f23e4b1bf154f10b6b279e (duration: 00m 17s) |
[production] |
|
2015-05-27
§
|
| 23:46 |
<awight> |
updated payments from 858b87319daa3d66f62eb32e08cefc6b061748d1 to aa66797553fbcfb63f7cf29abccc44d060b65db0 |
[production] |
| 23:31 |
<twentyafterfour> |
Finished scap: scap, now with 10% less fail (duration: 22m 07s) |
[production] |
| 23:26 |
<awight> |
payments rolled back to 858b87319daa3d66f62eb32e08cefc6b061748d1 |
[production] |
| 23:24 |
<awight> |
updated payments from 858b87319daa3d66f62eb32e08cefc6b061748d1 to aa66797553fbcfb63f7cf29abccc44d060b65db0 |
[production] |
| 23:09 |
<twentyafterfour> |
Started scap: scap, now with 10% less fail |
[production] |
| 22:57 |
<ori> |
rebuilt wikiversions.cdb and synchronized wikiversions files: (no message) |
[production] |
| 21:49 |
<mutante> |
restarted hhvm on mw1250,mw1254,mw1256 |
[production] |
| 21:47 |
<mutante> |
restarted hhvm on mw1017,mw1243,mw1244 |
[production] |
| 21:42 |
<bblack> |
restarting hhvm everywhere on 30s intervals between hosts |
[production] |
| 21:10 |
<twentyafterfour> |
Synchronized php-1.26wmf8: Fix ConfirmEdit fatal Change-Id: I22353669a85391c3d9760a5253cac1263e895cf9 (duration: 01m 08s) |
[production] |
| 20:47 |
<twentyafterfour> |
Purged l10n cache for 1.26wmf6 |
[production] |
| 20:45 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.26wmf8 |
[production] |
| 20:41 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.26wmf7 |
[production] |
| 20:36 |
<twentyafterfour> |
Finished scap: testwiki to php-1.26wmf8 and rebuild l10n cache (duration: 67m 53s) |
[production] |
| 19:28 |
<twentyafterfour> |
Started scap: testwiki to php-1.26wmf8 and rebuild l10n cache |
[production] |
| 19:18 |
<twentyafterfour> |
Started scap: testwiki to php-1.26wmf8 and rebuild l10n cache |
[production] |
| 18:12 |
<moritzm> |
Uploaded gridengine_6.2u5-4+wmf2 for precise-wikimedia to apt.wikimedia.org |
[production] |
| 17:55 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: repool pc1002 (duration: 00m 13s) |
[production] |
| 17:42 |
<paravoid> |
rebooting asw-d2-eqiad |
[production] |
| 17:41 |
<ottomata> |
initiating controlled shutdown of kafka broker analytics1018 in anticipation of switch reboot |
[production] |
| 15:33 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: depool pc1002 (duration: 00m 13s) |
[production] |
| 15:02 |
<cmjohnson1> |
powering down cp1069 to relocate within the same rack |
[production] |
| 14:47 |
<cmjohnson1> |
powering down cp1070 to relocate within the same rack |
[production] |
| 13:30 |
<hashar> |
All Jenkins slaves are disconnected due to some ssh error. CI is down. |
[production] |
| 13:27 |
<hashar> |
restarting Jenkins for java upgrade |
[production] |
| 13:13 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: repool pc1001 (duration: 00m 13s) |
[production] |
| 11:16 |
<akosiaris> |
rebooting ganeti100{1..4} for bridge networking configuration |
[production] |
| 09:59 |
<paravoid> |
powercycling ms-be1001; dead, console unresponsive |
[production] |
| 06:35 |
<springle> |
clone dbstore2001 data to dbstore2002 |
[production] |
| 05:48 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed May 27 05:47:25 UTC 2015 (duration 47m 24s) |
[production] |
| 02:53 |
<LocalisationUpdate> |
completed (1.26wmf7) at 2015-05-27 02:52:25+00:00 |
[production] |
| 02:48 |
<l10nupdate> |
Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 52s) |
[production] |