|
2015-09-01
ยง
|
| 17:28 |
<dcausse> |
freezing elasticsearch indices before applying ferm fules on master |
[production] |
| 17:23 |
<aude@tin> |
Synchronized php-1.26wmf20/extensions/Wikidata: Fix for change dispatcher (duration: 00m 20s) |
[production] |
| 16:45 |
<jynus> |
performing schema change on testwiki and metawiki |
[production] |
| 16:12 |
<robh> |
policy.wikimedia.org dns change happening now |
[production] |
| 15:59 |
<chasemp> |
ferm for elastic1003/2/1(master) |
[production] |
| 15:57 |
<krenair@tin> |
Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/235168/ (duration: 00m 13s) |
[production] |
| 15:51 |
<YuviPanda> |
stopped replicate-tools on labstore1002, and cleaned out lockdir |
[production] |
| 15:47 |
<reedy@tin> |
Synchronized php-1.26wmf20/extensions/SecurePoll/: Stop cronspam (duration: 00m 13s) |
[production] |
| 15:47 |
<mark> |
labstore1002: echo 10000 > /sys/block/md123/md/sync_speed_min |
[production] |
| 15:47 |
<valhallasw`cloud> |
git reset --hard cdnjs on tools-web-static-01 |
[tools] |
| 15:44 |
<mark> |
labstore1002: update-initramfs -k all -u |
[production] |
| 15:38 |
<mark> |
labstore1002: mdadm /dev/md/slice51 --add /dev/sd{bh,bg,bf,be,bd,bc} |
[production] |
| 15:36 |
<moritzm> |
disabled ferm in analytic1028, needs some more work on possibly dynamic mapreduce ports |
[production] |
| 15:16 |
<mark> |
labstore1002: mdadm /dev/md/slice15 --re-add /dev/sd{bb,ba,az} |
[production] |
| 15:14 |
<mark> |
labstore1002: mdadm /dev/md/slice15 --re-add /dev/sdaw |
[production] |
| 15:07 |
<mark> |
labstore1002: mdadm --zero-superblock /dev/sd{aw,bh,bg,bf,be,bd,bc,bb,ba,az}1 |
[production] |
| 15:04 |
<moritzm> |
enabled ferm in analytic1028 (initial hadoop worker) |
[production] |
| 15:04 |
<mark> |
labstore1002: mdadm --zero-superblock /dev/sdax1 && mdadm /dev/md/slice15 --re-add /dev/sdax |
[production] |
| 15:03 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/231465/ - VE for all new enwiki accounts (duration: 00m 13s) |
[production] |
| 14:58 |
<mark> |
labstore1002: mdadm /dev/md/slice15 --re-add /dev/sday |
[production] |
| 14:58 |
<mark> |
labstore1002: mdadm --zero-superblock /dev/sday1 |
[production] |
| 14:53 |
<mark> |
labstore1002: mdadm --stop /dev/md3 |
[production] |
| 14:37 |
<ebernhardson> |
reset elasticsearch cluster.routing.allocation.disk.high back to 90% |
[production] |
| 13:38 |
<krinkle@tin> |
Synchronized w/: Remove rl-test.php (duration: 00m 13s) |
[production] |
| 13:17 |
<moritzm> |
enabled ferm on db1048 |
[production] |
| 13:09 |
<moritzm> |
enabled ferm on labsdb100[467] |
[production] |
| 12:01 |
<YuviPanda> |
disable puppet on labsdb1006 |
[production] |
| 08:58 |
<moritzm> |
enabled ferm on labsdb1001 |
[production] |
| 08:58 |
<godog> |
fixup current graphite retention for metrics under "servers" hierarchy T96662 |
[production] |
| 08:51 |
<moritzm> |
enabled ferm on labsdb1002 |
[production] |
| 08:31 |
<moritzm> |
enabled ferm on labsdb1003 |
[production] |
| 08:29 |
<godog> |
repool mw1125 mw1142 after nutcracker failures |
[production] |
| 07:45 |
<jynus> |
cloning mysql data from es1010 to es1017 [ETA: 6h] |
[production] |
| 07:23 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool es1010 (duration: 00m 12s) |
[production] |
| 07:13 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool es1007, pool es1013 (duration: 00m 13s) |
[production] |
| 06:36 |
<mutante> |
uploaded survey2012 to dumps/dataset1001; ownership as it is for survey2011; - T110746 in time for midnight PST |
[production] |
| 06:23 |
<valhallasw`cloud> |
seems to have worked. SGE :( |
[tools] |
| 06:17 |
<valhallasw`cloud> |
going to restart sge_qmaster, hoping this solves the issue :/ |
[tools] |
| 06:07 |
<valhallasw`cloud> |
e.g. "queue instance "task@tools-exec-1211.eqiad.wmflabs" dropped because it is overloaded: np_load_avg=1.820000 (= 0.070000 + 0.50 * 14.000000 with nproc=4) >= 1.75" but the actual load is only 0.3?! |
[tools] |
| 06:06 |
<valhallasw`cloud> |
test job does not get submitted because all queues are overloaded?! |
[tools] |
| 06:06 |
<valhallasw`cloud> |
investigating SGE issues reported on irc/email |
[tools] |
| 05:18 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Sep 1 05:18:09 UTC 2015 (duration 18m 8s) |
[production] |
| 02:28 |
<l10nupdate@tin> |
LocalisationUpdate completed (1.26wmf20) at 2015-09-01 02:28:30+00:00 |
[production] |
| 02:25 |
<l10nupdate@tin> |
Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 06m 00s) |
[production] |
| 01:12 |
<James_F> |
Re-restarting grrrit-wm rolled back to 2f5de55ff75c3c268decfda7442dcdd62df0a42d |
[tools.lolrrit-wm] |
| 01:12 |
<James_F> |
Re-restarting grrrit-wm rolled back to 2f5de55ff75c3c268decfda7442dcdd62df0a42d |
[releng] |
| 00:54 |
<James_F> |
Restarted grrrit-wm with I7eb67e3482 as well as I48ed549dc2b. |
[releng] |
| 00:32 |
<James_F> |
Didn't work, rolled back grrrit-wm to 2f5de55ff75c3c268decfda7442dcdd62df0a42d. |
[releng] |
| 00:32 |
<James_F> |
Didn't work, r |
[releng] |
| 00:29 |
<James_F> |
Restarted grrrit-wm for I48ed549dc2b. |
[releng] |