|
2015-07-03
§
|
| 22:36 |
<Krenair> |
restarted apache on silver to see if it would make https://gerrit.wikimedia.org/r/#/c/221969/ take effect for T104360. It did not. |
[production] |
| 21:46 |
<ori> |
depooled mw1152 |
[production] |
| 20:12 |
<ori> |
restarted cassandra on restbase1001 |
[production] |
| 17:28 |
<ori> |
pooled mw1152 (HHVM image scaler) for debugging. |
[production] |
| 17:05 |
<krenair> |
Synchronized php-1.26wmf12/extensions/Collection/RenderingAPI.php: https://gerrit.wikimedia.org/r/#/c/222616/ - hoping this fixes T104708 (duration: 00m 44s) |
[production] |
| 15:35 |
<YuviPanda> |
mount /dev/mapper/backup-others--20150703 /srv/backup-others-20150703/ on labstore2001 |
[production] |
| 15:34 |
<YuviPanda> |
mkdir /srv/backup-others-20150703 on labstore2001 |
[production] |
| 15:34 |
<YuviPanda> |
mkfs -t ext4 /dev/mapper/backup-others--20150703 on labstore2001 completed |
[production] |
| 15:33 |
<YuviPanda> |
run mount -o ro /dev/mapper/labstore-others--20150703 /mnt/backup/others-20150703/ on labstore1002 |
[production] |
| 15:32 |
<YuviPanda> |
run mkdir /mnt/backup/others-20150703 on labstore1002 |
[production] |
| 15:31 |
<YuviPanda> |
run lvcreate -L 640G -s -n others-20150703 labstore/others on labstore1002 |
[production] |
| 15:29 |
<YuviPanda> |
running mkfs -t ext4 /dev/mapper/backup-others--20150703 on labstore2001 |
[production] |
| 15:28 |
<YuviPanda> |
run lvcreate -L 3.5T -n others-20150703 backup on labstore2001 |
[production] |
| 15:25 |
<YuviPanda> |
begin process of backing up others (all labs projects except tools) on to labstore2001 from labstore1002 |
[production] |
| 14:06 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: repool db1022 (low traffic) (duration: 00m 54s) |
[production] |
| 13:27 |
<jynus> |
Synchronized wmf-config/db-codfw.php: repool db2047 after maintenance (duration: 00m 22s) |
[production] |
| 13:27 |
<YuviPanda> |
interrupting tar |ssh | tar script and cleaning out destination again |
[production] |
| 13:17 |
<YuviPanda> |
clean out tar | ssh | tar target on labstore2001 |
[production] |
| 13:15 |
<YuviPanda> |
/dev/null filled up on labstore1002, aborting pipe of valuable user data into it. |
[production] |
| 13:13 |
<YuviPanda> |
run cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs -cpf - . | pv -L 80M -C -p -r -e -b -t -B 32M -T > /dev/null on labstore1002 |
[production] |
| 13:02 |
<YuviPanda> |
interrupt tar | ssh | tar on labstore1002 and killed dest on labstore2001 |
[production] |
| 12:43 |
<mobrovac> |
restbase deploying restbase/deploy @ 1a826a5 |
[production] |
| 12:42 |
<YuviPanda> |
interrupt tar | ssh | tar on labstore1002, clean out destination on labstore2001 |
[production] |
| 12:37 |
<YuviPanda> |
interrupted tar | ssh | tar on labstore1002 and cleaned out dest on labstore2001 |
[production] |
| 12:33 |
<YuviPanda> |
rm -rf /srv/backup-tools-20150703/* on labstore2001 |
[production] |
| 12:31 |
<mark> |
labstore2001: mount /srv/backup -o remount,ro |
[production] |
| 12:31 |
<YuviPanda> |
interrupt tar | ssh | tar on labstore1002 |
[production] |
| 12:29 |
<YuviPanda> |
cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs -cpf - . | ssh -i ~/.ssh/id_labstore root@labstore2001.codfw.wmnet "pv -L 80M -p -r -e -b -t -B 16M | tar --acls --xattrs -xpf - -C /srv/backup-tools-20150703" on labstore1002 |
[production] |
| 12:28 |
<YuviPanda> |
cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs cpf - . | ssh -i ~/.ssh/id_labstore root@labstore2001.codfw.wmnet "pv -L 80M -p -r -e -b -t -B 16M | tar --acls --xattrs xpf - -C /srv/backup-tools-20150703" on labstore1002 |
[production] |
| 12:09 |
<YuviPanda> |
running mount -o ro /dev/mapper/labstore-tools--20150703 /mnt/backup/tools-20150703/ now |
[production] |
| 11:57 |
<YuviPanda> |
run lvcreate -L 640G -s -n tools-20150703 labstore/tools on labstore1002 |
[production] |
| 11:50 |
<YuviPanda> |
running lvcreate -L 640G -s tools -n tools-20150703 labstore on labstore1002 |
[production] |
| 11:26 |
<YuviPanda> |
umount /mnt/backup/project/tools/ on labstore1002 |
[production] |
| 11:24 |
<YuviPanda> |
ran mount /dev/mapper/backup-tools--20150703 /srv/backup-tools-20150703/ on labstore2001 |
[production] |
| 11:22 |
<YuviPanda> |
mkdir /srv/backup-tools-20150703 on labstore2001 |
[production] |
| 11:13 |
<YuviPanda> |
run mkfs -t ext4 /dev/mapper/backup-tools--20150703 on labstore2001 |
[production] |
| 11:09 |
<YuviPanda> |
lvcreate -L 6TB -n tools-20150703 backup on labstore2001 |
[production] |
| 11:09 |
<jynus> |
reimports finished on dbstore2* hosts and puppet reenabled after T104471 was fixed |
[production] |
| 10:56 |
<mobrovac> |
restbase disabling puppet on restbase1005 to tweak JVM params for cassandra |
[production] |
| 10:50 |
<YuviPanda> |
started du of maps project on labstore2001 |
[production] |
| 09:36 |
<mobrovac> |
restbase restarting cassandra on rb1002 |
[production] |
| 06:19 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Jul 3 06:19:02 UTC 2015 (duration 19m 1s) |
[production] |
| 02:50 |
<urandom> |
restbase rolling restart |
[production] |
| 02:49 |
<LocalisationUpdate> |
completed (1.26wmf12) at 2015-07-03 02:49:31+00:00 |
[production] |
| 02:42 |
<l10nupdate> |
Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 11m 43s) |
[production] |
| 02:06 |
<ori> |
Synchronized php-1.26wmf12/extensions/CentralAuth: I0e5f2d3b2: Updated mediawiki/core Project: mediawiki/extensions/CentralAuth 7f8da7139714dd5089dd03e8679aba25c2c89c4d (duration: 00m 15s) |
[production] |