|
2015-07-03
§
|
| 15:31 |
<YuviPanda> |
run lvcreate -L 640G -s -n others-20150703 labstore/others on labstore1002 |
[production] |
| 15:29 |
<YuviPanda> |
running mkfs -t ext4 /dev/mapper/backup-others--20150703 on labstore2001 |
[production] |
| 15:28 |
<YuviPanda> |
run lvcreate -L 3.5T -n others-20150703 backup on labstore2001 |
[production] |
| 15:25 |
<YuviPanda> |
begin process of backing up others (all labs projects except tools) on to labstore2001 from labstore1002 |
[production] |
| 14:06 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: repool db1022 (low traffic) (duration: 00m 54s) |
[production] |
| 13:27 |
<jynus> |
Synchronized wmf-config/db-codfw.php: repool db2047 after maintenance (duration: 00m 22s) |
[production] |
| 13:27 |
<YuviPanda> |
interrupting tar |ssh | tar script and cleaning out destination again |
[production] |
| 13:17 |
<YuviPanda> |
clean out tar | ssh | tar target on labstore2001 |
[production] |
| 13:15 |
<YuviPanda> |
/dev/null filled up on labstore1002, aborting pipe of valuable user data into it. |
[production] |
| 13:13 |
<YuviPanda> |
run cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs -cpf - . | pv -L 80M -C -p -r -e -b -t -B 32M -T > /dev/null on labstore1002 |
[production] |
| 13:02 |
<YuviPanda> |
interrupt tar | ssh | tar on labstore1002 and killed dest on labstore2001 |
[production] |
| 12:43 |
<mobrovac> |
restbase deploying restbase/deploy @ 1a826a5 |
[production] |
| 12:42 |
<YuviPanda> |
interrupt tar | ssh | tar on labstore1002, clean out destination on labstore2001 |
[production] |
| 12:37 |
<YuviPanda> |
interrupted tar | ssh | tar on labstore1002 and cleaned out dest on labstore2001 |
[production] |
| 12:33 |
<YuviPanda> |
rm -rf /srv/backup-tools-20150703/* on labstore2001 |
[production] |
| 12:31 |
<mark> |
labstore2001: mount /srv/backup -o remount,ro |
[production] |
| 12:31 |
<YuviPanda> |
interrupt tar | ssh | tar on labstore1002 |
[production] |
| 12:29 |
<YuviPanda> |
cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs -cpf - . | ssh -i ~/.ssh/id_labstore root@labstore2001.codfw.wmnet "pv -L 80M -p -r -e -b -t -B 16M | tar --acls --xattrs -xpf - -C /srv/backup-tools-20150703" on labstore1002 |
[production] |
| 12:28 |
<YuviPanda> |
cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs cpf - . | ssh -i ~/.ssh/id_labstore root@labstore2001.codfw.wmnet "pv -L 80M -p -r -e -b -t -B 16M | tar --acls --xattrs xpf - -C /srv/backup-tools-20150703" on labstore1002 |
[production] |
| 12:09 |
<YuviPanda> |
running mount -o ro /dev/mapper/labstore-tools--20150703 /mnt/backup/tools-20150703/ now |
[production] |
| 11:57 |
<YuviPanda> |
run lvcreate -L 640G -s -n tools-20150703 labstore/tools on labstore1002 |
[production] |
| 11:50 |
<YuviPanda> |
running lvcreate -L 640G -s tools -n tools-20150703 labstore on labstore1002 |
[production] |
| 11:26 |
<YuviPanda> |
umount /mnt/backup/project/tools/ on labstore1002 |
[production] |
| 11:24 |
<YuviPanda> |
ran mount /dev/mapper/backup-tools--20150703 /srv/backup-tools-20150703/ on labstore2001 |
[production] |
| 11:22 |
<YuviPanda> |
mkdir /srv/backup-tools-20150703 on labstore2001 |
[production] |
| 11:13 |
<YuviPanda> |
run mkfs -t ext4 /dev/mapper/backup-tools--20150703 on labstore2001 |
[production] |
| 11:09 |
<YuviPanda> |
lvcreate -L 6TB -n tools-20150703 backup on labstore2001 |
[production] |
| 11:09 |
<jynus> |
reimports finished on dbstore2* hosts and puppet reenabled after T104471 was fixed |
[production] |
| 10:56 |
<mobrovac> |
restbase disabling puppet on restbase1005 to tweak JVM params for cassandra |
[production] |
| 10:50 |
<YuviPanda> |
started du of maps project on labstore2001 |
[production] |
| 09:36 |
<mobrovac> |
restbase restarting cassandra on rb1002 |
[production] |
| 06:19 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Fri Jul 3 06:19:02 UTC 2015 (duration 19m 1s) |
[production] |
| 02:50 |
<urandom> |
restbase rolling restart |
[production] |
| 02:49 |
<LocalisationUpdate> |
completed (1.26wmf12) at 2015-07-03 02:49:31+00:00 |
[production] |
| 02:42 |
<l10nupdate> |
Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 11m 43s) |
[production] |
| 02:06 |
<ori> |
Synchronized php-1.26wmf12/extensions/CentralAuth: I0e5f2d3b2: Updated mediawiki/core Project: mediawiki/extensions/CentralAuth 7f8da7139714dd5089dd03e8679aba25c2c89c4d (duration: 00m 15s) |
[production] |
|
2015-07-02
§
|
| 22:34 |
<legoktm> |
Synchronized php-1.26wmf12/extensions/CentralAuth/: Made use of new USE_MULTI_COMMIT flag in user merge jobs (duration: 00m 18s) |
[production] |
| 22:31 |
<legoktm> |
Synchronized php-1.26wmf12/extensions/UserMerge/: Added USE_MULTI_COMMIT flag to enable query batching (duration: 00m 26s) |
[production] |
| 21:51 |
<legoktm> |
Synchronized php-1.26wmf12/extensions/Interwiki/Interwiki_body.php: Add missing global $wgInterwikiViewOnly declaration (duration: 00m 15s) |
[production] |
| 21:37 |
<twentyafterfour> |
restarted apache2 or iridium after applying hotfix for phabricator css issue |
[production] |
| 21:22 |
<legoktm> |
Synchronized php-1.26wmf12/extensions/CentralNotice/: https://gerrit.wikimedia.org/r/222484 (duration: 00m 15s) |
[production] |
| 21:16 |
<cwdent> |
updated civicrm from 4fe0648ea9f36282731bf651a59ca1a617db6c08 to 04efc7d5c7bbb068f907125f2184692aee676123 |
[production] |
| 20:48 |
<legoktm> |
Synchronized wmf-config/CommonSettings.php: Disable global merge (duration: 00m 14s) |
[production] |
| 20:13 |
<andrewbogott> |
restarted keystone on labcontrol1001 |
[production] |
| 18:54 |
<bd808> |
Running sync-common on mw1111; fatal log showed it to be running 1.26wmf9 |
[production] |
| 18:30 |
<twentyafterfour> |
rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.26wmf12 |
[production] |
| 18:02 |
<YuviPanda> |
running exportfs -ra on labstore1002 |
[production] |
| 16:40 |
<bd808> |
Restarted logstash on logstash1001 due to OOM |
[production] |
| 16:05 |
<bblack> |
cp1065 undowntimed/repooled |
[production] |
| 16:04 |
<YuviPanda> |
clean out exports.d in labstore1002, will get regenerated. backup in /root/exports.backup |
[production] |