|
2015-07-05
§
|
| 12:55 |
<mobrovac> |
restbase rolling restart of cassandra to apply the 16G heap change https://gerrit.wikimedia.org/r/222899 |
[production] |
| 11:21 |
<_joe_> |
restarted cassandra on restbase1004 (again), seemingly crashed for a bad request |
[production] |
| 11:03 |
<_joe_> |
restarting cassandra on rb1003,4 and restbase on rb1002,3 |
[production] |
| 09:43 |
<bblack> |
restarted restbase on restbase1005 |
[production] |
| 08:40 |
<_joe_> |
collecting heaps on an api appserver, mw1115, as comparison |
[production] |
| 08:29 |
<_joe_> |
restaarted HHVM on mw1059 with heap profiling enabled, collecting data (will stop this evening). |
[production] |
| 08:27 |
<bblack> |
FYI: 08:15 < grrrit-wm> (CR) BBlack: [C: 2 V: 2] filter S:RI from wm2015register T45250 [puppet] - https://gerrit.wikimedia.org/r/222879 (owner: BBlack) |
[production] |
| 08:23 |
<_joe_> |
restarted hhvm because of ooms, not apache |
[production] |
| 08:23 |
<_joe_> |
restarted apache on mw1105,mw1092,90,82,78 |
[production] |
| 07:09 |
<bblack> |
restarted cassandra on restbase1004 |
[production] |
| 07:07 |
<bblack> |
restarted cassandra + restbase on restbase1005 |
[production] |
| 07:01 |
<jynus> |
Restarted HHVM for mw1112,1028,1057,1061,1069,1070,1084,1086 |
[production] |
| 02:57 |
<LocalisationUpdate> |
completed (1.26wmf12) at 2015-07-05 02:57:28+00:00 |
[production] |
|
2015-07-04
§
|
| 23:49 |
<Krenair> |
Ran "mwscript updateSpecialPages.php labswiki --override --only=Wantedpages" on silver, completed in 0.44 seconds |
[production] |
| 23:44 |
<Krenair> |
test morebots |
[production] |
| 21:22 |
<YuviPanda> |
restarted cassandra on restbase1004 per urandom |
[production] |
| 19:15 |
<YuviPanda> |
restarted cassandra on restbase1001 |
[production] |
| 17:15 |
<_joe_> |
restarted cassandra on restbase1001 |
[production] |
| 16:12 |
<krenair> |
Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 10m 35s) |
[production] |
| 12:56 |
<krinkle> |
Synchronized php-1.26wmf12/resources/src/mediawiki/mediawiki.Title.js: I1dae1e63e47 (duration: 00m 17s) |
[production] |
| 05:01 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Sat Jul 4 05:01:43 UTC 2015 (duration 1m 42s) |
[production] |
| 03:11 |
<ori> |
Promoted Krinkle and Krenair to admin, cloudadmin on wikitech, because duh. |
[production] |
| 02:39 |
<LocalisationUpdate> |
completed (1.26wmf12) at 2015-07-04 02:39:41+00:00 |
[production] |
| 02:29 |
<l10nupdate> |
Synchronized php-1.26wmf12/cache/l10n: (no message) (duration: 09m 59s) |
[production] |
| 01:00 |
<springle> |
reload haproxy dbproxy1004 |
[production] |
|
2015-07-03
§
|
| 23:59 |
<legoktm> |
Synchronized php-1.26wmf12/extensions/Translate/: Translate+UserMerge fixes (duration: 00m 17s) |
[production] |
| 23:55 |
<legoktm> |
Synchronized php-1.26wmf12/extensions/WikiLove/: WikiLove+UserMerge fixes (duration: 00m 18s) |
[production] |
| 23:24 |
<ori> |
Synchronized w/404.php: Force 'Transfer-Encoding: Chunked' header on 404 responses (duration: 00m 31s) |
[production] |
| 22:36 |
<Krenair> |
restarted apache on silver to see if it would make https://gerrit.wikimedia.org/r/#/c/221969/ take effect for T104360. It did not. |
[production] |
| 21:46 |
<ori> |
depooled mw1152 |
[production] |
| 20:12 |
<ori> |
restarted cassandra on restbase1001 |
[production] |
| 17:28 |
<ori> |
pooled mw1152 (HHVM image scaler) for debugging. |
[production] |
| 17:05 |
<krenair> |
Synchronized php-1.26wmf12/extensions/Collection/RenderingAPI.php: https://gerrit.wikimedia.org/r/#/c/222616/ - hoping this fixes T104708 (duration: 00m 44s) |
[production] |
| 15:35 |
<YuviPanda> |
mount /dev/mapper/backup-others--20150703 /srv/backup-others-20150703/ on labstore2001 |
[production] |
| 15:34 |
<YuviPanda> |
mkdir /srv/backup-others-20150703 on labstore2001 |
[production] |
| 15:34 |
<YuviPanda> |
mkfs -t ext4 /dev/mapper/backup-others--20150703 on labstore2001 completed |
[production] |
| 15:33 |
<YuviPanda> |
run mount -o ro /dev/mapper/labstore-others--20150703 /mnt/backup/others-20150703/ on labstore1002 |
[production] |
| 15:32 |
<YuviPanda> |
run mkdir /mnt/backup/others-20150703 on labstore1002 |
[production] |
| 15:31 |
<YuviPanda> |
run lvcreate -L 640G -s -n others-20150703 labstore/others on labstore1002 |
[production] |
| 15:29 |
<YuviPanda> |
running mkfs -t ext4 /dev/mapper/backup-others--20150703 on labstore2001 |
[production] |
| 15:28 |
<YuviPanda> |
run lvcreate -L 3.5T -n others-20150703 backup on labstore2001 |
[production] |
| 15:25 |
<YuviPanda> |
begin process of backing up others (all labs projects except tools) on to labstore2001 from labstore1002 |
[production] |
| 14:06 |
<jynus> |
Synchronized wmf-config/db-eqiad.php: repool db1022 (low traffic) (duration: 00m 54s) |
[production] |
| 13:27 |
<jynus> |
Synchronized wmf-config/db-codfw.php: repool db2047 after maintenance (duration: 00m 22s) |
[production] |
| 13:27 |
<YuviPanda> |
interrupting tar |ssh | tar script and cleaning out destination again |
[production] |
| 13:17 |
<YuviPanda> |
clean out tar | ssh | tar target on labstore2001 |
[production] |
| 13:15 |
<YuviPanda> |
/dev/null filled up on labstore1002, aborting pipe of valuable user data into it. |
[production] |
| 13:13 |
<YuviPanda> |
run cd /mnt/backup/tools-20150703/ ; tar --acls --xattrs -cpf - . | pv -L 80M -C -p -r -e -b -t -B 32M -T > /dev/null on labstore1002 |
[production] |
| 13:02 |
<YuviPanda> |
interrupt tar | ssh | tar on labstore1002 and killed dest on labstore2001 |
[production] |
| 12:43 |
<mobrovac> |
restbase deploying restbase/deploy @ 1a826a5 |
[production] |