|
2016-09-21
§
|
| 11:16 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1086 after the ALTER table - T141951 (duration: 00m 47s) |
[production] |
| 11:07 |
<mobrovac> |
restbase deploy start of ca55669 |
[production] |
| 11:04 |
<marostegui> |
Rebuilding tables in db1082 (non pooled) - T137191 |
[production] |
| 11:03 |
<elukey> |
adding mw1197 back to serving live traffic after the reimage |
[production] |
| 10:51 |
<elukey> |
restarted varnishkafka on cp1048 (VSLQ_Dispatch: Varnish Log abandoned or overrun.) |
[production] |
| 10:45 |
<elukey> |
adding mw1196 back to serving live traffic after the reimage |
[production] |
| 10:06 |
<moritzm> |
rebooting lithium for kernel security update |
[production] |
| 09:39 |
<moritzm> |
reimaging mw1173-mw1175 to jessie |
[production] |
| 09:26 |
<marostegui> |
Stopping mysql at db1019 for a few days as it will be decommissioned - T146265 |
[production] |
| 09:19 |
<gehel> |
powercycling elastic1027 - T145404 |
[production] |
| 09:09 |
<godog> |
reimage bast3001.wikimedia.org with separate /srv |
[production] |
| 08:31 |
<marostegui> |
schema change on S7 - T141951 |
[production] |
| 08:30 |
<elukey> |
reimagining mw1196-7 to jessie |
[production] |
| 08:29 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depooling db1086 for an alter table - T141951 (duration: 00m 49s) |
[production] |
| 07:19 |
<elukey> |
Moved some hhvm logs (/var/log/hhvm) from root:adm to www-data:www-data on mw127[678] to remove cronspam (T132324) |
[production] |
| 07:18 |
<moritzm> |
reimaging mw1170-mw1172 to jessie |
[production] |
| 06:59 |
<marostegui> |
dropping tables in S1,S3,S4 - T54924 |
[production] |
| 06:21 |
<elukey> |
removing aqs100[123] from live traffic - aqs.svc.eqiad.wmnet - T144497 |
[production] |
| 06:03 |
<awight|afk> |
update SmashPig from 6651835c816ce62112cbfea96863e4244f133c17 to f308ba490635c75b6fe735e7dc1c558250202365 |
[production] |
| 03:58 |
<krinkle@tin> |
Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.js: I221cd6c2b (duration: 00m 46s) |
[production] |
| 03:56 |
<krinkle@tin> |
Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.requestIdleCallback.js: I221cd6c2b (duration: 00m 48s) |
[production] |
| 03:54 |
<krinkle@tin> |
Synchronized php-1.28.0-wmf.19/resources/src/mediawiki/mediawiki.requestIdleCallback.js: I221cd6c2b (duration: 00m 47s) |
[production] |
| 03:31 |
<awight> |
update SmashPig from 4530dc9113a83bc5492a7da6f8e9ee3a6c789cda to 6651835c816ce62112cbfea96863e4244f133c17 |
[production] |
| 03:13 |
<awight> |
update SmashPig from f1f55096ac20899fa9700aa0f80dece5b47a7fd3 to 4530dc9113a83bc5492a7da6f8e9ee3a6c789cda |
[production] |
| 02:46 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Wed Sep 21 02:46:22 UTC 2016 (duration 7m 11s) |
[production] |
| 02:45 |
<mutante> |
thumbor1002 moved nginx access logs to /srv for more space on / |
[production] |
| 02:39 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.18) (duration: 16m 28s) |
[production] |
| 02:11 |
<mutante> |
thumbor1001/1002 - moved logs from /var/log/thumbor to /srv/thumborlogs to free some space, the actual issue is in /tmp though. lots of systemd-private-* dirs with large sizes. like https://bugzilla.redhat.com/show_bug.cgi?id=1183684 ? |
[production] |
| 01:58 |
<awight> |
update SmashPig from 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c to f1f55096ac20899fa9700aa0f80dece5b47a7fd3 |
[production] |
| 01:51 |
<mutante> |
thumbor servers ran out of disk space |
[production] |
| 01:15 |
<awight> |
updated SmashPig to 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c |
[production] |
| 00:45 |
<eileen> |
jobs enabled again - update failed to run due to trigger - which didn't affect staging maybe not on - will edit & try again |
[production] |
|
2016-09-20
§
|
| 23:49 |
<eileen> |
disabled Dedupe CiviCRM contacts |
[production] |
| 23:47 |
<eileen> |
disabled Project Dedupe Major gifts contacts |
[production] |
| 23:47 |
<eileen> |
disabled CiviCRM contacts (high numbers) |
[production] |
| 23:46 |
<eileen> |
disabled Thank you mail send |
[production] |
| 23:46 |
<eileen> |
disabled Donations queue consume |
[production] |
| 23:44 |
<eileen> |
CiviCRM update from de1df9ec4fe58487eeb61cc69160f228e009f2cf to 5393a13727e8dbad05ffef9ddd44e965eb9282d1 |
[production] |
| 23:35 |
<catrope@tin> |
Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.js: Always use requestIdleCallback polyfill for batchEval (T146099) (duration: 00m 46s) |
[production] |
| 23:22 |
<catrope@tin> |
Synchronized php-1.28.0-wmf.19/extensions/Echo/: SWAT (duration: 00m 55s) |
[production] |
| 23:20 |
<catrope@tin> |
Synchronized php-1.28.0-wmf.19/extensions/TimedMediaHandler: SWAT (duration: 00m 50s) |
[production] |
| 23:17 |
<catrope@tin> |
Synchronized php-1.28.0-wmf.18/extensions/TimedMediaHandler: SWAT (duration: 00m 50s) |
[production] |
| 22:26 |
<mobrovac> |
change-prop deploying 4417255 |
[production] |
| 21:41 |
<mutante> |
powercycled mw1294 (down, frozen console) |
[production] |
| 21:26 |
<thcipriani> |
starting branch cut for wmf.20 |
[production] |
| 21:17 |
<chasemp> |
rsync initial transfer of others on labstore1001 to labstore1004 |
[production] |
| 21:08 |
<awight> |
update SmashPig from db68be988194c960aebca691d0fd8e6a6d24246a to 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c |
[production] |
| 20:26 |
<Pchelolo> |
restbase deploy ca41acd3f |
[production] |
| 20:20 |
<Pchelolo> |
restbase deploy ca41acd3f canary on restbase1007 |
[production] |
| 20:13 |
<Pchelolo> |
restbase deploy ca41acd3f to staging |
[production] |