|
2015-11-29
§
|
| 21:25 |
<jynus> |
importing user.user_touched (s7) from dbstore1002 to sanitarium. s7 lag on labs replicas will be higher for some minutes. |
[production] |
| 20:51 |
<jynus> |
importing user.user_touched (s6) from dbstore1002 to sanitarium. s6 lag on labs replicas will be higher for some minutes. |
[production] |
| 20:28 |
<jynus> |
importing user.user_touched (s5) from dbstore1002 to sanitarium. s5 lag on labs replicas will be higher for some minutes. |
[production] |
| 19:51 |
<jynus> |
importing user.user_touched (s4) from dbstore1002 to sanitarium. s4 lab will be affected for some minutes. |
[production] |
| 04:50 |
<gwicke> |
restarted cassandra on restbase1009 to avoid it running out of disk space; had large compaction (~2TB) at 80% and only 64G disk space left |
[production] |
| 03:01 |
<YuviPanda> |
run chown -R l10nupdate: /var/lib/l10nupdate/mediawiki for Reedy on tin |
[production] |
| 02:28 |
<Reedy> |
l10nupdate failed because some git objects owned by 997:l10nupdate |
[production] |
| 02:00 |
<l10nupdate@tin> |
LocalisationUpdate failed: git pull of core failed |
[production] |
|
2015-11-25
§
|
| 23:21 |
<krenair@tin> |
Synchronized private/README_BEFORE_MODIFYING_ANYTHING: 334ca105e92aaf7046e244ff39189f3823d31a7d (duration: 00m 32s) |
[production] |
| 22:14 |
<demon@tin> |
Finished scap: new MW release, swapping extdist config + msgs (duration: 23m 59s) |
[production] |
| 21:50 |
<demon@tin> |
Started scap: new MW release, swapping extdist config + msgs |
[production] |
| 20:31 |
<gwicke> |
running `nodetool cleanup` on restbase1007 to make sure that we don't have extra sstables from the 1001 decommision taking up space |
[production] |
| 19:13 |
<mobrovac> |
restbase deploy end of 74662c |
[production] |
| 19:12 |
<moritzm> |
installed django security updates on stat* and graphite hosts |
[production] |
| 18:47 |
<robh> |
removing cr2-ulsfo:xe-1/2/0, Patch ID 1062 as T118171 cancels that link |
[production] |
| 18:46 |
<mobrovac> |
restbase deploy start of 74662c |
[production] |
| 18:19 |
<bd808@tin> |
Synchronized README: Testing l10nupdate uid fix for T119165 (duration: 00m 28s) |
[production] |
| 18:04 |
<andrewbogott> |
restored pdns-recursor on holmium, again |
[production] |
| 18:00 |
<mobrovac> |
restbase canary deploy to restbase1001 of 74662c6 |
[production] |
| 17:44 |
<andrewbogott> |
killing pdns-recursor on holmium |
[production] |
| 16:53 |
<andrewbogott> |
killing pdns-recursor on holmium |
[production] |
| 16:16 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1044 after maintenance (duration: 00m 47s) |
[production] |
| 15:53 |
<godog> |
ban grafana kafka dashboard temporarily from graphite |
[production] |
| 14:57 |
<godog> |
bounce uwsgi on graphite1001 |
[production] |
| 14:52 |
<godog> |
stop puppet on restbase1* / restbase2* before https://gerrit.wikimedia.org/r/#/c/254372/ |
[production] |
| 13:09 |
<hashar@tin> |
Synchronized php-1.27.0-wmf.7/Rakefile: Added Rakefile https://gerrit.wikimedia.org/r/#/c/254423/ (duration: 00m 28s) |
[production] |
| 11:19 |
<jynus> |
applying ferm and p_s to db1044 (depooled) |
[production] |
| 10:57 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1044 for a brief maintenance (duration: 00m 28s) |
[production] |
| 08:08 |
<ori@tin> |
Synchronized wmf-config/jobqueue-eqiad.php: I9bea66df: Update job queue config to use 2 rdb1007 instances (duration: 00m 28s) |
[production] |
| 08:06 |
<akosiaris> |
update puppet compilers fact database |
[production] |
| 03:44 |
<TimStarling> |
on palladium ran puppet post-merge hook manually after puppet-merge failed with "error: Ref refs/remotes/origin/production is at 380706c5bd2f47d140ffa183dd22fe920927af7e but expected 897486120d645f1c30dcf68bfb3198e2f39ec639" |
[production] |
| 03:35 |
<krinkle@tin> |
Synchronized portals/prod/wikipedia.org/index.html: Fix Uncaught TypeError: innerHTML() is not a function (duration: 00m 28s) |
[production] |
| 03:12 |
<krinkle@tin> |
Synchronized wmf-config/CommonSettings.php: I4e21eda0f3 (duration: 01m 06s) |
[production] |
| 02:40 |
<krenair@tin> |
Synchronized php-1.27.0-wmf.7/cache/l10n: l10nupdate for 1.27.0-wmf.7 (duration: 06m 50s) |
[production] |
| 02:00 |
<l10nupdate@tin> |
LocalisationUpdate failed: git pull of core failed |
[production] |
| 01:31 |
<krinkle@tin> |
Synchronized php-1.27.0-wmf.7/StartProfiler.php: create symlink (duration: 01m 10s) |
[production] |
| 01:03 |
<mutante> |
re-armed keyholders on mira |
[production] |
| 00:46 |
<mutante> |
starting redis on rdb1007 (it died and was status: dead) |
[production] |
| 00:40 |
<mutante> |
tin: fixing l10nupdate UID (997->10002), file ownership |
[production] |
| 00:37 |
<ori> |
Migrating rdb1007 from redis::legacy to redis::instance; will involve a service restart. |
[production] |
| 00:20 |
<mutante> |
rebooting mira |
[production] |
| 00:14 |
<krenair@tin> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/255276/ (duration: 00m 28s) |
[production] |