|
2017-02-28
§
|
| 22:29 |
<mutante> |
(T157675) - delete salt keys - [neodymium:~] $ for mcnode in $(seq 2001 2016); do sudo salt-key -d mc${mcnode}.codfw.wmnet; done |
[production] |
| 22:26 |
<mutante> |
(T157675) - revoke puppet certs, deactivate nodes, rm from icinga. [puppetmaster1001:~] $ for mcnode in $(seq 2001 2016); do puppet node clean mc${mcnode}.codfw.wmnet && puppet node deactivate mc${mcnode}.codfw.wmnet ; done |
[production] |
| 21:58 |
<awight> |
update payments from 2a0c3b2148805fe258f124a700e7f3c34f306783 to 66d812517acc47a8a7e07b5fe269950b21c86be3 |
[production] |
| 21:51 |
<eileen1> |
update CiviCRM from a2875c55a5760c375b133c71a7ec22846b4d68c3 to 04b49b0393295610c530dc29cb0218e6755ef24f |
[production] |
| 21:44 |
<urandom> |
Updating RESTBase mobileapps tables (all remaining) to use time-windowed compaction |
[production] |
| 21:40 |
<maxsem@tin> |
Finished deploy [kartotherian/deploy@81db48c]: Second attempt at 81db48c6db1fdfea81a01796ccbb7cfeccc43052 (duration: 06m 39s) |
[production] |
| 21:34 |
<maxsem@tin> |
Started deploy [kartotherian/deploy@81db48c]: Second attempt at 81db48c6db1fdfea81a01796ccbb7cfeccc43052 |
[production] |
| 21:23 |
<MaxSem> |
Completely disabled kartotherian on maps-test2004, it just logs errors |
[production] |
| 21:05 |
<_joe_> |
manually installing nodejs on wasat T156922 |
[production] |
| 20:50 |
<maxsem@tin> |
Synchronized wmf-config/wikitech.php: https://gerrit.wikimedia.org/r/#/c/340357/2 (duration: 00m 40s) |
[production] |
| 20:33 |
<urandom> |
Updating RESTBase mobileapps tables (phase0) to use time-windowed compaction |
[production] |
| 20:30 |
<demon@tin> |
Synchronized wmf-config/wikitech.php: no moar forms on wikitech (duration: 00m 39s) |
[production] |
| 20:03 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.14 |
[production] |
| 19:34 |
<demon@tin> |
Synchronized php-1.29.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.geoFeatures.js: Roan made me do it (duration: 00m 39s) |
[production] |
| 19:26 |
<demon@tin> |
Finished scap: testwiki to wmf.14 + l10n bootstrap (duration: 55m 14s) |
[production] |
| 19:04 |
<urandom> |
Updating RESTBase mobileapps tables (wikimedia) to uses time-windowed compaction |
[production] |
| 18:31 |
<demon@tin> |
Started scap: testwiki to wmf.14 + l10n bootstrap |
[production] |
| 17:43 |
<urandom> |
Updating RESTBase mobileapps tables (wikipedia) to uses time-windowed compaction |
[production] |
| 17:11 |
<elukey> |
Analytics Hadoop cluster upgraded to CDH 5.10 |
[production] |
| 17:09 |
<jynus> |
disabling replication lag alerts on db1026 (depooled) |
[production] |
| 17:05 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=wdqs1001.eqiad.wmnet |
[production] |
| 17:04 |
<gehel> |
restarting blazegraph on wdqs1001 - T159245 |
[production] |
| 15:47 |
<jynus> |
running alter table on db1056 T147747 |
[production] |
| 15:30 |
<gehel> |
depooling wdqs1001 due to instability |
[production] |
| 15:29 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=wdqs1001.eqiad.wmnet |
[production] |
| 15:23 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1056 for maintenance (duration: 00m 40s) |
[production] |
| 14:51 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=wdqs1003.eqiad.wmnet |
[production] |
| 14:46 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1053 after maintenance (duration: 00m 39s) |
[production] |
| 14:35 |
<elukey> |
start the Analytics Hadoop cluster upgrade (https://etherpad.wikimedia.org/p/analytics-cdh5.10) |
[production] |
| 14:32 |
<marostegui> |
run pt-table-checksum on eowiki (s2) - T154485 |
[production] |
| 14:08 |
<phuedx@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: Make Page Previews use RESTBase on Beta Cluster (duration: 00m 42s) |
[production] |
| 14:02 |
<reedy@tin> |
Synchronized php-1.29.0-wmf.13/extensions/Dashiki/extension.json: Register JsonConfigModels (duration: 00m 42s) |
[production] |
| 13:57 |
<jynus> |
running alter table on db1036 T147747 |
[production] |
| 13:23 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1055 after maintenance (duration: 00m 39s) |
[production] |
| 13:03 |
<Reedy> |
ran namespaceDupes on meta to fix some Config pages |
[production] |
| 12:48 |
<_joe_> |
flushed memcached in codfw, restarting hhvm on appserver to flush APC in order to test warmup script |
[production] |
| 11:47 |
<gehel> |
restarting wdqs-blazegraph on wdqs1003 |
[production] |
| 11:40 |
<gehel> |
depooling wdqs1003 for investigation (high 5xx rate) |
[production] |
| 11:40 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=wdqs1003.eqiad.wmnet |
[production] |
| 11:22 |
<jynus> |
running alter table on db1053 T147747 |
[production] |
| 11:18 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1053 for maintenance (duration: 00m 40s) |
[production] |
| 10:56 |
<elukey> |
restart zookeeper on conf1002 |
[production] |
| 10:53 |
<marostegui> |
run pt-table-checksum on enwiktionary (s2) - T154485 |
[production] |
| 10:35 |
<elukey> |
restar zookeeper on conf1003 |
[production] |
| 10:28 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1026 for maintenance (duration: 00m 39s) |
[production] |
| 10:23 |
<marostegui> |
run pt-table-checksum on enwikiquote (s2) - T154485 |
[production] |
| 10:09 |
<marostegui> |
Deploy alter table s2 on all wikis for table user_groups - T155605 |
[production] |
| 10:00 |
<elukey> |
restart zookeeper on conf1001 |
[production] |