|
2019-05-20
§
|
| 06:00 |
<elukey> |
powercycle analytics1071 - soft lockups error messages in the dmesg |
[production] |
| 05:51 |
<marostegui> |
Reload haproxy on dbproxy1010 to depool labsdb1010 |
[production] |
| 05:42 |
<marostegui> |
Reload haproxy on dbproxy1010 and dbproxy1011 to repool labsdb1009 and restore original weights |
[production] |
| 05:39 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Pool db1126 into s8, db1134 into s1 T222682 (duration: 00m 49s) |
[production] |
| 05:38 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Pool db1126 into s8, db1134 into s1 T222682 (duration: 00m 49s) |
[production] |
| 05:12 |
<marostegui> |
Stop MySQL on db2046 |
[production] |
| 05:11 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2046 (duration: 00m 50s) |
[production] |
| 05:07 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2038 (duration: 00m 49s) |
[production] |
| 05:01 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2038 (duration: 00m 55s) |
[production] |
| 02:42 |
<cdanis> |
cdanis@cp1075.eqiad.wmnet ~ % sudo -i varnish-backend-restart |
[production] |
|
2019-05-19
§
|
| 20:16 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@4febe0c]: for abstract dumps, skip any processing of pages not in main namespace (duration: 00m 03s) |
[production] |
| 20:16 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@4febe0c]: for abstract dumps, skip any processing of pages not in main namespace |
[production] |
| 17:51 |
<mobrovac> |
bootstrap restbase1025-a - T219404 |
[production] |
| 13:26 |
<ebernhardson@deploy1001> |
Synchronized wmf-config/ProductionServices.php: T223734: Depool cloudelastic100[12] (duration: 00m 49s) |
[production] |
| 12:37 |
<reedy@deploy1001> |
Synchronized wmf-config/interwiki-labs.php: update (duration: 00m 57s) |
[production] |
| 10:32 |
<reedy@deploy1001> |
Synchronized wikiversions-labs.json: T223770 (duration: 00m 48s) |
[production] |
| 10:31 |
<reedy@deploy1001> |
Synchronized dblists/all-labs.dblist: T223770 (duration: 00m 51s) |
[production] |
| 10:12 |
<mobrovac> |
bootstrap restbase1024-c - T219404 |
[production] |
| 09:59 |
<ebernhardson> |
eqiad psi elasticsearch high disk watermark to 89% to allow unallocated shard to initialize |
[production] |
| 09:56 |
<ebernhardson> |
eqiad psi elasticsearch low disk watermark to 79% to allow unallocated shard to initialize |
[production] |
| 08:13 |
<jijiki> |
varnish-backend-restart on cp1087 |
[production] |
| 06:56 |
<mobrovac> |
bootstrap restbase1024-b - T219404 |
[production] |
| 05:09 |
<marostegui> |
varnish-backend-restart on cp1081 |
[production] |
|
2019-05-18
§
|
| 23:53 |
<bblack> |
rebooting lvs1015 for interface changes |
[production] |
| 22:44 |
<bblack> |
imaging lvs1013-lvs1015 |
[production] |
| 21:01 |
<bblack> |
depooling eqiad public front edge in authdns |
[production] |
| 19:18 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.5/extensions/Collection/templates/CollectionSuggestTemplate.php: T223742 / 89bd434a21a745ec (duration: 00m 49s) |
[production] |
| 19:16 |
<mobrovac> |
bootstrap restbase1024-a - T219404 |
[production] |
| 18:50 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T222146 / 9385b2dd66 (duration: 00m 50s) |
[production] |
| 16:53 |
<mobrovac> |
bootstrap restbase1023-c - T219404 |
[production] |
| 15:57 |
<krinkle@deploy1001> |
Synchronized php-1.34.0-wmf.5/extensions/TimedMediaHandler/includes/handlers/WebMHandler/WebMHandler.php: T223445 / a9df59c59d7a30 (duration: 00m 51s) |
[production] |
| 14:59 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: whitespace is srs (duration: 00m 49s) |
[production] |
| 14:56 |
<reedy@deploy1001> |
Synchronized wmf-config/flaggedrevs.php: Copy in default config (duration: 01m 04s) |
[production] |
| 13:51 |
<urandom> |
bootstrapping restbase1023-b - T219404 |
[production] |
| 05:41 |
<mobrovac> |
bootstrap rb1023-a - T219404 |
[production] |
| 02:37 |
<urandom> |
bootstrapping restbase1022-c - T219404 |
[production] |
|
2019-05-17
§
|
| 23:55 |
<urandom> |
bootstrapping restbase1022-b - T219404 |
[production] |
| 23:11 |
<foks> |
removing one file for legal compliance |
[production] |
| 15:20 |
<hashar@deploy1001> |
Synchronized php-1.34.0-wmf.5/includes/api/ApiUpload.php: Revert "Always validate uploads over api" - T223448 (T222994 T223446) (duration: 01m 00s) |
[production] |
| 15:18 |
<hashar> |
Deploying hotfix https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/510924/ . Should restore upload of large files on commons and other wikis #T223448 (poke T22994 T223446 ) |
[production] |
| 14:51 |
<mobrovac> |
bootstrap restbase1022-a - T219404 |
[production] |
| 14:43 |
<fsero> |
reenabling puppet puppet on mcrouter hosts for T221346, checks in place is there any alert for cert expiration and mcrouter this is the source :) |
[production] |
| 14:17 |
<jynus@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1098 & db1131 after maintenance (duration: 00m 49s) |
[production] |
| 14:09 |
<fsero> |
second round of setting up cert check, disablign puppet on mcrouter hosts T221346 |
[production] |
| 12:58 |
<mobrovac> |
bootstrap restbase1021-c - T219404 |
[production] |
| 10:59 |
<mobrovac> |
bootstrap restbase1021-b - T219404 |
[production] |
| 09:27 |
<godog> |
swift remove ms-be101[345] from rings - T220590 |
[production] |
| 09:02 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1083 (duration: 00m 48s) |
[production] |
| 08:53 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1083 (duration: 00m 49s) |
[production] |
| 08:43 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: More traffic to db1083 (duration: 00m 49s) |
[production] |