|
2017-11-16
ยง
|
| 20:51 |
<apergos> |
one more catchup rsync from ms1001 to labstore1006 kicking off |
[production] |
| 20:15 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers - third try (duration: 00m 49s) |
[production] |
| 20:11 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers - second try (duration: 00m 49s) |
[production] |
| 20:10 |
<demon@tin> |
Unlocked for deployment [operations/mediawiki-config]: No deploys, recovering from downtime (more narrow locking) (duration: 06m 49s) |
[production] |
| 20:03 |
<demon@tin> |
Locking from deployment [operations/mediawiki-config]: No deploys, recovering from downtime (more narrow locking) (planned duration: 360m 00s) |
[production] |
| 20:02 |
<twentyafterfour> |
deploy a9613b44b70cae79aefdd3422bba627c6516e1a0 to hotfix T180706 |
[production] |
| 19:56 |
<demon@tin> |
Unlocked for deployment [ALL REPOSITORIES]: No deploys, recovering from downtime (duration: 50m 28s) |
[production] |
| 19:54 |
<ayounsi@tin> |
Finished deploy [netbox/deploy@19f4f65]: (no justification provided) (duration: 00m 04s) |
[production] |
| 19:54 |
<ayounsi@tin> |
Started deploy [netbox/deploy@19f4f65]: (no justification provided) |
[production] |
| 19:49 |
<akosiaris> |
schedule extra downtime for s5 slaves |
[production] |
| 19:46 |
<urandom> |
Bootstapping Cassandra restbase2006-a (T179422) |
[production] |
| 19:13 |
<jynus> |
reset slave all on db1070 @ db1063-bin.001382:234464548 |
[production] |
| 19:05 |
<demon@tin> |
Locking from deployment [ALL REPOSITORIES]: No deploys, recovering from downtime (planned duration: 360m 00s) |
[production] |
| 19:05 |
<demon@tin> |
Unlocked for deployment [ALL REPOSITORIES]: Dealing with outage, no deploys for now (duration: 01m 22s) |
[production] |
| 19:03 |
<demon@tin> |
Locking from deployment [ALL REPOSITORIES]: Dealing with outage, no deploys for now (planned duration: 60m 00s) |
[production] |
| 18:48 |
<marostegui> |
dewikipedia and wikidata currently back to writable |
[production] |
| 18:44 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Pool only 'good' servers (duration: 00m 48s) |
[production] |
| 18:41 |
<mutante> |
dewikipedia and wikidata currently back to readonly-mode while wikidata is being worked on |
[production] |
| 18:41 |
<marostegui> |
Set s5 master read_only |
[production] |
| 18:25 |
<akosiaris> |
set mw2251 to inactive. T180724 |
[production] |
| 18:24 |
<akosiaris@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: name=mw2251.codfw.wmnet |
[production] |
| 17:57 |
<gehel> |
silencing wdqs |
[production] |
| 17:55 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Failover db1063 to db1070 (duration: 00m 46s) |
[production] |
| 17:44 |
<XioNoX> |
merging netbox CR |
[production] |
| 17:41 |
<bblack> |
disable port ge-5/0/39 on asw-c-eqiad (db1063) |
[production] |
| 17:40 |
<urandom> |
Decommissioning Cassandra, restbase1014-b.eqiad.wmnet (T179422) |
[production] |
| 17:36 |
<jynus> |
stopping slave on db1070 |
[production] |
| 17:28 |
<urandom> |
Converting 'enwiki parsoid' to size-tiered compaction (T179422) |
[production] |
| 17:19 |
<demon@tin> |
Finished scap: consistency (duration: 25m 12s) |
[production] |
| 17:18 |
<bblack> |
Temporary GeoDNS routing changes (eqsin traffic simulation using ulsfo) - https://gerrit.wikimedia.org/r/#/c/391357/ - expecting ~24h, West Asia latencies will probably increase, spike in cache misses, etc... |
[production] |
| 17:18 |
<urandom> |
Converting 'wikipedia parsoid' to size-tiered compaction (T179422) |
[production] |
| 17:16 |
<godog> |
reimage restbase2006 - T179422 |
[production] |
| 17:15 |
<urandom> |
Converting 'commons mobile' to size-tiered compaction (T179422) |
[production] |
| 17:05 |
<urandom> |
Converting 'others mobile' to size-tiered compaction (T179422) |
[production] |
| 16:54 |
<demon@tin> |
Started scap: consistency |
[production] |
| 16:54 |
<godog> |
reimage restbase2006 - T179422 |
[production] |
| 16:48 |
<jynus> |
stop and restart db1071 for upgrade and reconfiguration |
[production] |
| 16:48 |
<herron> |
beginning gradual cutover of codfw mw systems to puppet 4 master puppetmaster2001 |
[production] |
| 16:48 |
<godog> |
upgrade hpsa firmware to 6.06 on restbase2006 - T141756 |
[production] |
| 16:43 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Restore db1034 and db1079 original weight (duration: 00m 48s) |
[production] |
| 16:29 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: mariadb: Depool db1071, pool db1104 as api (duration: 00m 49s) |
[production] |
| 16:21 |
<moritzm> |
restarting apache on labs puppet masters to pick up openssl updates |
[production] |
| 16:06 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 48s) |
[production] |
| 15:47 |
<moritzm> |
rebooting ores1* for update to 4.9.51 |
[production] |
| 15:41 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 49s) |
[production] |
| 15:33 |
<moritzm> |
rebooting seaborgium (slapd) for update to 4.9.51 |
[production] |
| 15:22 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2038 - T178359 (duration: 00m 49s) |
[production] |
| 15:16 |
<milimetric@tin> |
Finished deploy [analytics/refinery@4ef15d3]: Mainly deploying the interlanguage navigation dataset (duration: 14m 29s) |
[production] |
| 15:13 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1034 weight (duration: 00m 48s) |
[production] |
| 15:10 |
<godog> |
upgrade prometheus-redis-exporter to 0.13-1 - T148637 |
[production] |