|
2017-11-27
§
|
| 15:52 |
<herron> |
beginning canary cutover/deployment of codfw prometheus servers to codfw puppet 4 puppetmasters |
[production] |
| 15:19 |
<chasemp> |
disable puppet across cloud things for cleanup |
[production] |
| 15:11 |
<herron> |
beginning canary cutover/deployment of codfw elasticsearch servers to codfw puppet 4 puppetmasters |
[production] |
| 14:22 |
<zeljkof> |
EU SWAT finished |
[production] |
| 14:19 |
<zfilipin@tin> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:393442|IP cap lift for Semaine contributive 2017-2018 (T181360)]] (duration: 00m 45s) |
[production] |
| 14:10 |
<addshore@tin> |
Synchronized php-1.31.0-wmf.8/extensions/AdvancedSearch: SWAT AdvancedSearch T181175 T181222, adjust links in beta section & fix usability of mobile search (duration: 00m 46s) |
[production] |
| 13:31 |
<moritzm> |
installing nspr security updates on trusty |
[production] |
| 13:22 |
<elukey> |
remove eventlogging replication support (log database) from dbstore1002 - T156844 |
[production] |
| 13:17 |
<moritzm> |
updating nginx on francium |
[production] |
| 13:01 |
<Pchelolo> |
stop cpjobqueue in eqiad for backlog accumulation |
[production] |
| 12:44 |
<Pchelolo> |
started parsoid linter script to generate load on cpjobqueue (python3 parsoid_reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://commons.wikimedia.org/w/api.php) |
[production] |
| 12:44 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: [[gerrit:393587]] BETA wmgMonologChannels FileImporter => debug (duration: 01m 01s) |
[production] |
| 12:43 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=inactive; selector: mw1276.eqiad.wmnet |
[production] |
| 12:35 |
<moritzm> |
powercycling mw1276 |
[production] |
| 12:21 |
<addshore@tin> |
Synchronized wmf-config/CommonSettings-labs.php: [[gerrit:393582]] LABS ONLY Actually call wfLoadExtension for FileExporter & Importer on beta BETA (T181383) (duration: 00m 44s) |
[production] |
| 12:14 |
<marostegui> |
Stop replication on db1109 to test table rename for s5/s8 failover |
[production] |
| 12:09 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@47d27dc]: Enable keep-alive T181007 (duration: 01m 03s) |
[production] |
| 12:08 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@47d27dc]: Enable keep-alive T181007 |
[production] |
| 11:47 |
<addshore@tin> |
Synchronized wmf-config/CommonSettings-labs.php: [[gerrit:393577]] LABS ONLY Enable FileImporter & FileExporter on BETA PT2/2 (T181383) (duration: 00m 45s) |
[production] |
| 11:46 |
<addshore@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: [[gerrit:393577]] LABS ONLY Enable FileImporter & FileExporter on BETA PT1/2 (T181383) (duration: 00m 45s) |
[production] |
| 11:44 |
<addshore@tin> |
Synchronized wmf-config/extension-list-labs: [[gerrit:393576]] BETA ONLY Add FileExporter & FileImporter to extension-list-labs (duration: 00m 45s) |
[production] |
| 11:27 |
<hashar> |
contint1001 enable Icinga "Disks space" notification again. It is no more complaing about Docker partitions | ping mutante | T178454 |
[production] |
| 11:17 |
<godog> |
bootstrap restbase1007-a - T179422 |
[production] |
| 11:05 |
<jdrewniak@tin> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:393559|Bumping portals to master (T128546)]] (duration: 00m 45s) |
[production] |
| 11:05 |
<jdrewniak@tin> |
Synchronized portals/prod/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:393559|Bumping portals to master (T128546)]] (duration: 00m 46s) |
[production] |
| 10:53 |
<moritzm> |
installing postgresql-common security updates |
[production] |
| 10:08 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@e35aa05]: Revert using keep-alive (duration: 00m 22s) |
[production] |
| 10:08 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@e35aa05]: Revert using keep-alive |
[production] |
| 09:42 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@b570d4e]: Make http agent use keep-alive (duration: 00m 48s) |
[production] |
| 09:41 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@b570d4e]: Make http agent use keep-alive |
[production] |
| 09:14 |
<godog> |
reimage restbase1007 - T179422 |
[production] |
| 08:55 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Fully pool db1097:3314 - T178359 (duration: 00m 43s) |
[production] |
| 08:40 |
<moritzm> |
installing openjdk security updates on hadoop, druid and kafka clusters |
[production] |
| 08:27 |
<marostegui> |
Deploy schema change on dbstore1002 and dbstore1001 - T174569 |
[production] |
| 08:22 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1097:3314 weight - T178359 (duration: 00m 45s) |
[production] |
| 07:20 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1097:3314 weight - T178359 (duration: 00m 45s) |
[production] |
| 07:10 |
<marostegui> |
Stop MySQL on db1021 as it will be decommissioned - T181378 |
[production] |
| 06:53 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 44s) |
[production] |
| 06:52 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 45s) |
[production] |
| 06:27 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Pool db1097:3314 with low weight - T178359 (duration: 00m 46s) |
[production] |
| 02:22 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.8) (duration: 05m 44s) |
[production] |
|
2017-11-25
§
|
| 19:31 |
<marostegui> |
Set 32:3 disk to offline on db1051 |
[production] |
| 16:09 |
<kartik@tin> |
Finished deploy [cxserver/deploy@11aecc9]: Update cxserver to 0c242c0, Pin service-runner to 2.4.2 (duration: 03m 29s) |
[production] |
| 16:05 |
<kartik@tin> |
Started deploy [cxserver/deploy@11aecc9]: Update cxserver to 0c242c0, Pin service-runner to 2.4.2 |
[production] |
| 16:05 |
<godog> |
unban statsd traffic from scb on graphite1001 - T181333 |
[production] |
| 15:45 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@e35aa05]: Rollback. Disable GC metric reporting T181333 (duration: 00m 31s) |
[production] |
| 15:45 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@e35aa05]: Rollback. Disable GC metric reporting T181333 |
[production] |
| 15:37 |
<volans> |
restarted statsd-proxy on graphite1001 (died during investigation) T181333 |
[production] |
| 14:34 |
<godog> |
rolling restart of cxserver to alleviate metrics leak - T181333 |
[production] |
| 14:26 |
<godog> |
restart cxserver on scb100[34] - T181333 |
[production] |