|
2019-12-06
§
|
| 03:53 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 03:53 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 02:12 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.8/extensions/SecurePoll/cli/dump.php: T239968 (duration: 01m 04s) |
[production] |
| 01:34 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/SecurePoll/cli/dump.php: T239968 (duration: 01m 00s) |
[production] |
| 01:25 |
<reedy@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/SecurePoll/cli/dump.php: T239968 (duration: 01m 01s) |
[production] |
| 01:09 |
<ejegg> |
updated fundraising internal dashboard from 3a93d2aba4 to d9d74429baa |
[production] |
| 01:08 |
<ejegg> |
updated payments-wiki from 81921bd04a to 00632a397c |
[production] |
| 01:04 |
<catrope@deploy1001> |
Synchronized private/PrivateSettings.php: HMAC value for Kask config (T222099) (duration: 00m 59s) |
[production] |
| 01:02 |
<reedy@deploy1001> |
Synchronized private/PrivateSettings.php: wmgSessionStoreHMACKey T222099 (duration: 01m 07s) |
[production] |
| 00:47 |
<catrope@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Use PHP serialization with HMAC for Kask session serialization (T222099) (duration: 01m 01s) |
[production] |
| 00:08 |
<catrope@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Add *.archives.go.jp to $wgCopyUploadsDomains (T238476) (duration: 01m 00s) |
[production] |
|
2019-12-05
§
|
| 23:44 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T235263 Turn off redirect on exact search match for Commons (duration: 01m 00s) |
[production] |
| 23:04 |
<ebernhardson> |
[cloudelastic-chi] reduce indices.recovery.max_bytes_per_sec from 512mb->128mb |
[production] |
| 22:30 |
<herron@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 22:28 |
<herron@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 22:07 |
<krinkle@deploy1001> |
Synchronized wmf-config/: I64e5ebe5fcd6b - removes arclamp.php (duration: 01m 01s) |
[production] |
| 22:03 |
<mutante> |
phabricator - git-ssh.wikimedia.org has been fixed and is up again (T238956) |
[production] |
| 22:01 |
<mutante> |
phab1001 - restarting ssh-phab to listen on additional LVS IP |
[production] |
| 22:00 |
<krinkle@deploy1001> |
Synchronized php-1.35.0-wmf.8/includes/libs/rdbms/database/: T233342 (duration: 01m 02s) |
[production] |
| 21:55 |
<twentyafterfour> |
stopping phd on phab1003 and starting on phab1001 |
[production] |
| 21:50 |
<mutante> |
phab1003 - remove IPv6 service IP for git-ssh from lo:LVS |
[production] |
| 21:34 |
<mutante> |
puppetmaster2001: deleting /var/run/confd-template/.git-ssh*.err to fix confd template compilation alerts |
[production] |
| 21:33 |
<mutante> |
puppetmaster1001: deleting /var/run/confd-template/.git-ssh*.err to fix confd template compilation alerts |
[production] |
| 21:19 |
<mutante> |
phab1001 - systemctl restart ssh-phab (to make it listen on IPv6, race between puppet adding the IP and starting the service) |
[production] |
| 21:09 |
<bblack> |
ns0.wikimedia.org: restore routing to authdns1001 |
[production] |
| 21:03 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=phab1001-vcs.eqiad.wmnet |
[production] |
| 21:00 |
<mutante> |
phab1001 - reload apache2, removed /ws/ rewrite for wstunnel for aphlict |
[production] |
| 21:00 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:58 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:56 |
<bblack> |
cr[12]-eqiad: delete leftover static route of ns2->authdns1001 from esams work, which was blinding icinga to the real ns2 :P |
[production] |
| 20:49 |
<mholloway-shell@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 20:48 |
<twentyafterfour> |
successfully migrated to phab1001 with no apparent user impact! |
[production] |
| 20:47 |
<mholloway-shell@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 20:46 |
<mholloway-shell@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 20:43 |
<bblack> |
ns0.wikimedia.org: re-routing auth traffic from authdns1001 (reimaging) to dns1001 |
[production] |
| 20:41 |
<mutante> |
running puppet on all cp* for phab change |
[production] |
| 20:36 |
<volker-e@deploy1001> |
Finished deploy [design/style-guide@437023f]: Deploy design/style-guide: (duration: 00m 08s) |
[production] |
| 20:36 |
<volker-e@deploy1001> |
Started deploy [design/style-guide@437023f]: Deploy design/style-guide: |
[production] |
| 20:29 |
<twentyafterfour> |
migrating back to phab1001, minimal downtime expected |
[production] |
| 20:12 |
<mutante> |
phab1001 - rebooting to hopefully clear "microcode vuln" icinga alert |
[production] |
| 20:11 |
<onimisionipe> |
ban cloudelastic1002 from shard allocation - T230088 |
[production] |
| 20:10 |
<bblack> |
ns1.wikimedia.org: restoring normal routing to the newly-reimaged authdns2001 |
[production] |
| 19:56 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 19:53 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:47 |
<urbanecm@deploy1001> |
Synchronized php-1.35.0-wmf.8/extensions/Linter/extension.json: SWAT: afcfdce: Revert "Revert "Implement ParserLogLinterData hook"" (3/3, T238456) (duration: 01m 00s) |
[production] |
| 19:46 |
<urbanecm@deploy1001> |
Synchronized php-1.35.0-wmf.8/extensions/Linter/includes/ApiRecordLint.php: SWAT: afcfdce: Revert "Revert "Implement ParserLogLinterData hook"" (2/3, T238456) (duration: 01m 09s) |
[production] |
| 19:44 |
<urbanecm@deploy1001> |
Synchronized php-1.35.0-wmf.8/extensions/Linter/includes/Hooks.php: SWAT: afcfdce: Revert "Revert "Implement ParserLogLinterData hook"" (1/3, T238456) (duration: 01m 11s) |
[production] |
| 19:41 |
<urbanecm@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/Linter/includes/ApiRecordLint.php: SWAT: 7b7f326: Implement ParserLogLinterData hook (3/3, T238456) (duration: 01m 04s) |
[production] |
| 19:39 |
<urbanecm@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/Linter/extension.json: SWAT: 7b7f326: Implement ParserLogLinterData hook (2/3, T238456) (duration: 01m 05s) |
[production] |
| 19:37 |
<urbanecm@deploy1001> |
Synchronized php-1.35.0-wmf.5/extensions/Linter/includes/Hooks.php: SWAT: 7b7f326: Implement ParserLogLinterData hook (1/3, T238456) (duration: 01m 09s) |
[production] |