|
2019-12-05
§
|
| 06:51 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:51 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1101:3317, db1101:3318', diff saved to https://phabricator.wikimedia.org/P9814 and previous config saved to /var/cache/conftool/dbconfig/20191205-064845-marostegui.json |
[production] |
| 06:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1101:3317, db1101:3318', diff saved to https://phabricator.wikimedia.org/P9813 and previous config saved to /var/cache/conftool/dbconfig/20191205-063103-marostegui.json |
[production] |
| 06:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1101:3317, db1101:3318', diff saved to https://phabricator.wikimedia.org/P9812 and previous config saved to /var/cache/conftool/dbconfig/20191205-061453-marostegui.json |
[production] |
| 05:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3317 for upgrade', diff saved to https://phabricator.wikimedia.org/P9811 and previous config saved to /var/cache/conftool/dbconfig/20191205-055756-marostegui.json |
[production] |
| 03:37 |
<twentyafterfour> |
leaving phabricator on phab1003 for tonight while phab1001 raid syncs, will pick it up tomorrow to decide where to go from here |
[production] |
| 03:32 |
<twentyafterfour@deploy1001> |
Finished deploy [phabricator/deployment@UNKNOWN]: deploy release/2019-08-22/1 to phab1001 (duration: 01m 36s) |
[production] |
| 03:30 |
<twentyafterfour@deploy1001> |
Started deploy [phabricator/deployment@UNKNOWN]: deploy release/2019-08-22/1 to phab1001 |
[production] |
| 03:29 |
<twentyafterfour@deploy1001> |
Finished deploy [phabricator/deployment@UNKNOWN]: deploy release/2019-08-22/1 to phab1001 (duration: 00m 22s) |
[production] |
| 03:29 |
<twentyafterfour@deploy1001> |
Started deploy [phabricator/deployment@UNKNOWN]: deploy release/2019-08-22/1 to phab1001 |
[production] |
| 03:07 |
<mutante> |
phab1001 - now using AHCI mode after reinstall, performance much better. rsyncing /srv/repos from phab1003 again |
[production] |
| 02:32 |
<mutante> |
phab1001 - signed new puppet cert - initial puppet run in progress |
[production] |
| 02:27 |
<mutante> |
phab1001 - fixed boot order in BIOS to boot only from HDD, back at login |
[production] |
| 02:12 |
<ejegg> |
updated payments-wiki from f61c9f0692 to 81921bd04a |
[production] |
| 01:21 |
<mutante> |
phab1001 - rebooting to BIOS once more - "The settings were saved successfully." |
[production] |
| 01:19 |
<twentyafterfour> |
phab1001 back, still in legacy ide mode |
[production] |
| 01:12 |
<mutante> |
phab1001 - enabling Write Cache in BIOS |
[production] |
| 01:07 |
<mutante> |
phab1001 - System BIOS Settings > SATA Settings > Embedded SATA: switch from ATA to AHCI mode (T238956) |
[production] |
| 01:05 |
<mutante> |
phab1001 - powercycling |
[production] |
| 01:04 |
<mutante> |
telling phab1001 to boot into BIOS next time it boots via mgmt console (https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/Dell_PowerEdge_RN30#Reboot_and_boot_into_BIOS_then_console) |
[production] |
| 01:03 |
<twentyafterfour> |
phabricator switched back to phab1003 - reimaging phab1001 now |
[production] |
| 00:49 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=phab1001-vcs.eqiad.wmnet |
[production] |
| 00:49 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=phab1001-vcs.eqiad.wmnet |
[production] |
| 00:43 |
<cwhite@cumin1001> |
dbctl commit (dc=all): 'Depool db1062 T239874', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20191205-004256-cwhite.json |
[production] |
|
2019-12-04
§
|
| 23:38 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.35.0-wmf.5" |
[production] |
| 23:35 |
<brennen@deploy1001> |
Scap failed!: 9/11 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 23:30 |
<brennen@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.8 (duration: 01m 01s) |
[production] |
| 23:29 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.8 |
[production] |
| 23:22 |
<jforrester@deploy1001> |
Synchronized wmf-config/logging.php: Keep test consistent w/ operations/puppet, for logging (duration: 01m 02s) |
[production] |
| 23:21 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Keep test consistent w/ operations/puppet, for CS (duration: 01m 03s) |
[production] |
| 22:49 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Disable VisualEditor on Wikitech (and Labs Wikitech) (duration: 01m 02s) |
[production] |
| 22:45 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@e6afe36]: Update mobileapps to 9e9b042 (duration: 05m 48s) |
[production] |
| 22:39 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@e6afe36]: Update mobileapps to 9e9b042 |
[production] |
| 22:39 |
<bstorm_> |
powered off cloudstore1008, disabled sync from cloudstore1009, and downtimed both cloudstore1008 and cloudstore1009 for memory module replacement T239569 |
[production] |
| 22:37 |
<bstorm_> |
poweroff cloudstore1008 for memory module replacement |
[production] |
| 22:24 |
<RoanKattouw> |
T208369 ran mwscript extensions/GrowthExperiments/maintenance/deleteOldSurveys.php kowiki --cutoff 350 |
[production] |
| 22:21 |
<RoanKattouw> |
T208369 ran mwscript extensions/GrowthExperiments/maintenance/deleteOldSurveys.php cswiki --cutoff 350 |
[production] |
| 21:48 |
<eileen> |
civicrm revision changed from 6812488f3a to 4a72ad4e63, config revision is 9f4db1edad (CiviCRM security patches ) |
[production] |
| 21:38 |
<rzl@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw226[3456]\.codfw\.wmnet |
[production] |
| 21:11 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:09 |
<rzl@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:47 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@fc710ec] (thin): Weekly train deploy to labs/notebooks (duration: 00m 07s) |
[production] |
| 20:47 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@fc710ec] (thin): Weekly train deploy to labs/notebooks |
[production] |
| 20:33 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:31 |
<rzl@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:28 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@fc710ec]: Weekly train deploy (duration: 07m 09s) |
[production] |
| 20:28 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.35.0-wmf.5" |
[production] |
| 20:21 |
<milimetric@deploy1001> |
Started deploy [analytics/refinery@fc710ec]: Weekly train deploy |
[production] |
| 20:15 |
<milimetric@deploy1001> |
Finished deploy [analytics/refinery@fc710ec]: Weekly train deploy (duration: 06m 37s) |
[production] |