|
2019-12-05
§
|
| 12:21 |
<effie> |
Reimage mw2261.codfw.wmnet |
[production] |
| 12:16 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 6c9d168: Fix namespace name - napwikisource (T239547) (duration: 01m 02s) |
[production] |
| 10:48 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:44 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:38 |
<oblivian@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 10:35 |
<oblivian@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 10:26 |
<effie> |
reimage mw2260.codfw.wmnet |
[production] |
| 10:13 |
<oblivian@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 09:54 |
<ema> |
text@esams: disable ats-be origin server request coalescing T238494 |
[production] |
| 09:07 |
<marostegui> |
Upgrade db2094 and db2095 |
[production] |
| 08:38 |
<marostegui> |
Upgrade db2078 |
[production] |
| 08:09 |
<marostegui> |
Upgrade pc2007, pc2008, pc2009, pc2010 |
[production] |
| 08:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1062 from etcd T239188', diff saved to https://phabricator.wikimedia.org/P9821 and previous config saved to /var/cache/conftool/dbconfig/20191205-080909-marostegui.json |
[production] |
| 08:03 |
<elukey> |
remove logstash_cleanup_indices_apifeatureusage-search.svc.codfw.wmnet and logstash_cleanup_indices_apifeatureusage-search.svc.eqiad.wmnet from logstash1025,logstash1024,logstash1023,logstash2024,logstash2025 to reduce cronspam - T234854 |
[production] |
| 07:42 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1099:3311, db1099:3318', diff saved to https://phabricator.wikimedia.org/P9820 and previous config saved to /var/cache/conftool/dbconfig/20191205-074200-marostegui.json |
[production] |
| 07:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1099:3311, db1099:3318', diff saved to https://phabricator.wikimedia.org/P9819 and previous config saved to /var/cache/conftool/dbconfig/20191205-073209-marostegui.json |
[production] |
| 07:29 |
<_joe_> |
ran apt-get install manually on kubestagetcd1001 to fix broken packages |
[production] |
| 07:25 |
<_joe_> |
manually running package_builder_Clean_up_build_directory.service on boron |
[production] |
| 07:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1099:3311, db1099:3318', diff saved to https://phabricator.wikimedia.org/P9818 and previous config saved to /var/cache/conftool/dbconfig/20191205-072314-marostegui.json |
[production] |
| 07:22 |
<_joe_> |
umounting /proc,/sys,/dev from /var/cache/pbuilder/build/cow.6815 on boron to allow reaping it away |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1099:3311, db1099:3318', diff saved to https://phabricator.wikimedia.org/P9817 and previous config saved to /var/cache/conftool/dbconfig/20191205-071445-marostegui.json |
[production] |
| 07:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1099:3311, db1099:3318 for upgrade', diff saved to https://phabricator.wikimedia.org/P9816 and previous config saved to /var/cache/conftool/dbconfig/20191205-070631-marostegui.json |
[production] |
| 06:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1101:3317, db1101:3318', diff saved to https://phabricator.wikimedia.org/P9815 and previous config saved to /var/cache/conftool/dbconfig/20191205-065536-marostegui.json |
[production] |
| 06:51 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 06:51 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 06:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1101:3317, db1101:3318', diff saved to https://phabricator.wikimedia.org/P9814 and previous config saved to /var/cache/conftool/dbconfig/20191205-064845-marostegui.json |
[production] |
| 06:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1101:3317, db1101:3318', diff saved to https://phabricator.wikimedia.org/P9813 and previous config saved to /var/cache/conftool/dbconfig/20191205-063103-marostegui.json |
[production] |
| 06:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1101:3317, db1101:3318', diff saved to https://phabricator.wikimedia.org/P9812 and previous config saved to /var/cache/conftool/dbconfig/20191205-061453-marostegui.json |
[production] |
| 05:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101:3317 for upgrade', diff saved to https://phabricator.wikimedia.org/P9811 and previous config saved to /var/cache/conftool/dbconfig/20191205-055756-marostegui.json |
[production] |
| 03:37 |
<twentyafterfour> |
leaving phabricator on phab1003 for tonight while phab1001 raid syncs, will pick it up tomorrow to decide where to go from here |
[production] |
| 03:32 |
<twentyafterfour@deploy1001> |
Finished deploy [phabricator/deployment@UNKNOWN]: deploy release/2019-08-22/1 to phab1001 (duration: 01m 36s) |
[production] |
| 03:30 |
<twentyafterfour@deploy1001> |
Started deploy [phabricator/deployment@UNKNOWN]: deploy release/2019-08-22/1 to phab1001 |
[production] |
| 03:29 |
<twentyafterfour@deploy1001> |
Finished deploy [phabricator/deployment@UNKNOWN]: deploy release/2019-08-22/1 to phab1001 (duration: 00m 22s) |
[production] |
| 03:29 |
<twentyafterfour@deploy1001> |
Started deploy [phabricator/deployment@UNKNOWN]: deploy release/2019-08-22/1 to phab1001 |
[production] |
| 03:07 |
<mutante> |
phab1001 - now using AHCI mode after reinstall, performance much better. rsyncing /srv/repos from phab1003 again |
[production] |
| 02:32 |
<mutante> |
phab1001 - signed new puppet cert - initial puppet run in progress |
[production] |
| 02:27 |
<mutante> |
phab1001 - fixed boot order in BIOS to boot only from HDD, back at login |
[production] |
| 02:12 |
<ejegg> |
updated payments-wiki from f61c9f0692 to 81921bd04a |
[production] |
| 01:21 |
<mutante> |
phab1001 - rebooting to BIOS once more - "The settings were saved successfully." |
[production] |
| 01:19 |
<twentyafterfour> |
phab1001 back, still in legacy ide mode |
[production] |
| 01:12 |
<mutante> |
phab1001 - enabling Write Cache in BIOS |
[production] |
| 01:07 |
<mutante> |
phab1001 - System BIOS Settings > SATA Settings > Embedded SATA: switch from ATA to AHCI mode (T238956) |
[production] |
| 01:05 |
<mutante> |
phab1001 - powercycling |
[production] |
| 01:04 |
<mutante> |
telling phab1001 to boot into BIOS next time it boots via mgmt console (https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/Dell_PowerEdge_RN30#Reboot_and_boot_into_BIOS_then_console) |
[production] |
| 01:03 |
<twentyafterfour> |
phabricator switched back to phab1003 - reimaging phab1001 now |
[production] |
| 00:49 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=phab1001-vcs.eqiad.wmnet |
[production] |
| 00:49 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=phab1001-vcs.eqiad.wmnet |
[production] |
| 00:43 |
<cwhite@cumin1001> |
dbctl commit (dc=all): 'Depool db1062 T239874', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20191205-004256-cwhite.json |
[production] |