|
2020-03-27
§
|
| 09:37 |
<mutante> |
sodium - running ftpsync as user mirror (T248660) |
[production] |
| 09:36 |
<mutante> |
sodium fixing root owned files in /srv/mirrors/debian to be owned by mirror:mirror (T248660) |
[production] |
| 09:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1098:3316', diff saved to https://phabricator.wikimedia.org/P10799 and previous config saved to /var/cache/conftool/dbconfig/20200327-093214-marostegui.json |
[production] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1098:3316 for schema change', diff saved to https://phabricator.wikimedia.org/P10798 and previous config saved to /var/cache/conftool/dbconfig/20200327-093106-marostegui.json |
[production] |
| 07:58 |
<marostegui> |
Deploy schema change on s2 codfw - this will generate lag on s2 codfw - T248333 |
[production] |
| 07:36 |
<elukey> |
execute 'rm /etc/logrotate.d/ceph-common' on cloudvirt[1,2]* and cloudcontrol* to stop daily cronspam (file not in the puppet catalog anymore) |
[production] |
| 07:32 |
<moritzm> |
installing grub2 updates from Stretch point release |
[production] |
| 07:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1130 after schema change', diff saved to https://phabricator.wikimedia.org/P10796 and previous config saved to /var/cache/conftool/dbconfig/20200327-072334-marostegui.json |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1130 for schema change', diff saved to https://phabricator.wikimedia.org/P10795 and previous config saved to /var/cache/conftool/dbconfig/20200327-070224-marostegui.json |
[production] |
| 07:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1082 after schema change', diff saved to https://phabricator.wikimedia.org/P10794 and previous config saved to /var/cache/conftool/dbconfig/20200327-070014-marostegui.json |
[production] |
| 06:31 |
<marostegui> |
Deploy schema change on db1082, this will generate lag on s5 labs |
[production] |
| 06:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1082 for schema change', diff saved to https://phabricator.wikimedia.org/P10793 and previous config saved to /var/cache/conftool/dbconfig/20200327-063042-marostegui.json |
[production] |
|
2020-03-26
§
|
| 23:07 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: ce63a4e: Enable wmgUseFooterContactLink for cswiki (T248584; take II) (duration: 00m 57s) |
[production] |
| 23:05 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: ce63a4e: Enable wmgUseFooterContactLink for cswiki (T248584) (duration: 00m 58s) |
[production] |
| 22:51 |
<krinkle@deploy1001> |
Synchronized php-1.35.0-wmf.25/includes/user/UserRightsProxy.php: I9121f5aae (4/4) (duration: 00m 58s) |
[production] |
| 22:50 |
<krinkle@deploy1001> |
Synchronized php-1.35.0-wmf.25/includes/search/SearchMySQL.php: I9121f5aae (3/4) (duration: 00m 58s) |
[production] |
| 22:48 |
<krinkle@deploy1001> |
Synchronized php-1.35.0-wmf.25/includes/objectcache/SqlBagOStuff.php: I9121f5aae (2/4) (duration: 00m 58s) |
[production] |
| 22:44 |
<krinkle@deploy1001> |
Synchronized php-1.35.0-wmf.25/includes/jobqueue/jobs/RecentChangesUpdateJob.php: I9121f5aae (1/4) (duration: 01m 00s) |
[production] |
| 22:05 |
<ejegg> |
updated fundraising CiviCRM from f1cb23e809 to cf2e2c11c3 |
[production] |
| 21:43 |
<mholloway-shell@deploy1001> |
Synchronized php-1.35.0-wmf.25/extensions/MachineVision: Fix: Stop sorting label suggestions by Wikidata ID in ApiQueryImageLabels (duration: 01m 00s) |
[production] |
| 21:34 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 21:32 |
<cdanis> |
cdanis@re0.cr1-eqsin# set chassis afeb slot 0 inline-services flex-flow-sizing cdanis@re0.cr1-eqsin# commit comment "flex-flow-sizing T248394" |
[production] |
| 21:31 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:30 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 21:27 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:27 |
<mholloway-shell@deploy1001> |
Finished deploy [mobileapps/deploy@f34260c]: Update mobileapps to 3f30f20c (duration: 03m 07s) |
[production] |
| 21:24 |
<mholloway-shell@deploy1001> |
Started deploy [mobileapps/deploy@f34260c]: Update mobileapps to 3f30f20c |
[production] |
| 21:15 |
<cdanis> |
repool ulsfo |
[production] |
| 21:12 |
<cdanis> |
applied flow-table-size configuration to cr4-ulsfo which did not need a reboot to apply it T248394 |
[production] |
| 16:52 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:50 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:43 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:40 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:34 |
<XioNoX> |
stop exchanging full BGP view between eqiad and codfw - T246721 |
[production] |
| 16:19 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:18 |
<XioNoX> |
stop advertising 208.80.152.0/22 from eqiad - T246721 |
[production] |
| 16:15 |
<mutante> |
signing puppet cert for miscweb1002, installed buster, added insetup role (T247887) |
[production] |
| 16:15 |
<ebernhardson> |
set cloudelastic-chi wikidatawiki_content to 0 replicas while reindexing |
[production] |
| 16:14 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:14 |
<moritzm> |
rebooting mw2150 for some tests |
[production] |
| 16:12 |
<XioNoX> |
stop advertising 2620:0:860::/46 from eqiad - T246721 |
[production] |
| 16:12 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:11 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:11 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:10 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:58 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:53 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:51 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 15:51 |
<moritzm> |
installing grub2 updates from Stretch point release |
[production] |
| 15:49 |
<XioNoX> |
start advertising 208.80.154.0/23 from eqiad - T246721 |
[production] |