|
2020-03-27
§
|
| 20:51 |
<ejegg> |
updated payments-wiki from db618f429d to 1640f5e21e |
[production] |
| 15:15 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@33e67f9]: fix Identity->Projects with keystone Queens (duration: 03m 35s) |
[production] |
| 15:12 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@33e67f9]: fix Identity->Projects with keystone Queens |
[production] |
| 14:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1129 after schema change', diff saved to https://phabricator.wikimedia.org/P10807 and previous config saved to /var/cache/conftool/dbconfig/20200327-144125-marostegui.json |
[production] |
| 14:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1129 for schema change', diff saved to https://phabricator.wikimedia.org/P10806 and previous config saved to /var/cache/conftool/dbconfig/20200327-142240-marostegui.json |
[production] |
| 14:19 |
<moritzm> |
updating linux-image-4.9.0-11-amd64 where applicable |
[production] |
| 13:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1076 after schema change', diff saved to https://phabricator.wikimedia.org/P10805 and previous config saved to /var/cache/conftool/dbconfig/20200327-133022-marostegui.json |
[production] |
| 13:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1076 for schema change', diff saved to https://phabricator.wikimedia.org/P10804 and previous config saved to /var/cache/conftool/dbconfig/20200327-130706-marostegui.json |
[production] |
| 13:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1105:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P10803 and previous config saved to /var/cache/conftool/dbconfig/20200327-130542-marostegui.json |
[production] |
| 12:49 |
<Amir1> |
ladsgroup@mwmaint1002:~$ mwscript createAndPromote.php --wiki=labswiki --force "Ladsgroup" --interface-admin |
[production] |
| 12:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1105:3312 for schema change', diff saved to https://phabricator.wikimedia.org/P10802 and previous config saved to /var/cache/conftool/dbconfig/20200327-122144-marostegui.json |
[production] |
| 12:20 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1103:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P10801 and previous config saved to /var/cache/conftool/dbconfig/20200327-122058-marostegui.json |
[production] |
| 12:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1103:3312 for schema change', diff saved to https://phabricator.wikimedia.org/P10800 and previous config saved to /var/cache/conftool/dbconfig/20200327-120234-marostegui.json |
[production] |
| 11:54 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase-backend,name=restbase202[123].codfw.wmnet |
[production] |
| 11:51 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase-ssl,name=restbase202[123].codfw.wmnet |
[production] |
| 11:45 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase,name=restbase2023.codfw.wmnet |
[production] |
| 11:44 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase,name=restbase2022.codfw.wmnet |
[production] |
| 11:44 |
<oblivian@puppetmaster1001> |
conftool action : edit; selector: dc=codfw,cluster=restbase,service=restbase-ssl,name=restbase202[1].codfw.wmnet |
[production] |
| 11:44 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase,name=restbase2021.codfw.wmnet |
[production] |
| 10:55 |
<mutante> |
revoke puppet cert webserver-misc-apps.discovery.wmnet and recreate with additional SANs for new VMs |
[production] |
| 10:45 |
<mutante> |
miscweb1002 - upload and unpack RackTables-0.21.4 (T247646 T247648) |
[production] |
| 10:28 |
<marostegui> |
Alter db2125 s2 to set page_restrictions to default NULL - T248333 |
[production] |
| 10:12 |
<mutante> |
miscweb1002 - sed -i 's/tin.eqiad/deployment.eqiad/g' /srv/deployment/iegreview/iegreview-cache/.config T247648 |
[production] |
| 10:04 |
<vgutierrez> |
upload trafficserver 8.0.6-1wm4 to apt.wm.o (buster) - T245616 T170567 |
[production] |
| 10:03 |
<mutante> |
sodium - find /srv/mirrors/debian/ -user root -exec chown -h mirror:mirror {} \; (-h to also fix symbolic links); sudo -u mirror ftpsync (T248660) |
[production] |
| 10:02 |
<marostegui> |
Alter db2084:3315 enwikivoyage.page to set page_restrictions to default NULL - T248333 |
[production] |
| 10:01 |
<marostegui> |
Alter db1096:3315 enwikivoyage.page to set page_restrictions to default NULL - T248333 |
[production] |
| 09:37 |
<mutante> |
sodium - running ftpsync as user mirror (T248660) |
[production] |
| 09:36 |
<mutante> |
sodium fixing root owned files in /srv/mirrors/debian to be owned by mirror:mirror (T248660) |
[production] |
| 09:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1098:3316', diff saved to https://phabricator.wikimedia.org/P10799 and previous config saved to /var/cache/conftool/dbconfig/20200327-093214-marostegui.json |
[production] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1098:3316 for schema change', diff saved to https://phabricator.wikimedia.org/P10798 and previous config saved to /var/cache/conftool/dbconfig/20200327-093106-marostegui.json |
[production] |
| 07:58 |
<marostegui> |
Deploy schema change on s2 codfw - this will generate lag on s2 codfw - T248333 |
[production] |
| 07:36 |
<elukey> |
execute 'rm /etc/logrotate.d/ceph-common' on cloudvirt[1,2]* and cloudcontrol* to stop daily cronspam (file not in the puppet catalog anymore) |
[production] |
| 07:32 |
<moritzm> |
installing grub2 updates from Stretch point release |
[production] |
| 07:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1130 after schema change', diff saved to https://phabricator.wikimedia.org/P10796 and previous config saved to /var/cache/conftool/dbconfig/20200327-072334-marostegui.json |
[production] |
| 07:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1130 for schema change', diff saved to https://phabricator.wikimedia.org/P10795 and previous config saved to /var/cache/conftool/dbconfig/20200327-070224-marostegui.json |
[production] |
| 07:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1082 after schema change', diff saved to https://phabricator.wikimedia.org/P10794 and previous config saved to /var/cache/conftool/dbconfig/20200327-070014-marostegui.json |
[production] |
| 06:31 |
<marostegui> |
Deploy schema change on db1082, this will generate lag on s5 labs |
[production] |
| 06:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1082 for schema change', diff saved to https://phabricator.wikimedia.org/P10793 and previous config saved to /var/cache/conftool/dbconfig/20200327-063042-marostegui.json |
[production] |
|
2020-03-26
§
|
| 23:07 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: ce63a4e: Enable wmgUseFooterContactLink for cswiki (T248584; take II) (duration: 00m 57s) |
[production] |
| 23:05 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: ce63a4e: Enable wmgUseFooterContactLink for cswiki (T248584) (duration: 00m 58s) |
[production] |
| 22:51 |
<krinkle@deploy1001> |
Synchronized php-1.35.0-wmf.25/includes/user/UserRightsProxy.php: I9121f5aae (4/4) (duration: 00m 58s) |
[production] |
| 22:50 |
<krinkle@deploy1001> |
Synchronized php-1.35.0-wmf.25/includes/search/SearchMySQL.php: I9121f5aae (3/4) (duration: 00m 58s) |
[production] |
| 22:48 |
<krinkle@deploy1001> |
Synchronized php-1.35.0-wmf.25/includes/objectcache/SqlBagOStuff.php: I9121f5aae (2/4) (duration: 00m 58s) |
[production] |
| 22:44 |
<krinkle@deploy1001> |
Synchronized php-1.35.0-wmf.25/includes/jobqueue/jobs/RecentChangesUpdateJob.php: I9121f5aae (1/4) (duration: 01m 00s) |
[production] |
| 22:05 |
<ejegg> |
updated fundraising CiviCRM from f1cb23e809 to cf2e2c11c3 |
[production] |
| 21:43 |
<mholloway-shell@deploy1001> |
Synchronized php-1.35.0-wmf.25/extensions/MachineVision: Fix: Stop sorting label suggestions by Wikidata ID in ApiQueryImageLabels (duration: 01m 00s) |
[production] |
| 21:34 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 21:32 |
<cdanis> |
cdanis@re0.cr1-eqsin# set chassis afeb slot 0 inline-services flex-flow-sizing cdanis@re0.cr1-eqsin# commit comment "flex-flow-sizing T248394" |
[production] |
| 21:31 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |