2351-2400 of 10000 results (70ms)
2020-03-30 §
06:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1074 after schema change', diff saved to https://phabricator.wikimedia.org/P10813 and previous config saved to /var/cache/conftool/dbconfig/20200330-062858-marostegui.json [production]
06:26 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:04 <marostegui> Deploy schema change on db1074 with replication, this will generate lag on s2 labs [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074 for schema change', diff saved to https://phabricator.wikimedia.org/P10812 and previous config saved to /var/cache/conftool/dbconfig/20200330-060338-marostegui.json [production]
05:40 <vgutierrez> pool cp2027 - T247340 [production]
05:13 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:10 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
04:55 <vgutierrez> Enable TLS Session tickets in ulsfo - T245616 [production]
04:32 <vgutierrez> upgrade ATS to version 8.0.6-1wm4 on ulsfo - T245616 [production]
2020-03-29 §
08:24 <elukey> powercycle elastic1059 - mgmt/serial console stuck, no ssh - racadm getsel shows a lot of OEM errors occurred, nothing specific [production]
2020-03-28 §
16:54 <elukey> restart yarn on analytics1071 [production]
12:05 <vgutierrez> preemptive restart of ats-tls on cp1081 and cp3062 - T248736 [production]
11:32 <vgutierrez> restart ats-tls on cp1077 - T248736 [production]
08:34 <vgutierrez> pool cp1089 [production]
08:30 <vgutierrez> restarting ats-tls on cp1089 [production]
2020-03-27 §
20:51 <ejegg> updated payments-wiki from db618f429d to 1640f5e21e [production]
15:15 <andrew@deploy1001> Finished deploy [horizon/deploy@33e67f9]: fix Identity->Projects with keystone Queens (duration: 03m 35s) [production]
15:12 <andrew@deploy1001> Started deploy [horizon/deploy@33e67f9]: fix Identity->Projects with keystone Queens [production]
14:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1129 after schema change', diff saved to https://phabricator.wikimedia.org/P10807 and previous config saved to /var/cache/conftool/dbconfig/20200327-144125-marostegui.json [production]
14:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1129 for schema change', diff saved to https://phabricator.wikimedia.org/P10806 and previous config saved to /var/cache/conftool/dbconfig/20200327-142240-marostegui.json [production]
14:19 <moritzm> updating linux-image-4.9.0-11-amd64 where applicable [production]
13:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1076 after schema change', diff saved to https://phabricator.wikimedia.org/P10805 and previous config saved to /var/cache/conftool/dbconfig/20200327-133022-marostegui.json [production]
13:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1076 for schema change', diff saved to https://phabricator.wikimedia.org/P10804 and previous config saved to /var/cache/conftool/dbconfig/20200327-130706-marostegui.json [production]
13:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1105:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P10803 and previous config saved to /var/cache/conftool/dbconfig/20200327-130542-marostegui.json [production]
12:49 <Amir1> ladsgroup@mwmaint1002:~$ mwscript createAndPromote.php --wiki=labswiki --force "Ladsgroup" --interface-admin [production]
12:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1105:3312 for schema change', diff saved to https://phabricator.wikimedia.org/P10802 and previous config saved to /var/cache/conftool/dbconfig/20200327-122144-marostegui.json [production]
12:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1103:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P10801 and previous config saved to /var/cache/conftool/dbconfig/20200327-122058-marostegui.json [production]
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1103:3312 for schema change', diff saved to https://phabricator.wikimedia.org/P10800 and previous config saved to /var/cache/conftool/dbconfig/20200327-120234-marostegui.json [production]
11:54 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase-backend,name=restbase202[123].codfw.wmnet [production]
11:51 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase-ssl,name=restbase202[123].codfw.wmnet [production]
11:45 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase,name=restbase2023.codfw.wmnet [production]
11:44 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase,name=restbase2022.codfw.wmnet [production]
11:44 <oblivian@puppetmaster1001> conftool action : edit; selector: dc=codfw,cluster=restbase,service=restbase-ssl,name=restbase202[1].codfw.wmnet [production]
11:44 <hnowlan@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase,name=restbase2021.codfw.wmnet [production]
10:55 <mutante> revoke puppet cert webserver-misc-apps.discovery.wmnet and recreate with additional SANs for new VMs [production]
10:45 <mutante> miscweb1002 - upload and unpack RackTables-0.21.4 (T247646 T247648) [production]
10:28 <marostegui> Alter db2125 s2 to set page_restrictions to default NULL - T248333 [production]
10:12 <mutante> miscweb1002 - sed -i 's/tin.eqiad/deployment.eqiad/g' /srv/deployment/iegreview/iegreview-cache/.config T247648 [production]
10:04 <vgutierrez> upload trafficserver 8.0.6-1wm4 to apt.wm.o (buster) - T245616 T170567 [production]
10:03 <mutante> sodium - find /srv/mirrors/debian/ -user root -exec chown -h mirror:mirror {} \; (-h to also fix symbolic links); sudo -u mirror ftpsync (T248660) [production]
10:02 <marostegui> Alter db2084:3315 enwikivoyage.page to set page_restrictions to default NULL - T248333 [production]
10:01 <marostegui> Alter db1096:3315 enwikivoyage.page to set page_restrictions to default NULL - T248333 [production]
09:37 <mutante> sodium - running ftpsync as user mirror (T248660) [production]
09:36 <mutante> sodium fixing root owned files in /srv/mirrors/debian to be owned by mirror:mirror (T248660) [production]
09:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1098:3316', diff saved to https://phabricator.wikimedia.org/P10799 and previous config saved to /var/cache/conftool/dbconfig/20200327-093214-marostegui.json [production]
09:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1098:3316 for schema change', diff saved to https://phabricator.wikimedia.org/P10798 and previous config saved to /var/cache/conftool/dbconfig/20200327-093106-marostegui.json [production]
07:58 <marostegui> Deploy schema change on s2 codfw - this will generate lag on s2 codfw - T248333 [production]
07:36 <elukey> execute 'rm /etc/logrotate.d/ceph-common' on cloudvirt[1,2]* and cloudcontrol* to stop daily cronspam (file not in the puppet catalog anymore) [production]
07:32 <moritzm> installing grub2 updates from Stretch point release [production]
07:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1130 after schema change', diff saved to https://phabricator.wikimedia.org/P10796 and previous config saved to /var/cache/conftool/dbconfig/20200327-072334-marostegui.json [production]