2551-2600 of 10000 results (63ms)
2020-03-30 §
10:34 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
10:33 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
10:33 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
09:59 <hoo> Temporary modified dumpsgen's crontab on snapshot1008 so that the Wikidata JSON dumps start at 9:59 UTC today (T248612) [production]
09:56 <hoo@deploy1001> Synchronized php-1.35.0-wmf.25/extensions/Wikibase/repo/maintenance/DumpEntities.php: DumpEntities: Fix DB group default override (T248612) (duration: 01m 02s) [production]
09:19 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:15 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:30 <vgutierrez> pool cp2029 - T248816 [production]
08:12 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:12 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
08:12 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
08:10 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:53 <vgutierrez> depool & decommission cp2002 - T248818 [production]
07:48 <marostegui> Run cloudcontrol1003:~# wmcs-wikireplica-dns to promote dbproxy1018 to wikireplicas active proxy T231520 [production]
07:40 <marostegui> Replace dbproxy1010 with dbproxy1011 for wiki replicas, analytics - T231520 [production]
07:28 <marostegui> Deploy schema change on labswiki (wikitech) - T248333 [production]
07:26 <marostegui> Deploy schema change on s4 codfw, this will generate lag on codfw - T248333 [production]
07:17 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
07:17 <vgutierrez@cumin1001> START - Cookbook sre.hosts.decommission [production]
07:10 <vgutierrez> depool and decommission cp2001 - T248815 [production]
06:52 <vgutierrez> pool cp2028 - T247340 [production]
06:29 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1074 after schema change', diff saved to https://phabricator.wikimedia.org/P10813 and previous config saved to /var/cache/conftool/dbconfig/20200330-062858-marostegui.json [production]
06:26 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:04 <marostegui> Deploy schema change on db1074 with replication, this will generate lag on s2 labs [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074 for schema change', diff saved to https://phabricator.wikimedia.org/P10812 and previous config saved to /var/cache/conftool/dbconfig/20200330-060338-marostegui.json [production]
05:40 <vgutierrez> pool cp2027 - T247340 [production]
05:13 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:10 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
04:55 <vgutierrez> Enable TLS Session tickets in ulsfo - T245616 [production]
04:32 <vgutierrez> upgrade ATS to version 8.0.6-1wm4 on ulsfo - T245616 [production]
2020-03-29 §
08:24 <elukey> powercycle elastic1059 - mgmt/serial console stuck, no ssh - racadm getsel shows a lot of OEM errors occurred, nothing specific [production]
2020-03-28 §
16:54 <elukey> restart yarn on analytics1071 [production]
12:05 <vgutierrez> preemptive restart of ats-tls on cp1081 and cp3062 - T248736 [production]
11:32 <vgutierrez> restart ats-tls on cp1077 - T248736 [production]
08:34 <vgutierrez> pool cp1089 [production]
08:30 <vgutierrez> restarting ats-tls on cp1089 [production]
2020-03-27 §
20:51 <ejegg> updated payments-wiki from db618f429d to 1640f5e21e [production]
15:15 <andrew@deploy1001> Finished deploy [horizon/deploy@33e67f9]: fix Identity->Projects with keystone Queens (duration: 03m 35s) [production]
15:12 <andrew@deploy1001> Started deploy [horizon/deploy@33e67f9]: fix Identity->Projects with keystone Queens [production]
14:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1129 after schema change', diff saved to https://phabricator.wikimedia.org/P10807 and previous config saved to /var/cache/conftool/dbconfig/20200327-144125-marostegui.json [production]
14:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1129 for schema change', diff saved to https://phabricator.wikimedia.org/P10806 and previous config saved to /var/cache/conftool/dbconfig/20200327-142240-marostegui.json [production]
14:19 <moritzm> updating linux-image-4.9.0-11-amd64 where applicable [production]
13:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1076 after schema change', diff saved to https://phabricator.wikimedia.org/P10805 and previous config saved to /var/cache/conftool/dbconfig/20200327-133022-marostegui.json [production]
13:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1076 for schema change', diff saved to https://phabricator.wikimedia.org/P10804 and previous config saved to /var/cache/conftool/dbconfig/20200327-130706-marostegui.json [production]
13:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1105:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P10803 and previous config saved to /var/cache/conftool/dbconfig/20200327-130542-marostegui.json [production]
12:49 <Amir1> ladsgroup@mwmaint1002:~$ mwscript createAndPromote.php --wiki=labswiki --force "Ladsgroup" --interface-admin [production]
12:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1105:3312 for schema change', diff saved to https://phabricator.wikimedia.org/P10802 and previous config saved to /var/cache/conftool/dbconfig/20200327-122144-marostegui.json [production]
12:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1103:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P10801 and previous config saved to /var/cache/conftool/dbconfig/20200327-122058-marostegui.json [production]
12:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1103:3312 for schema change', diff saved to https://phabricator.wikimedia.org/P10800 and previous config saved to /var/cache/conftool/dbconfig/20200327-120234-marostegui.json [production]