551-600 of 10000 results (77ms)
2023-04-24 ยง
12:56 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
12:29 <cgoubert@deploy2002> helmfile [staging] DONE helmfile.d/services/push-notifications: apply [production]
12:28 <cgoubert@deploy2002> helmfile [staging] START helmfile.d/services/push-notifications: apply [production]
12:28 <claime> Deploying push-notifications staging for switch to mw-api-int - T334061 [production]
11:23 <cgoubert@cumin1001> conftool action : set/weight=30; selector: dc=codfw,cluster=api_appserver,service=canary [production]
11:21 <cgoubert@cumin1001> conftool action : set/weight=25; selector: dc=codfw,cluster=appserver,service=canary [production]
11:19 <cgoubert@cumin1001> conftool action : set/weight=30; selector: dc=eqiad,cluster=appserver,service=canary [production]
11:18 <cgoubert@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:17 <cgoubert@cumin1001> START - Cookbook sre.dns.netbox [production]
11:14 <cgoubert@cumin1001> conftool action : set/weight=10; selector: dc=codfw,cluster=parsoid,service=canary [production]
11:13 <cgoubert@cumin1001> conftool action : set/weight=10; selector: dc=eqiad,cluster=parsoid,service=canary [production]
11:13 <claime> Fixing appserver clusters canary weights [production]
10:56 <jynus> deployed new ssh key for jcrespo on production cluster [production]
10:29 <claime> Datacenter switchover live testing setting db to read-only and back in eqiad successful - T327920 [production]
10:29 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) [production]
10:29 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite [production]
10:29 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) [production]
10:29 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly [production]
10:27 <claime> Datacenter switchover live testing setting db to read-only and back in eqiad - T327920 [production]
10:26 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Ilooremeta out of all services on: 801 hosts [production]
10:26 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Ilooremeta out of all services on: 801 hosts [production]
10:24 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Ilooremeta out of all services on: 1262 hosts [production]
10:22 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Ilooremeta out of all services on: 1262 hosts [production]
10:22 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Hghani out of all services on: 1262 hosts [production]
10:20 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Hghani out of all services on: 1262 hosts [production]
10:18 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Hghani out of all services on: 801 hosts [production]
10:18 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Hghani out of all services on: 801 hosts [production]
10:17 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Hibashaath out of all services on: 801 hosts [production]
10:17 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Hibashaath out of all services on: 801 hosts [production]
10:16 <jmm@cumin2002> END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Hibashaath out of all services on: 1262 hosts [production]
10:14 <jmm@cumin2002> START - Cookbook sre.idm.logout Logging Hibashaath out of all services on: 1262 hosts [production]
10:11 <marostegui> Enable replication eqiad -> codfw on s1 dbmaint eqiad T335266 [production]
10:10 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 38 hosts with reason: Enabling replication T335266 [production]
10:09 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on 38 hosts with reason: Enabling replication T335266 [production]
10:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 35 hosts with reason: Enabling replication T335266 [production]
10:07 <marostegui> Enable replication eqiad -> codfw on s4 dbmaint eqiad T335266 [production]
10:07 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on 35 hosts with reason: Enabling replication T335266 [production]
10:07 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 24 hosts with reason: Enabling replication T335266 [production]
10:06 <marostegui> Enable replication eqiad -> codfw on s3 dbmaint eqiad T335266 [production]
10:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on 24 hosts with reason: Enabling replication T335266 [production]
10:01 <moritzm> installing git security updates [production]
09:55 <slyngs> Update LDAP schema wmf-user: T148048 [production]
09:55 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 28 hosts with reason: Enabling replication T335266 [production]
09:55 <marostegui> Enable replication eqiad -> codfw on s7 dbmaint eqiad T335266 [production]
09:54 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on 28 hosts with reason: Enabling replication T335266 [production]
09:25 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host an-worker1110.eqiad.wmnet [production]
09:21 <moritzm> upgrade php-excimer on mw canaries to 1.0.2-1+wmf3+buster1 (which rebases Excimer to 1.1.1) T332964 [production]
08:45 <moritzm> uploaded php-excimer 1.0.2-1+wmf3+buster1 (which rebases Excimer to 1.1.1) to component/php74 for buster-wikimedia T332964 [production]
08:44 <marostegui> Enable replication eqiad -> codfw on s8 dbmaint eqiad T335266 [production]
08:44 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 34 hosts with reason: Enabling replication T335266 [production]