3401-3450 of 10000 results (41ms)
2024-08-01 §
18:10 <brennen@deploy1003> rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.16 refs T366961 [production]
18:00 <brennen> 1.43.0-wmf.16 train (T366961): no current blockers, logs cluttered but not too scary, rolling to all wikis. [production]
07:47 <ayounsi@cumin1002> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: netbox 4 sync - ayounsi@cumin1002" [production]
07:39 <ayounsi@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: netbox 4 sync - ayounsi@cumin1002" [production]
2024-07-31 §
20:12 <cjming@deploy1003> Finished scap: Backport for [[gerrit:1056495|[wmf-config] Remove trailing slash in SSO domain]] (duration: 08m 04s) [production]
20:06 <cjming@deploy1003> cjming, d3r1ck01: Backport for [[gerrit:1056495|[wmf-config] Remove trailing slash in SSO domain]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:04 <cjming@deploy1003> Started scap sync-world: Backport for [[gerrit:1056495|[wmf-config] Remove trailing slash in SSO domain]] [production]
18:17 <brennen@deploy1003> rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.16 refs T366961 [production]
18:09 <brennen> 1.43.0-wmf.16 train (T366961): no current blockers, logs clean, rolling to group1. [production]
2024-07-30 §
18:53 <brennen@deploy1003> rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.16 refs T366961 [production]
18:33 <brennen> 1.43.0-wmf.16 train (T366961): blockers resolved, rolling to group0 [production]
04:07 <mwpresync@deploy1003> Pruned MediaWiki: 1.43.0-wmf.13 (duration: 06m 51s) [production]
03:02 <mwpresync@deploy1003> Started scap sync-world: testwikis to 1.43.0-wmf.16 refs T366961 [production]
2024-07-29 §
13:45 <logmsgbot> lucaswerkmeister-wmde@deploy1003 Synchronized php-1.43.0-wmf.15/extensions/ContentTranslation/extension.json: Backport for [[gerrit:1057853|AX: Unregister "axArticleFooterEntrypointRegistrar" hook handler (T363338)]] (duration: 06m 36s) [production]
13:24 <logmsgbot> lucaswerkmeister-wmde@deploy1003 Synchronized wmf-config/: Backport for [[gerrit:1055434|Enable mul language code on Wikidata (limited mode) (T330281)]] (duration: 06m 47s) [production]
2024-07-25 §
18:12 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.15 refs T366960 [production]
17:56 <ayounsi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on netbox2003.codfw.wmnet,netbox1003.eqiad.wmnet with reason: netbox upgrade prep work [production]
17:56 <ayounsi@cumin1002> START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on netbox2003.codfw.wmnet,netbox1003.eqiad.wmnet with reason: netbox upgrade prep work [production]
17:06 <swfrench-wmf> running homer 'cr*eqiad*' commit 'T351074' for k8s worker reimage [production]
2024-07-24 §
18:10 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 to 1.43.0-wmf.15 refs T366960 [production]
2024-07-23 §
18:14 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group0 to 1.43.0-wmf.15 refs T366960 [production]
18:13 <swfrench-wmf> sudo cumin 'A:lvs-secondary-eqiad or A:lvs-low-traffic-eqiad' 'ipvsadm --delete-service --tcp-service 10.2.2.1:443' (appservers-https eqiad) - T367949 [production]
18:11 <swfrench-wmf> sudo cumin 'A:lvs-secondary-eqiad or A:lvs-low-traffic-eqiad' 'ipvsadm --delete-service --tcp-service 10.2.2.22:443' (api-https eqiad) - T367949 [production]
18:11 <swfrench-wmf> sudo cumin 'A:lvs-secondary-eqiad or A:lvs-low-traffic-eqiad' 'ipvsa [production]
18:10 <swfrench-wmf> sudo cumin 'A:lvs-secondary-codfw or A:lvs-low-traffic-codfw' 'ipvsa [production]
18:08 <swfrench-wmf> sudo cumin 'A:lvs-secondary-codfw or A:lvs-low-traffic-codfw' 'ipvsa [production]
17:58 <swfrench-wmf> sudo cumin 'A:lvs-low-traffic-eqiad' 'systemctl restart pybal.service' - T367949 [production]
17:51 <swfrench-wmf> sudo cumin 'A:lvs-secondary-eqiad' 'systemctl restart pybal.service' - T367949 [production]
17:46 <logmsgbot> nshahquinn-wmf@deploy1002 Finished deploy [airflow-dags/analytics_product@ebd9e13]: (no justification provided) (duration: 00m 07s) [production]
17:46 <logmsgbot> nshahquinn-wmf@deploy1002 Started deploy [airflow-dags/analytics_product@ebd9e13]: (no justification provided) [production]
17:44 <swfrench-wmf> sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service' - T367949 [production]
17:28 <swfrench-wmf> run-puppet-agent on O:lvs::balancer to pick up switch to service_setup, removal of profile::lvs::realserver::pools - T367949 [production]
17:17 <swfrench-wmf> run-puppet-agent on A:dnsbox to pick up switch to lvs_setup - T367949 [production]
17:06 <swfrench-wmf> ran authdns-update on dns1004 to pick up removal of appservers / api records - T367949 [production]
04:01 <mwpresync@deploy1002> Pruned MediaWiki: 1.43.0-wmf.12 (duration: 01m 00s) [production]
03:54 <mwpresync@deploy1002> Finished scap: testwikis to 1.43.0-wmf.15 refs T366960 (duration: 51m 50s) [production]
03:03 <mwpresync@deploy1002> Started scap sync-world: testwikis to 1.43.0-wmf.15 refs T366960 [production]
2024-07-18 §
22:19 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on elastic1100.eqiad.wmnet with reason: catch up on indexing [production]
22:19 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 4:00:00 on elastic1100.eqiad.wmnet with reason: catch up on indexing [production]
21:01 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.14 refs T366959 [production]
18:17 <swfrench-wmf> api-ro.discovery.wmnet now resolves to failoid - T367949 [production]
18:03 <swfrench-wmf> appservers-ro.discovery.wmnet now resolves to failoid - T367949 [production]
08:13 <aklapper@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.43.0-wmf.14 refs T366959 [production]
2024-07-17 §
23:16 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2220.codfw.wmnet with reason: Maintenance [production]
23:15 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2220.codfw.wmnet with reason: Maintenance [production]
22:26 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2208.codfw.wmnet with reason: Maintenance [production]
22:26 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2208.codfw.wmnet with reason: Maintenance [production]
22:26 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2200.codfw.wmnet with reason: Maintenance [production]
22:26 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2200.codfw.wmnet with reason: Maintenance [production]
22:25 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2198.codfw.wmnet with reason: Maintenance [production]