2501-2550 of 10000 results (124ms)
2024-06-26 ยง
21:29 <cjming@deploy1002> Started scap: Backport for [[gerrit:1050005|Homepage: don't load yesterdays edits on desktop (T368405)]] [production]
21:29 <hashar> restarting CI Jenkins [production]
21:13 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host cp5020.eqsin.wmnet with OS bullseye [production]
21:13 <brett@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5020.eqsin.wmnet with OS bullseye [production]
21:05 <cjming@deploy1002> Finished scap: Backport for [[gerrit:1050002|Homepage: log rendering time for each module and each wiki (T368405)]] (duration: 14m 01s) [production]
20:59 <eileen> config revision changed from 0b822cd3 to 994e7b81 [production]
20:57 <cjming@deploy1002> cjming, migr: Continuing with sync [production]
20:55 <cjming@deploy1002> cjming, migr: Backport for [[gerrit:1050002|Homepage: log rendering time for each module and each wiki (T368405)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:51 <cjming@deploy1002> Started scap: Backport for [[gerrit:1050002|Homepage: log rendering time for each module and each wiki (T368405)]] [production]
20:50 <jdrewniak@deploy1002> Finished scap: Backport for [[gerrit:1049972|Enable user pages and select special pages in dark mode (1.43.0-wmf.11) (T366364 T366375 T367375 T367581 T367582 T367583)]] (duration: 08m 09s) [production]
20:47 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host cp5020.eqsin.wmnet with OS bullseye [production]
20:45 <jdrewniak@deploy1002> jdlrobson, jdrewniak: Continuing with sync [production]
20:44 <jdrewniak@deploy1002> jdlrobson, jdrewniak: Backport for [[gerrit:1049972|Enable user pages and select special pages in dark mode (1.43.0-wmf.11) (T366364 T366375 T367375 T367581 T367582 T367583)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:42 <jdrewniak@deploy1002> Started scap: Backport for [[gerrit:1049972|Enable user pages and select special pages in dark mode (1.43.0-wmf.11) (T366364 T366375 T367375 T367581 T367582 T367583)]] [production]
20:40 <jdrewniak@deploy1002> Synchronized portals: Wikimedia Portals Update: [[gerrit:1050007| Bumping portals to master (T128546)]] (duration: 06m 58s) [production]
20:33 <jdrewniak@deploy1002> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:1050007| Bumping portals to master (T128546)]] (duration: 07m 27s) [production]
20:28 <brett@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp5020.eqsin.wmnet [production]
20:21 <cjming@deploy1002> Finished scap: Backport for [[gerrit:1049947|Update QuickSurvey coverage rate for Automoderator patroller workstream survey (T362969)]] (duration: 08m 46s) [production]
20:15 <cjming@deploy1002> cjming, kgraessle: Continuing with sync [production]
20:14 <cjming@deploy1002> cjming, kgraessle: Backport for [[gerrit:1049947|Update QuickSurvey coverage rate for Automoderator patroller workstream survey (T362969)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:12 <cjming@deploy1002> Started scap: Backport for [[gerrit:1049947|Update QuickSurvey coverage rate for Automoderator patroller workstream survey (T362969)]] [production]
20:08 <mutante> lists1001:/lib/systemd/system# rm wmf_auto_restart_apache2.* ; systemctl reset-failed - reaction to monitoring alert "FIRING: SystemdUnitFailed: wmf_auto_restart_apache2.service on lists1001:9100" [production]
20:08 <brett@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp5019.eqsin.wmnet [production]
20:05 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5019.eqsin.wmnet with OS bullseye [production]
19:48 <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.43.0-wmf.11 refs T366956 [production]
19:40 <jhathaway@deploy1002> Finished scap: (no justification provided) (duration: 02m 38s) [production]
19:39 <jhathaway@deploy1002> Started scap: (no justification provided) [production]
19:33 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5019.eqsin.wmnet with reason: host reimage [production]
19:28 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp5019.eqsin.wmnet with reason: host reimage [production]
19:18 <cmooney@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:16 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
19:11 <ottomata> re-enabling varnishkafka-eventlogging and varnish /beacon/event handling on cache text nodes. /beacon/event/ redirects which breaks the MediaWikiPingback usage - T238230 [production]
19:02 <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.43.0-wmf.11 refs T366956 [production]
18:56 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host cp5019.eqsin.wmnet with OS bullseye [production]
18:55 <brett@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5019.eqsin.wmnet with OS bullseye [production]
18:26 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:26 <sukhe@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove ntp.anycast.wmnet - sukhe@cumin1002" [production]
18:25 <sukhe@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove ntp.anycast.wmnet - sukhe@cumin1002" [production]
18:24 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1211 (T364069)', diff saved to https://phabricator.wikimedia.org/P65490 and previous config saved to /var/cache/conftool/dbconfig/20240626-182355-marostegui.json [production]
18:23 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance [production]
18:23 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance [production]
18:23 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209 (T364069)', diff saved to https://phabricator.wikimedia.org/P65489 and previous config saved to /var/cache/conftool/dbconfig/20240626-182333-marostegui.json [production]
18:23 <sukhe@cumin1002> START - Cookbook sre.dns.netbox [production]
18:19 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host cp5019.eqsin.wmnet with OS bullseye [production]
18:17 <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.43.0-wmf.11 refs T366956 [production]
18:14 <sukhe> # etcdctl --username root --endpoints https://conf1007.eqiad.wmnet:4001 rmdir /conftool/v1/pools/${site}/dnsbox/ntp: T366360 [production]
18:12 <brett@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp5019.eqsin.wmnet [production]
18:08 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P65488 and previous config saved to /var/cache/conftool/dbconfig/20240626-180824-marostegui.json [production]
18:07 <xcollazo@deploy1002> Finished deploy [airflow-dags/analytics@5121748]: Deploying latest DAGs to analytics Airflow instance. (duration: 00m 39s) [production]
18:06 <xcollazo@deploy1002> Started deploy [airflow-dags/analytics@5121748]: Deploying latest DAGs to analytics Airflow instance. [production]