1351-1400 of 10000 results (50ms)
2022-03-17 §
04:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1146:3314 (T300775)', diff saved to https://phabricator.wikimedia.org/P22737 and previous config saved to /var/cache/conftool/dbconfig/20220317-040634-marostegui.json [production]
04:06 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance [production]
04:06 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1146.eqiad.wmnet with reason: Maintenance [production]
02:57 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1016.eqiad.wmnet with OS bullseye [production]
02:07 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1016.eqiad.wmnet with OS bullseye [production]
02:07 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1016.eqiad.wmnet with OS bullseye [production]
01:11 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1016.eqiad.wmnet with OS bullseye [production]
2022-03-16 §
23:52 <tzatziki> Removing two files for legal compliance [production]
21:17 <cjming> end running skin update preference maintenance script [production]
20:52 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:40 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: [no-op] 8efa537: GrowthExperiments: Set GEWelcomeSurveyShowMailingListQuestion (T303240) (duration: 00m 53s) [production]
20:38 <robh@cumin1001> START - Cookbook sre.hosts.provision for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:35 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.26/extensions/WikimediaMaintenance/: 9ba157b: Add insert option for update skin preferences script (T299104) (duration: 00m 50s) [production]
20:34 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.25/extensions/WikimediaMaintenance/: ebfc516: Add script to update vector skin preferences (T299104) (duration: 00m 51s) [production]
20:32 <robh@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:24 <pt1979@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1025.eqiad.wmnet with OS bullseye [production]
20:13 <robh@cumin1001> START - Cookbook sre.hosts.provision for host dumpsdata1006.mgmt.eqiad.wmnet with reboot policy FORCED [production]
20:13 <urbanecm@deploy1002> Synchronized docroot/noc/db.php: f649199: Migrate wmfDatacenter(s) to wmgDatacenter(s) (T45956; 3/3) (duration: 00m 49s) [production]
20:12 <urbanecm@deploy1002> Synchronized multiversion/: f649199: Migrate wmfDatacenter(s) to wmgDatacenter(s) (T45956; 2/3) (duration: 00m 50s) [production]
20:11 <urbanecm@deploy1002> Synchronized wmf-config/: f649199: Migrate wmfDatacenter(s) to wmgDatacenter(s) (T45956; 1/3) (duration: 00m 50s) [production]
19:22 <otto@deploy1002> Finished deploy [analytics/refinery@2d2056a] (hadoop-test): (no justification provided) (duration: 07m 50s) [production]
19:14 <otto@deploy1002> Started deploy [analytics/refinery@2d2056a] (hadoop-test): (no justification provided) [production]
18:32 <sukhe> running: homer "cr*-drmrs*" commit "Gerrit 771359: Set up BGP peering in drmrs for Wikidough." [production]
18:09 <aqu@deploy1002> Finished deploy [airflow-dags/analytics_test@257960f]: Migrate session_length/daily from Oozie to Airflow [airflow-dags/analytics_test@257960f] (duration: 00m 08s) [production]
18:09 <aqu@deploy1002> Started deploy [airflow-dags/analytics_test@257960f]: Migrate session_length/daily from Oozie to Airflow [airflow-dags/analytics_test@257960f] [production]
18:02 <aqu@deploy1002> Finished deploy [airflow-dags/analytics@257960f]: Migrate session_length/daily from Oozie to Airflow [airflow-dags/analytics@257960f] (duration: 00m 08s) [production]
18:02 <aqu@deploy1002> Started deploy [airflow-dags/analytics@257960f]: Migrate session_length/daily from Oozie to Airflow [airflow-dags/analytics@257960f] [production]
18:00 <razzi@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on karapace1001.eqiad.wmnet with reason: Setting up karapace for the first time [production]
18:00 <razzi@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on karapace1001.eqiad.wmnet with reason: Setting up karapace for the first time [production]
17:36 <dancy@deploy1002> Synchronized multiversion/MWMultiVersion.php: Config: [[gerrit:771001|mwscript: Support --force-version flag (T303878)]] (duration: 00m 57s) [production]
17:21 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=drmrs,cluster=cache_text,service=ats-tls [production]
17:21 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=drmrs,cluster=cache_text,service=varnish-fe [production]
17:21 <sukhe@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=drmrs,cluster=cache_text,service=ats-be [production]
17:13 <aqu@deploy1002> Finished deploy [analytics/refinery@d039471] (hadoop-test): Migrate session_length/daily from Oozie to Airflow [analytics/refinery@d039471] (duration: 07m 23s) [production]
17:11 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp6016.drmrs.wmnet with OS buster [production]
17:06 <aqu@deploy1002> Started deploy [analytics/refinery@d039471] (hadoop-test): Migrate session_length/daily from Oozie to Airflow [analytics/refinery@d039471] [production]
17:06 <aqu@deploy1002> Finished deploy [analytics/refinery@d039471] (thin): Migrate session_length/daily from Oozie to Airflow [analytics/refinery@d039471] (duration: 00m 07s) [production]
17:06 <aqu@deploy1002> Started deploy [analytics/refinery@d039471] (thin): Migrate session_length/daily from Oozie to Airflow [analytics/refinery@d039471] [production]
17:04 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
17:04 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
16:48 <aqu@deploy1002> Finished deploy [analytics/refinery@d039471]: Migrate session_length/daily from Oozie to Airflow [analytics/refinery@d039471] (duration: 25m 49s) [production]
16:45 <pt1979@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1024.eqiad.wmnet with OS bullseye [production]
16:36 <Emperor> rolling restart of ms-fe10[09-12] so they know about removal of older proxies T303733 [production]
16:30 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp6016.drmrs.wmnet with reason: host reimage [production]
16:28 <Emperor> moving swiftrepl and stats reporter host from ms-fe1005 to ms-fe1009 T303733 [production]
16:27 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp6016.drmrs.wmnet with reason: host reimage [production]
16:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
16:27 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance [production]
16:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1113:3315 (T297189)', diff saved to https://phabricator.wikimedia.org/P22734 and previous config saved to /var/cache/conftool/dbconfig/20220316-162721-marostegui.json [production]
16:22 <aqu@deploy1002> Started deploy [analytics/refinery@d039471]: Migrate session_length/daily from Oozie to Airflow [analytics/refinery@d039471] [production]