701-750 of 10000 results (24ms)
2020-10-09 §
07:34 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
00:16 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
00:00 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
2020-10-08 §
23:42 <ryankemper> `cloudelastic1006` done. Writes thawed, maintenance window lifted; restarts are done for `cloudelastic` [production]
23:37 <ryankemper> `cloudelastic1005` done [production]
23:31 <ryankemper> `cloudelastic1004` done [production]
23:27 <ryankemper> `cloudelastic1003` done [production]
23:23 <ryankemper> `cloudelastic1002` done [production]
23:16 <tgr_> Evening deploys done [production]
23:16 <ryankemper> `cloudelastic1001` is done restarting and cluster is green again. Proceeding to `cloudelastic1002` [production]
23:16 <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:632797|Enable logging of session cookie changes everywhere (T264793)]] (duration: 01m 01s) [production]
23:04 <ryankemper> Beginning cluster restarts one server at a time. For each server, the process is depool->restart elasticsearch services->wait for services to restart and then pool->wait for cluster to return to green status before starting next server [production]
23:01 <ryankemper> Writes are frozen for `cloudelastic`: `/usr/local/bin/mwscript extensions/CirrusSearch/maintenance/FreezeWritesToCluster.php --wiki=enwiki --cluster=cloudelastic` on `mwmaint2001` => `Applied cluster-wide freeze` [production]
22:56 <ryankemper> `sudo apt policy wmf-elasticsearch-search-plugins` shows correct state: `Installed: 6.5.4-4~stretch` [production]
22:56 <ryankemper> `sudo -E cumin -b 6 C:role::elasticsearch::cloudelastic 'DEBIAN_FRONTEND=noninteractive sudo apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install wmf-elasticsearch-search-plugins'` [production]
22:54 <ryankemper> About to start plugin upgrade followed by restarts of `cloudelastic`. Maintenance window set for the next 2 hours on `cloudelastic100[1-6]` [production]
21:54 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@a923949]: search_satisfaction: update druid datasource to match previous data (duration: 01m 04s) [production]
21:53 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@a923949]: search_satisfaction: update druid datasource to match previous data [production]
21:52 <hashar@deploy1001> Synchronized php-1.36.0-wmf.10/includes/session/SessionBackend.php: Deduplicate SessionBackend::logPersistenceChange calls - T264793 (duration: 01m 01s) [production]
21:05 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:00 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
21:00 <volans@cumin1001> END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [production]
21:00 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
20:50 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:45 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
20:43 <volans> deploying Netbox DNS zone consolidation - T264273 [production]
20:11 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:09 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:24 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@3b11443]: search_satisfaction: Alias sample multiplier to expected name (duration: 01m 09s) [production]
19:23 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@3b11443]: search_satisfaction: Alias sample multiplier to expected name [production]
18:57 <volker-e@deploy1001> Finished deploy [design/style-guide@b1166af]: Deploy design/style-guide: (duration: 00m 06s) [production]
18:57 <volker-e@deploy1001> Started deploy [design/style-guide@b1166af]: Deploy design/style-guide: [production]
18:17 <tchanders@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:632908|Enable Special:Investigate by default on production (T264357)]] (duration: 01m 06s) [production]
17:50 <root@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:49 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@945e5c1]: airflow: Set search satisfaction dag start date to oldest current available data (duration: 11m 55s) [production]
17:44 <root@cumin1001> START - Cookbook sre.dns.netbox [production]
17:37 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@945e5c1]: airflow: Set search satisfaction dag start date to oldest current available data [production]
17:31 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
17:30 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:27 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
17:26 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
17:23 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
17:16 <shdubsh> install prometheus-rsyslog-exporter_0.0.0+git20201008 on centrallog1001 - T210137 [production]
16:25 <mutante> rebooting cloudvirt1023 - trying PXE boot [production]
16:19 <hashar> Restarting CI Jenkins [production]
16:15 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:09 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
16:08 <volans@cumin1001> END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [production]
16:08 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
14:21 <marostegui> Set global innodb_change_buffering = all; on pc2009 T263443 [production]