3001-3050 of 10000 results (26ms)
2020-09-17 §
05:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es2015 after cloning es2031 T261717', diff saved to https://phabricator.wikimedia.org/P12615 and previous config saved to /var/cache/conftool/dbconfig/20200917-050739-marostegui.json [production]
04:53 <marostegui> Deploy schema change on s1 eqiad primary master - T238966 [production]
01:22 <Krinkle> krinkle@mwmaint1002 synced docroot/noc – https://gerrit.wikimedia.org/r/620138 [production]
01:22 <Krinkle> krinkle@mwmaint2001 synced docroot/noc – https://gerrit.wikimedia.org/r/620138 [production]
2020-09-16 §
23:41 <catrope@deploy1001> Synchronized php-1.36.0-wmf.8/extensions/FlaggedRevs: T262970 (duration: 01m 06s) [production]
23:40 <catrope@deploy1001> Synchronized php-1.36.0-wmf.9/extensions/FlaggedRevs: T262970 (duration: 01m 06s) [production]
23:37 <catrope@deploy1001> Synchronized php-1.36.0-wmf.9/extensions/GrowthExperiments/: Fix styling for mobile start module (T258008); Revert wider task card on desktop (T263042, T258704); Fix width of sidebar modules in narrow mode in variant A (T263068) (duration: 01m 09s) [production]
22:24 <shdubsh> install prometheus-icinga-exporter 0.11 on icinga2001 [production]
20:19 <cdanis@cumin1001> END (PASS) - Cookbook sre.network.cf (exit_code=0) [production]
20:19 <cdanis@cumin1001> START - Cookbook sre.network.cf [production]
20:10 <robh@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
20:04 <robh@cumin1001> START - Cookbook sre.dns.netbox [production]
18:14 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable Vector search in header on testwiki and officewiki (T262207) (duration: 01m 04s) [production]
18:00 <brennen@deploy1001> Synchronized php-1.36.0-wmf.9/extensions/MobileFrontend: Backport: [[gerrit:627793|Check $coords matched some nodes before comparing contents (T263034)]] (duration: 01m 06s) [production]
17:58 <joal@deploy1001> Finished deploy [analytics/refinery@07056b0] (thin): Regular analytics weekly train THIN [analytics/refinery@07056b0] (duration: 00m 08s) [production]
17:58 <joal@deploy1001> Started deploy [analytics/refinery@07056b0] (thin): Regular analytics weekly train THIN [analytics/refinery@07056b0] [production]
17:51 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
17:50 <joal@deploy1001> Started deploy [analytics/refinery@07056b0]: Regular analytics weekly train [analytics/refinery@07056b0] [production]
17:15 <hnowlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
17:11 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:03 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
16:45 <hnowlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
16:40 <hnowlan@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
16:32 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
16:13 <marostegui> Start mysql on db1093, db1109 and db1123 after pdu work is done [production]
16:12 <ryankemper> `wdqs` deploy complete, service is healthy [production]
16:09 <elukey> reinstall buster on an-tool1009 after a lot of tests (ganeti VM, so it is a manual work) [production]
16:00 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
15:58 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
15:49 <ryankemper> sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 60 && systemctl restart wdqs-categories && sleep 30 && pool'; sudo -E cumin 'A:wdqs-test' 'systemctl restart wdqs-categories' [production]
15:49 <ryankemper> `sudo -E cumin -b 4 'A:wdqs-all' 'systemctl restart wdqs-updater'` [production]
15:48 <ryankemper@deploy1001> Finished deploy [wdqs/wdqs@b7e2d0b]: 0.3.48 (duration: 14m 40s) [production]
15:37 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:627871|Rename wmgWikibaseClientLocalEntitySourceName to wmgWikibaseClientItemAndPropertySourceName on Beta (T258060)]] (production no-op) (duration: 01m 04s) [production]
15:35 <ryankemper> Canary `wdqs1003` query tests looks good, proceeding to wdqs deploy for rest of fleet [production]
15:33 <ryankemper@deploy1001> Started deploy [wdqs/wdqs@b7e2d0b]: 0.3.48 [production]
15:33 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:622994|Remove `wmgWikibaseClientLocalEntitySourceName` from InitialiseSettings.php (T258060)]] (duration: 01m 05s) [production]
15:27 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:622993|Use `wmgWikibaseClientItemAndPropertySourceName` instead of `wmgWikibaseClientLocalEntitySourceName` in Wikibase.php (T258060)]] (duration: 01m 02s) [production]
15:21 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:622612|Add `wmgWikibaseClientItemAndPropertySourceName` to InitialiseSettings.php (T258060)]] (duration: 01m 06s) [production]
14:47 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) [production]
14:41 <volans> uploaded spicerack_0.0.43 to apt.wikimedia.org buster-wikimedia [production]
14:39 <cmjohnson1> pdu swap rack d7-eqiad, missed this in earlier log entry [production]
14:34 <jiji@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
14:02 <Urbanecm> Change email address of User:Oversight@enwiki to oversight-en-wp@wikipedia.org as OTRS is back up (T262733) [production]
13:48 <marostegui> Start mysql on db1121 after PDU work [production]
13:46 <James_F> Restarting CI Jenkins for T262827 [production]
13:08 <jmm@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=mw2256.codfw.wmnet [production]
13:08 <liw@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.9 [production]
12:58 <elukey> upload hue_4.7.1-1+deb10u1 to buster-wikimedia [production]
12:56 <cdanis@cumin1001> END (PASS) - Cookbook sre.network.cf (exit_code=0) [production]
12:56 <cdanis@cumin1001> START - Cookbook sre.network.cf [production]