3551-3600 of 10000 results (32ms)
2020-08-27 §
09:15 <kormat> enabling replication from db2079 to db1109 (s8) T243373 [production]
09:07 <kormat> enabling replication from db2090 to db1081 (s4) T243373 [production]
08:53 <kormat> enabling replication from pc2009 to pc1009 (pc3) T243373 [production]
08:44 <kormat> enabling replication from pc2008 to pc1008 (pc2) T243373 [production]
08:13 <marostegui> Enable replication codfw -> eqiad on pc1 T243373 [production]
08:01 <gehel> manual cleanup of stale wdqs deploy crontab on wdqs1009 [production]
07:35 <marostegui> Move pc2010 under pc2007 T243373 [production]
07:16 <moritzm> installing ghostscript security updates on stretch [production]
06:50 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
06:49 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:46 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:45 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:34 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:31 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1079,db1082', diff saved to https://phabricator.wikimedia.org/P12392 and previous config saved to /var/cache/conftool/dbconfig/20200827-060652-marostegui.json [production]
05:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1079,db1082', diff saved to https://phabricator.wikimedia.org/P12391 and previous config saved to /var/cache/conftool/dbconfig/20200827-055815-marostegui.json [production]
05:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1079,db1082', diff saved to https://phabricator.wikimedia.org/P12390 and previous config saved to /var/cache/conftool/dbconfig/20200827-055522-marostegui.json [production]
05:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1079', diff saved to https://phabricator.wikimedia.org/P12389 and previous config saved to /var/cache/conftool/dbconfig/20200827-055126-marostegui.json [production]
05:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1082', diff saved to https://phabricator.wikimedia.org/P12388 and previous config saved to /var/cache/conftool/dbconfig/20200827-055104-marostegui.json [production]
05:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P12387 and previous config saved to /var/cache/conftool/dbconfig/20200827-054259-marostegui.json [production]
05:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1074 db1085 db1078', diff saved to https://phabricator.wikimedia.org/P12386 and previous config saved to /var/cache/conftool/dbconfig/20200827-054114-marostegui.json [production]
05:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1082', diff saved to https://phabricator.wikimedia.org/P12385 and previous config saved to /var/cache/conftool/dbconfig/20200827-053814-marostegui.json [production]
05:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1134 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12384 and previous config saved to /var/cache/conftool/dbconfig/20200827-053558-marostegui.json [production]
05:35 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1074, db1085, db1079', diff saved to https://phabricator.wikimedia.org/P12383 and previous config saved to /var/cache/conftool/dbconfig/20200827-053509-marostegui.json [production]
05:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1074, db1085, db1079', diff saved to https://phabricator.wikimedia.org/P12382 and previous config saved to /var/cache/conftool/dbconfig/20200827-053100-marostegui.json [production]
05:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1134', diff saved to https://phabricator.wikimedia.org/P12381 and previous config saved to /var/cache/conftool/dbconfig/20200827-052925-marostegui.json [production]
05:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1085', diff saved to https://phabricator.wikimedia.org/P12380 and previous config saved to /var/cache/conftool/dbconfig/20200827-052818-marostegui.json [production]
05:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1074', diff saved to https://phabricator.wikimedia.org/P12379 and previous config saved to /var/cache/conftool/dbconfig/20200827-052413-marostegui.json [production]
05:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1079', diff saved to https://phabricator.wikimedia.org/P12378 and previous config saved to /var/cache/conftool/dbconfig/20200827-051609-marostegui.json [production]
05:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1134 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12377 and previous config saved to /var/cache/conftool/dbconfig/20200827-051546-marostegui.json [production]
05:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1134 after MCR changes', diff saved to https://phabricator.wikimedia.org/P12376 and previous config saved to /var/cache/conftool/dbconfig/20200827-050754-marostegui.json [production]
05:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1085', diff saved to https://phabricator.wikimedia.org/P12375 and previous config saved to /var/cache/conftool/dbconfig/20200827-050727-marostegui.json [production]
04:53 <marostegui> Stop db1074 and db2107 in sync to fix drifts on s2 change_tag - T260042 [production]
04:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074', diff saved to https://phabricator.wikimedia.org/P12374 and previous config saved to /var/cache/conftool/dbconfig/20200827-045329-marostegui.json [production]
04:04 <ryankemper@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: name=cloudelastic1006.wikimedia.org [production]
04:03 <ryankemper@puppetmaster1001> conftool action : set/pooled=yes:weight=10; selector: name=cloudelastic1005.wikimedia.org [production]
04:01 <ryankemper@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cloudelastic1005.wikimedia.org [production]
02:03 <mutante> shutting down install3001,install4001,install5001 VMs (no OS yet, but please also don't delete, debugging in progress, shutting them down until I continue on T254157) [production]
2020-08-26 §
23:35 <eileen> civicrm revision changed from d2e80f7522 to 481ab742db, config revision is e2ab4d7c1f [production]
23:00 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
22:36 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
22:30 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
22:26 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
22:20 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
19:51 <XioNoX> standardize pfw3-eqiad [production]
19:33 <marxarelli> 1.36.0-wmf.6 promoted to group1 (T257974). logs show no new errors [production]
19:24 <dduvall@deploy1001> Synchronized php: group1 wikis to 1.36.0-wmf.6 (duration: 01m 03s) [production]
19:23 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.36.0-wmf.6 [production]
18:21 <Urbanecm> Morning B&C done [production]
18:19 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 945b97cff8b8a1e4bb43b613fc93b099f74945f7: Added import sources for mlwiktionary (T260716) (duration: 01m 05s) [production]