851-900 of 10000 results (64ms)
2020-03-26 ยง
16:11 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
16:10 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:58 <volans@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:53 <volans@cumin1001> START - Cookbook sre.dns.netbox [production]
15:51 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
15:51 <moritzm> installing grub2 updates from Stretch point release [production]
15:49 <XioNoX> start advertising 208.80.154.0/23 from eqiad - T246721 [production]
15:49 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
15:48 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:46 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:40 <XioNoX> start advertising 2620:0:861::/48 from eqiad - T246721 [production]
15:20 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
15:17 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:15 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
15:12 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
15:10 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
15:02 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:02 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:02 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:02 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:02 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:02 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:02 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:02 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:01 <mutante> T247887 - create Ganeti VM miscweb1002.eqiad.wmnet in the ganeti01.svc.eqiad.wmnet cluster on row C with 1 vCPUs, 2GB of RAM, 20GB of disk in the private network. [production]
15:01 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:01 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:59 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
14:59 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
14:59 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
14:50 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
14:47 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
14:26 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
14:23 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1110 after schema change', diff saved to https://phabricator.wikimedia.org/P10787 and previous config saved to /var/cache/conftool/dbconfig/20200326-135625-marostegui.json [production]
13:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1110 for schema change', diff saved to https://phabricator.wikimedia.org/P10786 and previous config saved to /var/cache/conftool/dbconfig/20200326-132940-marostegui.json [production]
13:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1097:3315 after schema change', diff saved to https://phabricator.wikimedia.org/P10785 and previous config saved to /var/cache/conftool/dbconfig/20200326-130122-marostegui.json [production]
12:57 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: eventgate-main to use envoy T244843 (duration: 01m 07s) [production]
12:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1097:3315 for schema change', diff saved to https://phabricator.wikimedia.org/P10784 and previous config saved to /var/cache/conftool/dbconfig/20200326-123302-marostegui.json [production]
12:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1096:3315 after schema change', diff saved to https://phabricator.wikimedia.org/P10783 and previous config saved to /var/cache/conftool/dbconfig/20200326-123157-marostegui.json [production]
12:25 <mutante> analytics1028 - performing a puppet change on every run (all other hosts doing this were fixed just recently) [production]
12:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1096:3315 for schema change', diff saved to https://phabricator.wikimedia.org/P10782 and previous config saved to /var/cache/conftool/dbconfig/20200326-121859-marostegui.json [production]
11:38 <awight> EU SWAT done [production]
11:37 <awight@deploy1001> Synchronized php-1.35.0-wmf.25/extensions/TwoColConflict: SWAT: [[gerrit:583576|Two hotfixes for guided tour (T248465)]] (duration: 01m 07s) [production]
11:25 <mutante> sodium - running ftpsync to get Debian mirror in sync [production]
11:23 <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T231517: [cirrus] force cloudelastic replica count to 1 (duration: 01m 05s) [production]
11:21 <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T231517: [cirrus] force cloudelastic replica count to 1 (duration: 01m 06s) [production]
11:12 <urbanecm@deploy1001> Synchronized php-1.35.0-wmf.25/extensions/ContentTranslation/modules/ui/mw.cx.ui.Categories.js: SWAT: 1ea6bad: Allow publishing to continue even with broken categories (T248302) (duration: 01m 07s) [production]
11:06 <urbanecm@deploy1001> Synchronized wmf-config/throttle.php: SWAT: d1bb0b1: Removed expired throttle.php entries (duration: 01m 09s) [production]
11:00 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]