2451-2500 of 10000 results (65ms)
2020-03-26 ยง
15:10 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
15:02 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:02 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:02 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:02 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:02 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:02 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:02 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:02 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:01 <mutante> T247887 - create Ganeti VM miscweb1002.eqiad.wmnet in the ganeti01.svc.eqiad.wmnet cluster on row C with 1 vCPUs, 2GB of RAM, 20GB of disk in the private network. [production]
15:01 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:01 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:59 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
14:59 <dzahn@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
14:59 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
14:50 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
14:47 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
14:26 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
14:23 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1110 after schema change', diff saved to https://phabricator.wikimedia.org/P10787 and previous config saved to /var/cache/conftool/dbconfig/20200326-135625-marostegui.json [production]
13:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1110 for schema change', diff saved to https://phabricator.wikimedia.org/P10786 and previous config saved to /var/cache/conftool/dbconfig/20200326-132940-marostegui.json [production]
13:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1097:3315 after schema change', diff saved to https://phabricator.wikimedia.org/P10785 and previous config saved to /var/cache/conftool/dbconfig/20200326-130122-marostegui.json [production]
12:57 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: eventgate-main to use envoy T244843 (duration: 01m 07s) [production]
12:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1097:3315 for schema change', diff saved to https://phabricator.wikimedia.org/P10784 and previous config saved to /var/cache/conftool/dbconfig/20200326-123302-marostegui.json [production]
12:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1096:3315 after schema change', diff saved to https://phabricator.wikimedia.org/P10783 and previous config saved to /var/cache/conftool/dbconfig/20200326-123157-marostegui.json [production]
12:25 <mutante> analytics1028 - performing a puppet change on every run (all other hosts doing this were fixed just recently) [production]
12:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1096:3315 for schema change', diff saved to https://phabricator.wikimedia.org/P10782 and previous config saved to /var/cache/conftool/dbconfig/20200326-121859-marostegui.json [production]
11:38 <awight> EU SWAT done [production]
11:37 <awight@deploy1001> Synchronized php-1.35.0-wmf.25/extensions/TwoColConflict: SWAT: [[gerrit:583576|Two hotfixes for guided tour (T248465)]] (duration: 01m 07s) [production]
11:25 <mutante> sodium - running ftpsync to get Debian mirror in sync [production]
11:23 <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T231517: [cirrus] force cloudelastic replica count to 1 (duration: 01m 05s) [production]
11:21 <dcausse@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T231517: [cirrus] force cloudelastic replica count to 1 (duration: 01m 06s) [production]
11:12 <urbanecm@deploy1001> Synchronized php-1.35.0-wmf.25/extensions/ContentTranslation/modules/ui/mw.cx.ui.Categories.js: SWAT: 1ea6bad: Allow publishing to continue even with broken categories (T248302) (duration: 01m 07s) [production]
11:06 <urbanecm@deploy1001> Synchronized wmf-config/throttle.php: SWAT: d1bb0b1: Removed expired throttle.php entries (duration: 01m 09s) [production]
11:00 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
10:58 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
10:54 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop' for release 'production' . [production]
10:16 <XioNoX> esams/knams: advertise 185.15.58.0/23 instead of 185.15.56.0/22 - T207753 [production]
09:50 <elukey> reboot stat1008 - gpu + drivers in a weird state after multiple tests [production]
09:00 <XioNoX> push v4 conditional advertising on cr3-knams - T236785 [production]
08:44 <marostegui> Deploy schema change on s5 codfw, lag will show up on codfw - T248333 [production]
08:27 <XioNoX> troubleshot v6 conditional advertisement from cr3-knams - T236785 [production]
07:58 <XioNoX> remove BGP session to AS8001 in eqiad (down and not replying to email) [production]
07:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1085 after schema change', diff saved to https://phabricator.wikimedia.org/P10781 and previous config saved to /var/cache/conftool/dbconfig/20200326-074033-marostegui.json [production]
07:31 <marostegui> Deploy schema change on db1085, lag will appear on s6 on labs [production]
07:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1085 for schema change', diff saved to https://phabricator.wikimedia.org/P10780 and previous config saved to /var/cache/conftool/dbconfig/20200326-073048-marostegui.json [production]
07:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1093 after schema change', diff saved to https://phabricator.wikimedia.org/P10779 and previous config saved to /var/cache/conftool/dbconfig/20200326-070746-marostegui.json [production]
06:59 <marostegui> Deploy schema change on db1093 [production]
06:59 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1093 for schema change', diff saved to https://phabricator.wikimedia.org/P10778 and previous config saved to /var/cache/conftool/dbconfig/20200326-065929-marostegui.json [production]
06:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1088 after schema change', diff saved to https://phabricator.wikimedia.org/P10777 and previous config saved to /var/cache/conftool/dbconfig/20200326-065814-marostegui.json [production]