951-1000 of 10000 results (26ms)
2020-06-03 §
05:35 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
05:32 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
05:14 <XioNoX> turn cr1-codfw:fpc0 online - T254110 [production]
05:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1138 T253808', diff saved to https://phabricator.wikimedia.org/P11369 and previous config saved to /var/cache/conftool/dbconfig/20200603-050911-marostegui.json [production]
01:00 <reedy@deploy1001> Synchronized wmf-config/CommonSettings-labs.php: wgLocalVirtualHosts (duration: 01m 06s) [production]
00:59 <krinkle@deploy1001> Synchronized wmf-config/mc.php: Ic27b605aec0118c5 (duration: 01m 11s) [production]
2020-06-02 §
23:58 <ejegg> updated fundraising CiviCRM from 657c4b9455 to eb156dffa4 [production]
23:55 <ejegg> updated payments-wiki from 1942a537ef to e46114d8b1 [production]
22:48 <cstone> civicrm revision changed from d1cd99166f to 657c4b9455 [production]
21:48 <reedy@deploy1001> Synchronized wmf-config/interwiki-labs.php: laaaaabs (duration: 01m 05s) [production]
21:23 <reedy@deploy1001> Synchronized multiversion/MWMultiVersion.php: beta apiportalwiki T254185 (duration: 01m 06s) [production]
21:21 <reedy@deploy1001> Synchronized wmf-config/config/apiportalwiki.yaml: beta apiportalwiki T254185 (duration: 01m 05s) [production]
21:20 <reedy@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: beta apiportalwiki T254185 (duration: 01m 05s) [production]
21:19 <reedy@deploy1001> Synchronized wikiversions-labs.json: beta apiportalwiki T254185 (duration: 01m 05s) [production]
21:17 <reedy@deploy1001> Synchronized dblists/all-labs.dblist: beta apiportalwiki T254185 (duration: 01m 06s) [production]
21:12 <cdanis> repooled wtp1032 T254258 [production]
20:24 <reedy@deploy1001> Synchronized composer.lock: Update (duration: 01m 06s) [production]
20:02 <jforrester@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.35 [production]
19:59 <jforrester@deploy1001> Finished scap: testwikis wikis to 1.35.0-wmf.35 (duration: 93m 52s) [production]
18:25 <jforrester@deploy1001> Started scap: testwikis wikis to 1.35.0-wmf.35 [production]
18:22 <jforrester@deploy1001> Pruned MediaWiki: 1.35.0-wmf.31 (duration: 19m 59s) [production]
18:05 <cdanis> fixing g+w permissions of deploy1001 /srv/mediawiki-staging/php-*/.git/objects/* [production]
17:20 <James_F> 1.35.0-wmf.35 was branched at 8d7015037d44c4fe21eee8e8a040720b838bc169 for T253023 [production]
16:55 <cstone> SmashPig revision changed from 44690f761c to b9de3c7aac [production]
15:57 <ejegg> updated payments-wiki from d11efeb1cf to 1942a537ef [production]
15:50 <cdanis> thumbor1003 and thumbor1004 blipped, no obvious explanation, logs gathered at P11365 P11366 P11367 [production]
15:49 <XioNoX> push frack fw rules - T254260 [production]
15:48 <mutante> contint1001 - rm -rf /mnt/docker (T224591) [production]
15:45 <mutante> contint1001 - restarting docker afer changed data-root path (T224591) [production]
15:37 <cdanis@cumin1001> conftool action : set/pooled=no; selector: name=wtp1032.* [production]
15:35 <cdanis> power cycling wtp1032 which is bootlooping? https://phabricator.wikimedia.org/P11364 [production]
15:31 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
15:24 <rzl@cumin1001> conftool action : set/pooled=yes; selector: name=thumbor100[34].* [production]
15:23 <XioNoX> repool codfw - T254216 [production]
15:19 <XioNoX> rollback ospf changes - T254216 [production]
15:09 <hnowlan@deploy1001> Finished deploy [cpjobqueue/deploy@8a53ff1]: (no justification provided) (duration: 02m 33s) [production]
15:07 <XioNoX> reboot cr1-codfw:fpc5 - T254216 [production]
15:06 <hnowlan@deploy1001> Started deploy [cpjobqueue/deploy@8a53ff1]: (no justification provided) [production]
15:05 <hnowlan> shifting all high traffic cpjobqueue rules to k8s [production]
14:57 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:56 <XioNoX> depref ulsfo-codfw link - T254216 [production]
14:51 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:50 <jynus@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
14:49 <jynus@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:49 <XioNoX> prefer eqsin-ulsfo tunnel - T254216 [production]
14:47 <cdanis@cumin1001> conftool action : set/pooled=no; selector: name=thumbor100[34].* [production]
14:38 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
14:31 <XioNoX> depool codfw - T254216 [production]
14:09 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
13:45 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]