2701-2750 of 10000 results (28ms)
2020-06-02 ยง
21:19 <reedy@deploy1001> Synchronized wikiversions-labs.json: beta apiportalwiki T254185 (duration: 01m 05s) [production]
21:17 <reedy@deploy1001> Synchronized dblists/all-labs.dblist: beta apiportalwiki T254185 (duration: 01m 06s) [production]
21:12 <cdanis> repooled wtp1032 T254258 [production]
20:24 <reedy@deploy1001> Synchronized composer.lock: Update (duration: 01m 06s) [production]
20:02 <jforrester@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.35 [production]
19:59 <jforrester@deploy1001> Finished scap: testwikis wikis to 1.35.0-wmf.35 (duration: 93m 52s) [production]
18:25 <jforrester@deploy1001> Started scap: testwikis wikis to 1.35.0-wmf.35 [production]
18:22 <jforrester@deploy1001> Pruned MediaWiki: 1.35.0-wmf.31 (duration: 19m 59s) [production]
18:05 <cdanis> fixing g+w permissions of deploy1001 /srv/mediawiki-staging/php-*/.git/objects/* [production]
17:20 <James_F> 1.35.0-wmf.35 was branched at 8d7015037d44c4fe21eee8e8a040720b838bc169 for T253023 [production]
16:55 <cstone> SmashPig revision changed from 44690f761c to b9de3c7aac [production]
15:57 <ejegg> updated payments-wiki from d11efeb1cf to 1942a537ef [production]
15:50 <cdanis> thumbor1003 and thumbor1004 blipped, no obvious explanation, logs gathered at P11365 P11366 P11367 [production]
15:49 <XioNoX> push frack fw rules - T254260 [production]
15:48 <mutante> contint1001 - rm -rf /mnt/docker (T224591) [production]
15:45 <mutante> contint1001 - restarting docker afer changed data-root path (T224591) [production]
15:37 <cdanis@cumin1001> conftool action : set/pooled=no; selector: name=wtp1032.* [production]
15:35 <cdanis> power cycling wtp1032 which is bootlooping? https://phabricator.wikimedia.org/P11364 [production]
15:31 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
15:24 <rzl@cumin1001> conftool action : set/pooled=yes; selector: name=thumbor100[34].* [production]
15:23 <XioNoX> repool codfw - T254216 [production]
15:19 <XioNoX> rollback ospf changes - T254216 [production]
15:09 <hnowlan@deploy1001> Finished deploy [cpjobqueue/deploy@8a53ff1]: (no justification provided) (duration: 02m 33s) [production]
15:07 <XioNoX> reboot cr1-codfw:fpc5 - T254216 [production]
15:06 <hnowlan@deploy1001> Started deploy [cpjobqueue/deploy@8a53ff1]: (no justification provided) [production]
15:05 <hnowlan> shifting all high traffic cpjobqueue rules to k8s [production]
14:57 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:56 <XioNoX> depref ulsfo-codfw link - T254216 [production]
14:51 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:50 <jynus@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
14:49 <jynus@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:49 <XioNoX> prefer eqsin-ulsfo tunnel - T254216 [production]
14:47 <cdanis@cumin1001> conftool action : set/pooled=no; selector: name=thumbor100[34].* [production]
14:38 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
14:31 <XioNoX> depool codfw - T254216 [production]
14:09 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
13:45 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
13:42 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
13:42 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
13:38 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
13:37 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
13:28 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
13:18 <dzahn@cumin1001> END (ERROR) - Cookbook sre.hosts.decommission (exit_code=97) [production]
13:18 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
13:18 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
13:18 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
13:05 <cdanis@deploy1001> Synchronized php-1.35.0-wmf.31/includes/specials/pagers/ContribsPager.php: revert contribs limit to 5000 T234450 (duration: 00m 57s) [production]
13:04 <cdanis@deploy1001> Synchronized php-1.35.0-wmf.32/includes/specials/pagers/ContribsPager.php: revert contribs limit to 5000 T234450 (duration: 00m 57s) [production]
13:03 <cdanis@deploy1001> Synchronized php-1.35.0-wmf.34/includes/specials/pagers/ContribsPager.php: revert contribs limit to 5000 T234450 (duration: 00m 58s) [production]
12:59 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]