|
2021-12-17
ยง
|
| 17:07 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 16:59 |
<milimetric@deploy1002> |
Started deploy [analytics/refinery@0778d1e]: Proper fix for mediawiki_skin_diff |
[production] |
| 16:59 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 16:56 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 16:53 |
<bblack> |
bast6001: shutdown->start |
[production] |
| 16:44 |
<bblack> |
ganeti6003 - rebooting |
[production] |
| 16:39 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 16:33 |
<godog> |
remove /var/log/swift/server.log.1 from thanos-be* - T297959 |
[production] |
| 16:29 |
<milimetric@deploy1002> |
Finished deploy [analytics/refinery@5c3bce1] (thin): Fix refine sanitize allowlist, remove mediawiki_skin_diff schema for now [THIN] (duration: 00m 07s) |
[production] |
| 16:29 |
<milimetric@deploy1002> |
Started deploy [analytics/refinery@5c3bce1] (thin): Fix refine sanitize allowlist, remove mediawiki_skin_diff schema for now [THIN] |
[production] |
| 16:28 |
<bblack> |
reboot bast6001 (downtimed) |
[production] |
| 16:26 |
<milimetric@deploy1002> |
Finished deploy [analytics/refinery@5c3bce1]: Fix refine sanitize allowlist, remove mediawiki_skin_diff schema for now (duration: 69m 48s) |
[production] |
| 16:02 |
<godog> |
root@thanos-be2004:/srv/log/swift# rm server.log.1 - T297959 |
[production] |
| 15:35 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 15:35 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . |
[production] |
| 15:35 |
<elukey@deploy1002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . |
[production] |
| 15:34 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . |
[production] |
| 15:34 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . |
[production] |
| 15:33 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 15:16 |
<milimetric@deploy1002> |
Started deploy [analytics/refinery@5c3bce1]: Fix refine sanitize allowlist, remove mediawiki_skin_diff schema for now |
[production] |
| 14:52 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2001.codfw.wmnet with OS buster |
[production] |
| 14:35 |
<milimetric@deploy1002> |
Finished deploy [analytics/refinery@e9f04c3] (hadoop-test): Fix sanitize allowlist problem [TEST] (duration: 69m 41s) |
[production] |
| 14:22 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reimage for host kafka-main2001.codfw.wmnet with OS buster |
[production] |
| 13:25 |
<milimetric@deploy1002> |
Started deploy [analytics/refinery@e9f04c3] (hadoop-test): Fix sanitize allowlist problem [TEST] |
[production] |
| 13:20 |
<milimetric@deploy1002> |
Finished deploy [analytics/refinery@e9f04c3] (thin): Fix sanitize allowlist problem [THIN] (duration: 00m 07s) |
[production] |
| 13:20 |
<milimetric@deploy1002> |
Started deploy [analytics/refinery@e9f04c3] (thin): Fix sanitize allowlist problem [THIN] |
[production] |
| 13:20 |
<milimetric@deploy1002> |
Finished deploy [analytics/refinery@e9f04c3]: Fix sanitize allowlist problem (duration: 26m 19s) |
[production] |
| 13:06 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host bast6001.wikimedia.org |
[production] |
| 13:04 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=druid1008.eqiad.wmnet,service=druid-public-broker |
[production] |
| 13:02 |
<btullis> |
upgraded druid packages on druid1008 |
[production] |
| 13:01 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=druid1008.eqiad.wmnet,service=druid-public-broker |
[production] |
| 13:00 |
<btullis> |
upgraded druid packages on an-druid1005 |
[production] |
| 12:58 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=druid1007.eqiad.wmnet,service=druid-public-broker |
[production] |
| 12:55 |
<btullis> |
upgrading druid packages on druid1007 |
[production] |
| 12:54 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=druid1007.eqiad.wmnet,service=druid-public-broker |
[production] |
| 12:53 |
<milimetric@deploy1002> |
Started deploy [analytics/refinery@e9f04c3]: Fix sanitize allowlist problem |
[production] |
| 12:52 |
<btullis> |
upgrading druid packages on an-druid1004 |
[production] |
| 12:51 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=druid1006.eqiad.wmnet,service=druid-public-broker |
[production] |
| 12:50 |
<btullis> |
upgrading druid packages on druid1006 |
[production] |
| 12:48 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=druid1006.eqiad.wmnet,service=druid-public-broker |
[production] |
| 12:47 |
<btullis> |
Upgrading druid packages on an-druid1003 |
[production] |
| 12:44 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=druid1005.eqiad.wmnet,service=druid-public-broker |
[production] |
| 12:43 |
<btullis> |
Upgraded druid packages on druid1005. |
[production] |
| 12:43 |
<btullis> |
Upgraded druid packages on an-druid1002. |
[production] |
| 12:43 |
<mmandere@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host bast6001.wikimedia.org |
[production] |
| 12:41 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=druid1005.eqiad.wmnet,service=druid-public-broker |
[production] |
| 12:40 |
<mmandere@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host bast6001.wikimedia.org |
[production] |
| 12:37 |
<btullis> |
upgrading druid packages on an-druid1002 |
[production] |
| 12:32 |
<mmandere@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host bast6001.wikimedia.org |
[production] |
| 12:31 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=druid1004.eqiad.wmnet,service=druid-public-broker |
[production] |