|
2020-09-14
§
|
| 21:32 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 21:30 |
<cdanis> |
T257527 ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕠🍺 sudo cumin 'R:Class ~ "(?i)profile::logstash::collector7"' 'enable-puppet "cdanis rolling out Ifa3c68e4"' |
[production] |
| 21:24 |
<cdanis> |
T257527 ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕠🍺 sudo cumin 'R:Class ~ "(?i)profile::logstash::collector7"' 'disable-puppet "cdanis rolling out Ifa3c68e4"' |
[production] |
| 21:05 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 21:03 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:38 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 20:36 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 20:26 |
<cdanis@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: a588eb0c6 T262087 modify wgEventStreams to reference NEL schema (duration: 00m 56s) |
[production] |
| 19:00 |
<Urbanecm> |
Morning B&C done |
[production] |
| 18:57 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: a5d56edc7460ac43492f9c04cff86c1b03e56fa4: e2f47980c371b52b1b66957f2bff2266745ab00a: Enable Special:Investigate on eswiki (T262436) (duration: 00m 56s) |
[production] |
| 18:49 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:47 |
<volans@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:38 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: 7d1939323cc3ea5dacf67a43d4d359c114203a66: Remove investigate from $wgAvailableRights (T260175) (duration: 00m 56s) |
[production] |
| 18:32 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: d2fa6533a594c8544342954eae19a4a0f7baeff0: Remove the investigate right from testwiki and frwiki (T260175) (duration: 00m 56s) |
[production] |
| 18:30 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.8/extensions/EventStreamConfig/includes/: a4c86089371319ae5a3bb6053c4a9b3e83130286: Default to using API json formatversion=2 (T251609) (duration: 00m 57s) |
[production] |
| 18:21 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 27ba5a1da1fb00e721cfa82dd4cd1fbac2541053: add new parse* servers to $wgLinterSubmitterWhitelist (T247441) (duration: 00m 56s) |
[production] |
| 18:15 |
<urbanecm@deploy1001> |
Synchronized wmf-config/flaggedrevs.php: 720e6cbfe1800fe32dc65c209240ba08706dbb17: flaggedrevs: Move setting of wgFlaggedRevsAutopromote and wgFlaggedRevsAutoconfirm out of wgExtensionFunctions (T237191) (duration: 00m 56s) |
[production] |
| 18:09 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 699f5e8c2a50f35e98850ea32f7847d183600351: Add logo Wordmark and Tagline for hywiki (T259985) (duration: 00m 55s) |
[production] |
| 18:08 |
<urbanecm@deploy1001> |
Synchronized static/images/mobile/copyright/: 699f5e8c2a50f35e98850ea32f7847d183600351: Add logo Wordmark and Tagline for hywiki (T259985) (duration: 00m 56s) |
[production] |
| 17:51 |
<mutante> |
all new parse* parsoid hardware pooled now and set to active in netbox, deploy in 10 min will add to $wgLinterSubmitterWhitelist (T247441) |
[production] |
| 17:41 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: dc=codfw,cluster=parsoid,name=parse20[1-2][0-9].codfw.wmnet |
[production] |
| 17:22 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: dc=codfw,cluster=parsoid,name=parse200[0-9].codfw.wmnet |
[production] |
| 17:16 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: dc=codfw,cluster=parsoid,name=parse2002.codfw.wmnet |
[production] |
| 16:51 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: dc=codfw,cluster=parsoid,name=parse2001.codfw.wmnet |
[production] |
| 16:48 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: dc=codfw,cluster=parsoid,name=parse2001.codfw.wmnet |
[production] |
| 16:36 |
<mutante> |
pooled the first of the new parsoid servers - parse2001 (T247441) |
[production] |
| 16:33 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: dc=codfw,cluster=parsoid,name=parse2001.codfw.wmnet |
[production] |
| 16:07 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: dc=codfw,cluster=parsoid,name=parse20[1-2][0-9].codfw.wmnet |
[production] |
| 16:04 |
<elukey> |
completed the rollout of restrictive kafka ferm rules on the Kafka jumbo cluster |
[production] |
| 16:03 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: dc=codfw,cluster=parsoid,name=parse200[0-9].codfw.wmnet |
[production] |
| 16:01 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: dc=codfw,cluster=parsoid,name=parse20[0-2][0-9].codfw.wmnet |
[production] |
| 15:59 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: dc=codfw,cluster=parsoid,name=parse2001.codfw.wmnet |
[production] |
| 15:58 |
<dzahn@cumin1001> |
conftool action : set/weight=10; selector: dc=codfw,cluster=parsoid,name=parse20[1-2][0-9].codfw.wmnet |
[production] |
| 15:54 |
<moritzm> |
restarting apache on webperf* to pick up GNU TLS security update |
[production] |
| 15:45 |
<moritzm> |
restarting apache/FPM on mw2271/m2272 (codfw canaries) to pick up GNU TLS update |
[production] |
| 15:35 |
<moritzm> |
installing gnutls28 security updates on stretch |
[production] |
| 15:23 |
<elukey> |
enable stricter ferm rules on kafka-jumbo1007 and kafka-jumbo1005 |
[production] |
| 15:17 |
<cicalese@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:626229|Allow public access to API Portal main page for private launch]] (duration: 00m 57s) |
[production] |
| 15:17 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:11 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:11 |
<cmjohnson1> |
completed pdu swap in eqiad racks d5/d6 |
[production] |
| 14:55 |
<elukey> |
ferm rules added to kafka-jumbo1009, 1006 and 1008 up to now |
[production] |
| 14:24 |
<milimetric@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 14:24 |
<milimetric@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 14:16 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 14:14 |
<milimetric@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 14:14 |
<milimetric@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 14:11 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 14:09 |
<milimetric@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 14:09 |
<milimetric@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |