|
2021-07-12
ยง
|
| 18:45 |
<majavah> |
upgrade deployment-cache-text06 to use varnish 6 (with profile::cache::varnish::frontend::packages_component), and run apt upgrade, T286506 |
[releng] |
| 18:43 |
<majavah> |
deployment-cache-text06 varnish not starting, T286506, causing an outage on text traffic on deployment-prep |
[releng] |
| 18:41 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@200b502]: Finalize event_default gobblin job - T271232 (duration: 03m 39s) |
[production] |
| 18:37 |
<otto@deploy1002> |
Started deploy [analytics/refinery@200b502]: Finalize event_default gobblin job - T271232 |
[production] |
| 18:37 |
<joal> |
Move /wmf/data/raw/event to /wmf/data/raw/event_camus and /wmf/data/raw/event_gobblin to /wmf/data/raw/event |
[analytics] |
| 18:36 |
<joal> |
Delete /year=2021/month=07/day=12/hour=14 of gobblin imported events |
[analytics] |
| 18:23 |
<majavah> |
hard reboot deployment-cache-text06 once I got in using a root ssh key |
[releng] |
| 18:17 |
<ottomata> |
stopped puppet and refines and imports for event data on an-launcher1002 in prep for gobblin finalization for event_default job |
[analytics] |
| 18:12 |
<legoktm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Enable Score using Shellbox on testwiki (T257066) (duration: 00m 58s) |
[production] |
| 16:56 |
<bstorm> |
deleted job 4720371 due to LDAP failure |
[tools] |
| 16:55 |
<bstorm> |
deleted 4946664 which was stuck from LDAP failure |
[tools.flickr] |
| 16:54 |
<bstorm> |
deleted job 4972670 which was stuck from LDAP failure |
[tools.urbanecmbot] |
| 16:51 |
<bstorm> |
cleared the E state from two job queues |
[tools] |
| 16:15 |
<majavah> |
hard reboot deployment-cache-text06, refusing to let me log in and console full of errors |
[releng] |
| 16:15 |
<ppchelko@deploy1002> |
Finished deploy [restbase/deploy@b05ade3]: Add newly created wikis T284929 T284457 T284392 (duration: 21m 24s) |
[production] |
| 16:01 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 18 hosts with reason: Deploying schema change to s4 T277116 - extending downtime |
[production] |
| 16:01 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on 18 hosts with reason: Deploying schema change to s4 T277116 - extending downtime |
[production] |
| 15:54 |
<ppchelko@deploy1002> |
Started deploy [restbase/deploy@b05ade3]: Add newly created wikis T284929 T284457 T284392 |
[production] |
| 15:45 |
<bstorm> |
silenced deployment prep alerts for another 60 days |
[metricsinfra] |
| 15:31 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Deploying schema change to s4 T277116 |
[production] |
| 15:31 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 18 hosts with reason: Deploying schema change to s4 T277116 |
[production] |
| 15:28 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Deploying schema change to s7 T277116 |
[production] |
| 15:28 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 15 hosts with reason: Deploying schema change to s7 T277116 |
[production] |
| 15:24 |
<elukey> |
expand ML k8s iBGP neighbors to include the master nodes (ref: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/704104) |
[production] |
| 15:16 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Deploying schema change to s2 T277116 |
[production] |
| 15:15 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 15 hosts with reason: Deploying schema change to s2 T277116 |
[production] |
| 15:10 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ldap-replica1002.wikimedia.org |
[production] |
| 15:08 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 13 hosts with reason: Deploying schema change to s5 T277116 |
[production] |
| 15:08 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 13 hosts with reason: Deploying schema change to s5 T277116 |
[production] |
| 15:00 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts ldap-replica1002.wikimedia.org |
[production] |
| 14:58 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 14 hosts with reason: Deploying schema change T277116 |
[production] |
| 14:58 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 14 hosts with reason: Deploying schema change T277116 |
[production] |
| 14:58 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ldap-replica1001.wikimedia.org |
[production] |
| 14:48 |
<Amir1> |
ran $ ./jjb-update 'wikidata-query-gui-build' (T286479) |
[releng] |
| 14:44 |
<majavah> |
fix merge conflict on deployment-puppetmaster04 |
[releng] |
| 14:44 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts ldap-replica1001.wikimedia.org |
[production] |
| 14:42 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ldap-replica2004.wikimedia.org |
[production] |
| 14:26 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts ldap-replica2004.wikimedia.org |
[production] |
| 14:25 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ldap-replica2003.wikimedia.org |
[production] |
| 14:24 |
<Krinkle> |
#cvn-commons rubin16 local_op, sysop confirmed https://ru.wikipedia.org/w/index.php?oldid=114382938 |
[cvn] |
| 14:15 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts ldap-replica2003.wikimedia.org |
[production] |
| 14:01 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=maps2010.codfw.wmnet |
[production] |
| 13:59 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@dd65f38]: event_default gobblin job - fix typo - T271232 (duration: 03m 30s) |
[production] |
| 13:56 |
<otto@deploy1002> |
Started deploy [analytics/refinery@dd65f38]: event_default gobblin job - fix typo - T271232 |
[production] |
| 13:52 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@0149c81]: Set event_default gobblin job max mappers=128 - T271232 (duration: 03m 16s) |
[production] |
| 13:49 |
<otto@deploy1002> |
Started deploy [analytics/refinery@0149c81]: Set event_default gobblin job max mappers=128 - T271232 |
[production] |
| 13:36 |
<otto@deploy1002> |
Finished deploy [analytics/refinery@1cb9e12]: Add event_default gobblin job - T271232 (duration: 03m 37s) |
[production] |
| 13:32 |
<otto@deploy1002> |
Started deploy [analytics/refinery@1cb9e12]: Add event_default gobblin job - T271232 |
[production] |
| 13:19 |
<James_F> |
Zuul: Add Voidwalker to the CI allow list |
[releng] |
| 13:19 |
<James_F> |
Zuul: Add R4356th to the CI allow list |
[releng] |