|
2020-10-29
ยง
|
| 14:20 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:08 |
<godog> |
bump FS for prometheus codfw global instance |
[production] |
| 13:54 |
<elukey> |
roll out profile::java on all zookeeper instances |
[production] |
| 13:53 |
<moritzm> |
installing Java 11 security updates |
[production] |
| 13:52 |
<bblack> |
authdns1001 - restart gdnsd - T266746 |
[production] |
| 13:46 |
<bblack> |
authdns2001 - restart gdnsd - T266746 |
[production] |
| 13:38 |
<bblack> |
staggered restart of gdnsd on dns[12345]001 (1/2 recursors in each DC) - T266746 |
[production] |
| 13:29 |
<bblack> |
staggered restart of gdnsd on dns[12345]002 (1/2 recursors in each DC) - T266746 |
[production] |
| 13:25 |
<Urbanecm> |
Correction: Obviously 1002 (T246539) |
[production] |
| 13:23 |
<Urbanecm> |
Start of `mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log` in a tmux session updateVarDumps at mwmaint2001 (wiki=idwiki; T246539) |
[production] |
| 13:21 |
<moritzm> |
installing bluez security updates on stretch |
[production] |
| 12:56 |
<marostegui> |
Make orchestrator discover pc2 T266485 |
[production] |
| 12:55 |
<marostegui> |
Deploy orchestrator grants on pc2 T266485 |
[production] |
| 12:44 |
<marostegui> |
Deploy grants for cluster alias on pc1 T266485 |
[production] |
| 12:35 |
<moritzm> |
upgrade idp-test* hosts to latest Java securiy updates |
[production] |
| 12:35 |
<moritzm> |
restart idp-test |
[production] |
| 12:34 |
<ariel@deploy1001> |
Finished deploy [dumps/dumps@4ed2cb9]: revinfo for page content jobs, tableinfo for list of known tables (duration: 00m 05s) |
[production] |
| 12:33 |
<ariel@deploy1001> |
Started deploy [dumps/dumps@4ed2cb9]: revinfo for page content jobs, tableinfo for list of known tables |
[production] |
| 12:01 |
<klausman@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 11:18 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.cf (exit_code=0) |
[production] |
| 11:18 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.cf |
[production] |
| 11:14 |
<Urbanecm> |
EU B&C window done |
[production] |
| 11:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 28152b7387082b79d71cfbf28be740ffe629ee50: Add another SDC property to search for matching media statements (T264925) (duration: 00m 58s) |
[production] |
| 11:11 |
<klausman@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 11:07 |
<klausman@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 11:07 |
<klausman@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 11:06 |
<klausman@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 11:06 |
<klausman@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 10:15 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:15 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 10:15 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 10:12 |
<elukey> |
restart tilerator on maps100[1,4] - redis errors in the logs |
[production] |
| 10:11 |
<elukey> |
restart tilerator on maps1002 - redis errors in the logs |
[production] |
| 10:03 |
<elukey@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 10:03 |
<elukey> |
drop 10.64.21.6/24 and 2620:0:861:105:10:64:21:6/64 from netbox (an-tool-ui1001 related records) |
[production] |
| 09:59 |
<oblivian@deploy1001> |
Synchronized wmf-config/ProductionServices.php: Fix cxserver's configuration to use envoy (duration: 00m 59s) |
[production] |
| 09:52 |
<elukey> |
add gdnsd.service to all gdnsd hosts (with LimitNOFILE=infinity as override) - no daemon restart done - T266746 |
[production] |
| 09:41 |
<marostegui> |
Deploy schema change on s8 wikidata codfw master (db2079) T264109 |
[production] |
| 09:33 |
<elukey> |
clean up 10.64.21.7/24 and 2620:0:861:105:10:64:21:7/64 from netbox (an-test-ui1001 already have ips previously allocated by makevm) |
[production] |
| 09:32 |
<elukey@cumin1001> |
END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) |
[production] |
| 09:23 |
<elukey@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 08:54 |
<vgutierrez> |
turn off ECDHE-ECDSA-AES128-SHA support on the main caching cluster - T258405 |
[production] |
| 08:54 |
<moritzm> |
fixing up stray jenkins auto restart timers on secondary releases server |
[production] |
| 08:53 |
<vgutierrez> |
A:cp (except cp3052, running varnish 5) upgrade libvmod-netmapper to 1.9-1 T266567 T264398 |
[production] |
| 08:48 |
<moritzm> |
fixing up stray mcelog auto restart timers on kubestage* |
[production] |
| 08:38 |
<moritzm> |
fixing up stray cas auto restart timers on secondary IDP servers |
[production] |
| 08:19 |
<moritzm> |
fixing up stray pmacctd auto restart timers on netflow* |
[production] |
| 08:19 |
<moritzm> |
fixing up stray pcacctd auto restart timers on netflow* |
[production] |
| 08:02 |
<marostegui> |
Disconnect replication codfw -> eqiad on s1 T266663 |
[production] |
| 07:56 |
<vgutierrez> |
set LimitNOFILE=500000 for gdnsd on authdns1001 |
[production] |