1901-1950 of 10000 results (30ms)
2020-10-29 ยง
14:36 <ppchelko@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:35 <ppchelko@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:34 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
14:33 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
14:29 <jmm@cumin1001> START - Cookbook sre.hosts.decommission [production]
14:26 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
14:25 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:25 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:24 <elukey> restart zookeeper on an-conf1001 for openjdk upgrades [production]
14:20 <jmm@cumin1001> START - Cookbook sre.hosts.decommission [production]
14:08 <godog> bump FS for prometheus codfw global instance [production]
13:54 <elukey> roll out profile::java on all zookeeper instances [production]
13:53 <moritzm> installing Java 11 security updates [production]
13:52 <bblack> authdns1001 - restart gdnsd - T266746 [production]
13:46 <bblack> authdns2001 - restart gdnsd - T266746 [production]
13:38 <bblack> staggered restart of gdnsd on dns[12345]001 (1/2 recursors in each DC) - T266746 [production]
13:29 <bblack> staggered restart of gdnsd on dns[12345]002 (1/2 recursors in each DC) - T266746 [production]
13:25 <Urbanecm> Correction: Obviously 1002 (T246539) [production]
13:23 <Urbanecm> Start of `mwscript extensions/AbuseFilter/maintenance/updateVarDumps.php --wiki=$wiki --print-orphaned-records-to=/tmp/urbanecm/$wiki-orphaned.log --progress-markers > $wiki.log` in a tmux session updateVarDumps at mwmaint2001 (wiki=idwiki; T246539) [production]
13:21 <moritzm> installing bluez security updates on stretch [production]
12:56 <marostegui> Make orchestrator discover pc2 T266485 [production]
12:55 <marostegui> Deploy orchestrator grants on pc2 T266485 [production]
12:44 <marostegui> Deploy grants for cluster alias on pc1 T266485 [production]
12:35 <moritzm> upgrade idp-test* hosts to latest Java securiy updates [production]
12:35 <moritzm> restart idp-test [production]
12:34 <ariel@deploy1001> Finished deploy [dumps/dumps@4ed2cb9]: revinfo for page content jobs, tableinfo for list of known tables (duration: 00m 05s) [production]
12:33 <ariel@deploy1001> Started deploy [dumps/dumps@4ed2cb9]: revinfo for page content jobs, tableinfo for list of known tables [production]
12:01 <klausman@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
11:18 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.cf (exit_code=0) [production]
11:18 <ayounsi@cumin1001> START - Cookbook sre.network.cf [production]
11:14 <Urbanecm> EU B&C window done [production]
11:12 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 28152b7387082b79d71cfbf28be740ffe629ee50: Add another SDC property to search for matching media statements (T264925) (duration: 00m 58s) [production]
11:11 <klausman@cumin1001> START - Cookbook sre.ganeti.makevm [production]
11:07 <klausman@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
11:07 <klausman@cumin1001> START - Cookbook sre.ganeti.makevm [production]
11:06 <klausman@cumin1001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
11:06 <klausman@cumin1001> START - Cookbook sre.ganeti.makevm [production]
10:15 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:15 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:15 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
10:12 <elukey> restart tilerator on maps100[1,4] - redis errors in the logs [production]
10:11 <elukey> restart tilerator on maps1002 - redis errors in the logs [production]
10:03 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
10:03 <elukey> drop 10.64.21.6/24 and 2620:0:861:105:10:64:21:6/64 from netbox (an-tool-ui1001 related records) [production]
09:59 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: Fix cxserver's configuration to use envoy (duration: 00m 59s) [production]
09:52 <elukey> add gdnsd.service to all gdnsd hosts (with LimitNOFILE=infinity as override) - no daemon restart done - T266746 [production]
09:41 <marostegui> Deploy schema change on s8 wikidata codfw master (db2079) T264109 [production]
09:33 <elukey> clean up 10.64.21.7/24 and 2620:0:861:105:10:64:21:7/64 from netbox (an-test-ui1001 already have ips previously allocated by makevm) [production]
09:32 <elukey@cumin1001> END (ERROR) - Cookbook sre.ganeti.makevm (exit_code=97) [production]
09:23 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]