451-500 of 10000 results (61ms)
2020-03-02 §
06:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1111 to s8 with minimal weight to check grants and any other issues T246447', diff saved to https://phabricator.wikimedia.org/P10564 and previous config saved to /var/cache/conftool/dbconfig/20200302-062435-marostegui.json [production]
06:04 <marostegui> Re-add db1111 to s8 in tendril and zarcillo - T246447 [production]
2020-03-01 §
17:54 <marostegui> Start replication on db1111 new host on s8 - T246447 [production]
17:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Reduce main traffic weight for db1087 as dumps are running ', diff saved to https://phabricator.wikimedia.org/P10563 and previous config saved to /var/cache/conftool/dbconfig/20200301-174536-marostegui.json [production]
16:08 <reedy@deploy1001> scap failed: average error rate on 5/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
06:02 <ariel@deploy1001> Finished deploy [dumps/dumps@8376c62]: refactor page content jobs, prefetch, and output file listings: see T246465 (duration: 00m 04s) [production]
06:02 <ariel@deploy1001> Started deploy [dumps/dumps@8376c62]: refactor page content jobs, prefetch, and output file listings: see T246465 [production]
2020-02-29 §
12:37 <reedy@deploy1001> Synchronized wmf-config/config/viwiki.yaml: T246511 (duration: 00m 56s) [production]
12:35 <reedy@deploy1001> Synchronized wikiversions-labs.json: T246511 (duration: 00m 56s) [production]
12:34 <reedy@deploy1001> Synchronized dblists/all-labs.dblist: T246511 (duration: 00m 57s) [production]
2020-02-28 §
21:31 <mutante> using planet1001 to manually hack APT sources to test new apt1001.wikimedia.org [production]
20:29 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
20:26 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
19:01 <milimetric@deploy1001> Finished deploy [analytics/refinery@0fc392f] (thin): Hotfix: going back to a safe version of geo udf (duration: 00m 07s) [production]
19:01 <milimetric@deploy1001> Started deploy [analytics/refinery@0fc392f] (thin): Hotfix: going back to a safe version of geo udf [production]
19:01 <milimetric@deploy1001> Finished deploy [analytics/refinery@0fc392f]: Hotfix: going back to a safe version of geo udf (duration: 13m 06s) [production]
18:47 <milimetric@deploy1001> Started deploy [analytics/refinery@0fc392f]: Hotfix: going back to a safe version of geo udf [production]
16:25 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
16:22 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
16:05 <oblivian@puppetmaster1001> conftool action : set/pooled=yes:weight=1; selector: cluster=kibana,service=kibana-next [production]
15:54 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:39 <moritzm> installing libperl4-corelibs-perl updates from Stretch point release [production]
15:36 <elukey@deploy1001> Finished deploy [analytics/refinery@28fa2fc]: fix for refinery-drop-older-than - part 2 (duration: 13m 40s) [production]
15:24 <marostegui> Stop replication on db1077 from db1111 (its master) - T246447 [production]
15:22 <elukey@deploy1001> Started deploy [analytics/refinery@28fa2fc]: fix for refinery-drop-older-than - part 2 [production]
14:17 <gehel> rolling restart of elasticsearch/eqiad for JVM upgrade completed [production]
14:16 <gehel@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-restart (exit_code=0) [production]
14:15 <elukey@deploy1001> Finished deploy [analytics/refinery@2db36f4]: Fix refinery-drop-older-than script (duration: 14m 01s) [production]
14:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase weight from 100 to 300', diff saved to https://phabricator.wikimedia.org/P10558 and previous config saved to /var/cache/conftool/dbconfig/20200228-141035-marostegui.json [production]
14:01 <elukey@deploy1001> Started deploy [analytics/refinery@2db36f4]: Fix refinery-drop-older-than script [production]
13:58 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-restart [production]
13:32 <marostegui> Reset idrac from db1114 [production]
12:11 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
12:06 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
11:57 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
11:04 <gehel@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-restart (exit_code=97) [production]
10:53 <jynus> labsdb1009-12 prometheus metrics restored after 90 minutes of unscheduled unavailability [production]
10:27 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-restart [production]
10:15 <gehel@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-restart (exit_code=99) [production]
10:13 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-restart [production]
10:01 <gehel@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-restart (exit_code=99) [production]
09:59 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-restart [production]
09:59 <gehel> starting rolling restart of elasticsearch/eqiad for JVM upgrade [production]
09:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1101:3318 from vslow,dump', diff saved to https://phabricator.wikimedia.org/P10555 and previous config saved to /var/cache/conftool/dbconfig/20200228-093653-marostegui.json [production]
09:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1087 into vslow,dump as it was there originally', diff saved to https://phabricator.wikimedia.org/P10554 and previous config saved to /var/cache/conftool/dbconfig/20200228-092631-marostegui.json [production]
09:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1087 after moving labs hosts back under it', diff saved to https://phabricator.wikimedia.org/P10553 and previous config saved to /var/cache/conftool/dbconfig/20200228-092453-marostegui.json [production]
09:21 <jynus> removed leftover labs prometheus target files from ops at prometheus1003, prometheus1004 [production]
08:44 <moritzm> installing openssh updates from buster point release [production]
08:44 <addshore> END warming wikidata term cache on db1126 for Q6-8 million T219123 (pass2 today) [production]