2501-2550 of 10000 results (83ms)
2020-02-28 ยง
16:02 <wm-bot> <root> Migrated to 2020 Kubernetes cluster [tools.quickpreset-migrate]
16:01 <wm-bot> <root> Migrated to 2020 Kubernetes cluster [tools.query]
16:01 <wm-bot> <root> Migrated to 2020 Kubernetes cluster [tools.quarrybot-enwiki]
16:00 <wm-bot> <root> Migrated to 2020 Kubernetes cluster [tools.qrcode-generator]
16:00 <wm-bot> <root> Running k8s-2020-migrate.sh with prefix=q [tools.admin]
16:00 <bd808> Devoicing stashbot in #wikimedia-cloud to reduce irc spam while migrating tools to 2020 Kubernetes cluster [tools]
15:54 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:39 <moritzm> installing libperl4-corelibs-perl updates from Stretch point release [production]
15:36 <elukey@deploy1001> Finished deploy [analytics/refinery@28fa2fc]: fix for refinery-drop-older-than - part 2 (duration: 13m 40s) [production]
15:28 <jeh> create OpenStack server group tools-elastic with anti-affinty policy enabled T236606 [tools]
15:24 <marostegui> Stop replication on db1077 from db1111 (its master) - T246447 [production]
15:22 <elukey@deploy1001> Started deploy [analytics/refinery@28fa2fc]: fix for refinery-drop-older-than - part 2 [production]
15:09 <jeh> create 3 new elasticsearch VMs tools-elastic-[1,2,3] T236606 [tools]
14:49 <joal> Drop test keyspaces in cassandra cluster [analytics]
14:20 <jeh> create new puppet prefixes for existing (no change in data) and new elasticsearch VMs [tools]
14:17 <gehel> rolling restart of elasticsearch/eqiad for JVM upgrade completed [production]
14:16 <gehel@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-restart (exit_code=0) [production]
14:15 <elukey@deploy1001> Finished deploy [analytics/refinery@2db36f4]: Fix refinery-drop-older-than script (duration: 14m 01s) [production]
14:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase weight from 100 to 300', diff saved to https://phabricator.wikimedia.org/P10558 and previous config saved to /var/cache/conftool/dbconfig/20200228-141035-marostegui.json [production]
14:01 <elukey@deploy1001> Started deploy [analytics/refinery@2db36f4]: Fix refinery-drop-older-than script [production]
13:58 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-restart [production]
13:32 <marostegui> Reset idrac from db1114 [production]
12:40 <hauskatze> Migrated https://github.com/wikimedia/mediawiki-extensions-NearbyPages to Gerrit; GitHub repo now watches the Gerrit one [releng]
12:11 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
12:06 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
11:57 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
11:04 <gehel@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-restart (exit_code=97) [production]
10:53 <jynus> labsdb1009-12 prometheus metrics restored after 90 minutes of unscheduled unavailability [production]
10:27 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-restart [production]
10:15 <gehel@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-restart (exit_code=99) [production]
10:13 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-restart [production]
10:01 <gehel@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-restart (exit_code=99) [production]
09:59 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-restart [production]
09:59 <gehel> starting rolling restart of elasticsearch/eqiad for JVM upgrade [production]
09:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1101:3318 from vslow,dump', diff saved to https://phabricator.wikimedia.org/P10555 and previous config saved to /var/cache/conftool/dbconfig/20200228-093653-marostegui.json [production]
09:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1087 into vslow,dump as it was there originally', diff saved to https://phabricator.wikimedia.org/P10554 and previous config saved to /var/cache/conftool/dbconfig/20200228-092631-marostegui.json [production]
09:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1087 after moving labs hosts back under it', diff saved to https://phabricator.wikimedia.org/P10553 and previous config saved to /var/cache/conftool/dbconfig/20200228-092453-marostegui.json [production]
09:21 <jynus> removed leftover labs prometheus target files from ops at prometheus1003, prometheus1004 [production]
08:44 <moritzm> installing openssh updates from buster point release [production]
08:44 <addshore> END warming wikidata term cache on db1126 for Q6-8 million T219123 (pass2 today) [production]
08:30 <moritzm> installing mariadb-10.3 update from buster point release (just client-side libs and tools, no mysqlds) [production]
08:24 <moritzm> installing cups updates from buster point release [production]
08:22 <marostegui> Stop db1087 and db2079 in sync [production]
08:22 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1087 to move labs hosts back under it', diff saved to https://phabricator.wikimedia.org/P10551 and previous config saved to /var/cache/conftool/dbconfig/20200228-082213-marostegui.json [production]
08:12 <addshore> START warming wikidata term cache on db1126 for Q6-8 million T219123 (pass2 today) (pass1 just finished) [production]
08:05 <moritzm> installing systemd bugfix update from Buster point release [production]
07:38 <addshore> START warming wikidata term cache on db1126 for Q6-8 million T219123 (pass1 today) [production]
07:31 <moritzm> installing gnutls28 bugfix update from Buster point release [production]
07:07 <bd808> Stopped broken webservice; edited package.json to have a valid entrypoint, restarted with proper node10 runtime. Still needs some work to fix redirect to invalid URL. [tools.unpkg]