1-50 of 10000 results (17ms)
2020-07-18 §
21:41 <shdubsh> restart logstash on logstash200[456] [production]
21:14 <shdubsh> bounce logstash on logstash1007 [production]
21:10 <shdubsh> bounce logstash on logstash1008 [production]
21:06 <shdubsh> bounce logstash on logstash1009 [production]
20:52 <marostegui> Due to db1082 crash there will be replication lag on s5 on labsdb hosts - T258336 [production]
20:37 <cdanis@cumin1001> dbctl commit (dc=all): 'depool db1082, it crashed', diff saved to https://phabricator.wikimedia.org/P11951 and previous config saved to /var/cache/conftool/dbconfig/20200718-203704-cdanis.json [production]
00:13 <dpifke> Performing one-time expiration of ArcLamp files older than 40 days (normal retention is 45 days), to solve disk space issue until either Ganeti issue is solved or compressed logfile support is merged. [production]
2020-07-17 §
22:23 <dpifke> Cherry picking https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/613740/ into beta ArcLamp. [releng]
21:16 <dpifke> Removing MongoDB packages and data from webperf1002. [production]
17:39 <dpifke@deploy1001> Finished deploy [performance/arc-lamp@a5d2fd3]: (no justification provided) (duration: 00m 05s) [production]
17:38 <dpifke@deploy1001> Started deploy [performance/arc-lamp@a5d2fd3]: (no justification provided) [production]
17:03 <thcipriani> restarting apache2 on deployment-mediawiki-07 [releng]
16:47 <bd808> Enabled Puppet on tools-proxy-06 following successful test (T102367) [tools]
16:29 <bd808> Disabled Puppet on tools-proxy-06 to test nginx config changes manually (T102367) [tools]
14:28 <thcipriani> integration-agent-docker-1011:/srv @ 20%, bringing node back online [releng]
13:53 <akosiaris> powercycle kubernetes2002 [production]
12:34 <elukey> deprecate pivot.wikimedia.org (to ease CAS work) [analytics]
12:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1104', diff saved to https://phabricator.wikimedia.org/P11944 and previous config saved to /var/cache/conftool/dbconfig/20200717-122400-marostegui.json [production]
12:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11941 and previous config saved to /var/cache/conftool/dbconfig/20200717-120126-marostegui.json [production]
11:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11940 and previous config saved to /var/cache/conftool/dbconfig/20200717-115155-marostegui.json [production]
11:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11939 and previous config saved to /var/cache/conftool/dbconfig/20200717-113800-marostegui.json [production]
11:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1104', diff saved to https://phabricator.wikimedia.org/P11938 and previous config saved to /var/cache/conftool/dbconfig/20200717-113050-marostegui.json [production]
11:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1104', diff saved to https://phabricator.wikimedia.org/P11937 and previous config saved to /var/cache/conftool/dbconfig/20200717-112413-marostegui.json [production]
09:15 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1280.eqiad.wmnet [production]
09:12 <elukey@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=mw1280.eqiad.wmnet [production]
09:04 <addshore> reload zuul for [[gerrit:612562]] T257549 layout: [mediawiki/extensions/Wikibase] make apitests voting [releng]
08:48 <moritzm> imported prometheus-atlas-exporter 1.0+git20191204.ffafab7-2 to buster-wikimedia T247967 [production]
08:29 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
08:05 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
07:54 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
07:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1104', diff saved to https://phabricator.wikimedia.org/P11936 and previous config saved to /var/cache/conftool/dbconfig/20200717-075124-marostegui.json [production]
07:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1111', diff saved to https://phabricator.wikimedia.org/P11935 and previous config saved to /var/cache/conftool/dbconfig/20200717-074335-marostegui.json [production]
07:34 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
07:34 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
07:33 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
07:33 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
07:32 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
07:30 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
06:30 <XioNoX> rename msw1-codfw interface range [production]
06:28 <XioNoX> rename msw1-eqiad interface range [production]
04:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1111', diff saved to https://phabricator.wikimedia.org/P11934 and previous config saved to /var/cache/conftool/dbconfig/20200717-044748-marostegui.json [production]
04:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1092', diff saved to https://phabricator.wikimedia.org/P11933 and previous config saved to /var/cache/conftool/dbconfig/20200717-044658-marostegui.json [production]
01:09 <samwilson1> Updated wsexport-dev01 and wsexport-prod01 to latest version of wsexport. T255512. [wikisource]
00:18 <dpifke> Cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/613359 into beta puppet. [releng]
2020-07-16 §
23:49 <Reedy> unstuck beta ci jobs [releng]
23:46 <Krinkle> Beta Jenkins has been stuck for 5h+ (beta-mediawiki-config-update-eqiad on https://integration.wikimedia.org/zuul/ ) [releng]
22:15 <mutante> testreduce1001 manually git clone 'scandium' branch of integration/visualdiff into /srv/visualdiff (T257906) [production]
21:54 <crusnov@deploy1001> Finished deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 part 3 (duration: 01m 49s) [production]
21:52 <crusnov@deploy1001> Started deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 part 3 [production]
21:42 <crusnov@deploy1001> Finished deploy [netbox/deploy@39c5cae]: Deploying Netbox 2.8.7 part 2 (duration: 01m 33s) [production]