1-50 of 10000 results (19ms)
2020-07-19 §
19:16 <marostegui> Upgrade and reboot db1085 T258360 [production]
18:57 <marostegui> Start mysql on db1082 T258336 [production]
18:51 <marostegui> Upgrade and reboot db1082 T258336 [production]
18:45 <cdanis@cumin1001> dbctl commit (dc=all): 'db1085 also crashed', diff saved to https://phabricator.wikimedia.org/P11952 and previous config saved to /var/cache/conftool/dbconfig/20200719-184511-cdanis.json [production]
18:06 <Urbanecm> Run mwscript emptyUserGroup.php --wiki=testwiki contestadmin (T256555) [production]
14:59 <ZI_Jony> restarted all bots due to ping timeout. [cvn]
2020-07-18 §
21:41 <shdubsh> restart logstash on logstash200[456] [production]
21:14 <shdubsh> bounce logstash on logstash1007 [production]
21:10 <shdubsh> bounce logstash on logstash1008 [production]
21:06 <shdubsh> bounce logstash on logstash1009 [production]
20:52 <marostegui> Due to db1082 crash there will be replication lag on s5 on labsdb hosts - T258336 [production]
20:37 <cdanis@cumin1001> dbctl commit (dc=all): 'depool db1082, it crashed', diff saved to https://phabricator.wikimedia.org/P11951 and previous config saved to /var/cache/conftool/dbconfig/20200718-203704-cdanis.json [production]
00:13 <dpifke> Performing one-time expiration of ArcLamp files older than 40 days (normal retention is 45 days), to solve disk space issue until either Ganeti issue is solved or compressed logfile support is merged. [production]
2020-07-17 §
22:23 <dpifke> Cherry picking https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/613740/ into beta ArcLamp. [releng]
21:16 <dpifke> Removing MongoDB packages and data from webperf1002. [production]
17:39 <dpifke@deploy1001> Finished deploy [performance/arc-lamp@a5d2fd3]: (no justification provided) (duration: 00m 05s) [production]
17:38 <dpifke@deploy1001> Started deploy [performance/arc-lamp@a5d2fd3]: (no justification provided) [production]
17:03 <thcipriani> restarting apache2 on deployment-mediawiki-07 [releng]
16:47 <bd808> Enabled Puppet on tools-proxy-06 following successful test (T102367) [tools]
16:29 <bd808> Disabled Puppet on tools-proxy-06 to test nginx config changes manually (T102367) [tools]
14:28 <thcipriani> integration-agent-docker-1011:/srv @ 20%, bringing node back online [releng]
13:53 <akosiaris> powercycle kubernetes2002 [production]
12:34 <elukey> deprecate pivot.wikimedia.org (to ease CAS work) [analytics]
12:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1104', diff saved to https://phabricator.wikimedia.org/P11944 and previous config saved to /var/cache/conftool/dbconfig/20200717-122400-marostegui.json [production]
12:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11941 and previous config saved to /var/cache/conftool/dbconfig/20200717-120126-marostegui.json [production]
11:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11940 and previous config saved to /var/cache/conftool/dbconfig/20200717-115155-marostegui.json [production]
11:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P11939 and previous config saved to /var/cache/conftool/dbconfig/20200717-113800-marostegui.json [production]
11:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1104', diff saved to https://phabricator.wikimedia.org/P11938 and previous config saved to /var/cache/conftool/dbconfig/20200717-113050-marostegui.json [production]
11:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1104', diff saved to https://phabricator.wikimedia.org/P11937 and previous config saved to /var/cache/conftool/dbconfig/20200717-112413-marostegui.json [production]
09:15 <elukey@puppetmaster1001> conftool action : set/pooled=yes; selector: name=mw1280.eqiad.wmnet [production]
09:12 <elukey@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=mw1280.eqiad.wmnet [production]
09:04 <addshore> reload zuul for [[gerrit:612562]] T257549 layout: [mediawiki/extensions/Wikibase] make apitests voting [releng]
08:48 <moritzm> imported prometheus-atlas-exporter 1.0+git20191204.ffafab7-2 to buster-wikimedia T247967 [production]
08:29 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
08:05 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
07:54 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
07:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1104', diff saved to https://phabricator.wikimedia.org/P11936 and previous config saved to /var/cache/conftool/dbconfig/20200717-075124-marostegui.json [production]
07:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1111', diff saved to https://phabricator.wikimedia.org/P11935 and previous config saved to /var/cache/conftool/dbconfig/20200717-074335-marostegui.json [production]
07:34 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
07:34 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
07:33 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
07:33 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
07:32 <akosiaris@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
07:30 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
06:30 <XioNoX> rename msw1-codfw interface range [production]
06:28 <XioNoX> rename msw1-eqiad interface range [production]
04:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1111', diff saved to https://phabricator.wikimedia.org/P11934 and previous config saved to /var/cache/conftool/dbconfig/20200717-044748-marostegui.json [production]
04:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1092', diff saved to https://phabricator.wikimedia.org/P11933 and previous config saved to /var/cache/conftool/dbconfig/20200717-044658-marostegui.json [production]
01:09 <samwilson1> Updated wsexport-dev01 and wsexport-prod01 to latest version of wsexport. T255512. [wikisource]
00:18 <dpifke> Cherry picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/613359 into beta puppet. [releng]