301-350 of 10000 results (72ms)
2019-09-30 §
09:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2091:3314 for a schema change - T233625', diff saved to https://phabricator.wikimedia.org/P9217 and previous config saved to /var/cache/conftool/dbconfig/20190930-091043-marostegui.json [production]
08:00 <moritzm> installing e2fsprogs security updates on Stretch/Buster [production]
07:56 <marostegui> Stop dbstore1003:3311 for troubleshooting [production]
06:47 <moritzm> installing exim security updates on buster [production]
2019-09-28 §
16:28 <vgutierrez> restarting acme-chief on acmechief1001 [production]
2019-09-27 §
22:44 <mutante> phab2001 - apt-get autoremove - remove unused python and ruby packages [production]
22:36 <mutante> phab2001 - upgrade php7.2 packages to 7.2.22 (T230024) [production]
22:03 <mutante> webperf1001, webperf2001: restart envoyproxy to pick up new cert with the right subject alt. names [production]
18:22 <mutante> mwdebug1001, mwdebug1002 - deleted from /srv/mediawiki/: php-1.34.0-wmf.16, .17, .18, .19 and .20 (current is .24) - usage back to about 57% (T234063) [production]
18:17 <mutante> mwdebug1001, mwdebug1002 - apt-get clean saves about 3GB and gets usage down from 94% to 87% on / (T234063) [production]
16:01 <XioNoX> delete BGP to AS34305 on cr2-esams [production]
15:34 <elukey> update pcc facts to add new hosts [production]
15:02 <moritzm> installing usb.ids update from Buster 10.1 point release [production]
14:45 <moritzm> installing ncurses bugfix update from Buster 10.1 point release [production]
14:39 <moritzm> installing postgresql-common bugfix update from Buster 10.1 point release [production]
14:32 <effie> Disable puppet and reload apache on mw* for 539465 and 539488 - T229792 [production]
13:33 <marostegui> Set candidate masters in dbctl T234039 [production]
13:31 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:29 <jmm@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:16 <moritzm> reimaging auth1002 to buster [production]
13:09 <akosiaris> reboot ganeti2001 T233906 [production]
13:08 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
13:08 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
13:03 <effie> Disable puppet on mwmaint1002 to test noc.wikimedia.org with PHP7 [production]
12:58 <elukey@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
12:56 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:48 <moritzm> installing openldap security updates on Buster [production]
12:43 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:41 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
12:37 <moritzm> killing stray processes from old openjdk-8 build on boron (probably test suite not properly terminated) [production]
12:30 <moritzm> installing glib2.0 security updates on Buster [production]
12:14 <moritzm> reimaging auth2001 to buster [production]
12:06 <moritzm> install gnupg2 security update from Buster 10.1 point release [production]
10:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2088:3311 db2091:3312 db2084:3314 db2089:3315 db2089:3316 db2087:3317 T233625', diff saved to https://phabricator.wikimedia.org/P9213 and previous config saved to /var/cache/conftool/dbconfig/20190927-104914-marostegui.json [production]
10:34 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
10:33 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
10:02 <urbanecm@deploy1001> Synchronized wmf-config/throttle.php: New throttle rule for Czech course (T234024) (duration: 00m 59s) [production]
09:30 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:30 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:06 <moritzm> running a few ferm tests on cp1008, puppet disabled [production]
07:36 <godog> swift eqiad-prod: remove ms-be1027 - T233289 [production]
05:42 <XioNoX> remove tcp-mss clamping from cr2-eqiad - T232602 [production]
05:30 <XioNoX> remove tcp-mss clamping from cr2-eqord - T232602 [production]
05:23 <XioNoX> remove tcp-mss clamping from cr1-eqiad - T232602 [production]
00:53 <twentyafterfour> hotfixing phabricator fatal exception refs T233998 [production]
2019-09-26 §
22:15 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T211620 Enable emails for certain notification types by default on officewiki (duration: 00m 56s) [production]
22:11 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Stop setting wgPageTriageNoIndexTemplates, never read (duration: 00m 57s) [production]
22:02 <jforrester@deploy1001> Synchronized wmf-config/filebackend.php: T228547 Stop sharding wgFileBackends shardViaHashLevels for math-render (duration: 00m 56s) [production]
21:59 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T228547 Stop setting wgMathFileBackend, wgMathPath, wgMathDirectory (unused) (duration: 00m 56s) [production]
21:57 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: T228547 Stop setting wgTexvc, wgMathTexvcCheckExecutable, wgMathCheckFiles (unused) (duration: 01m 00s) [production]