3251-3300 of 10000 results (67ms)
2019-06-18 §
10:28 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:28 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:24 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:24 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:24 <jbond42> reboot iron.wikimedia.org [production]
10:19 <jbond42> reboot bast4001 [production]
10:08 <jbond42> reboot bast5001 [production]
10:01 <moritzm> upgrading acmechief* to latest Buster [production]
09:53 <ema> upgrade varnish packages to 5.1.3-1wm10 on all A:cp (no restarts yet) [production]
09:42 <jbond42> I will start a rolling reboot of all bastion servers at 10:00UTC [production]
09:36 <jijiki> restarting php-fpm in mwdebug1002 [production]
09:19 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:19 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:08 <elukey> reboot analytics-tool1004 a second time to pick up the new kernel upgrades [production]
08:54 <akosiaris@deploy1001> scap-helm termbox finished [production]
08:54 <akosiaris@deploy1001> scap-helm termbox cluster codfw completed [production]
08:54 <akosiaris@deploy1001> scap-helm termbox cluster eqiad completed [production]
08:54 <akosiaris@deploy1001> scap-helm termbox upgrade --install -f termbox-values.yaml production stable/termbox [namespace: termbox, clusters: eqiad,codfw] [production]
08:52 <akosiaris> deploy termbox T220402 [production]
08:52 <akosiaris@deploy1001> scap-helm termbox finished [production]
08:52 <akosiaris@deploy1001> scap-helm termbox cluster codfw completed [production]
08:52 <akosiaris@deploy1001> scap-helm termbox cluster eqiad completed [production]
08:52 <akosiaris@deploy1001> scap-helm termbox upgrade --install -f termbox-values.yaml production stable/termbox [namespace: termbox, clusters: eqiad,codfw] [production]
08:23 <marostegui> Stop MySQL on db2039 - T225988 [production]
08:16 <marostegui> Remove db2039 from tendril and zarcillo - T225988 [production]
07:45 <elukey> roll restart of cassandra on aqs* to pick up new openjdk upgrades [production]
07:39 <elukey> reboot matomo1001 for kernel upgrades [production]
07:36 <elukey> reboot archiva1001 for kernel upgrades [production]
07:32 <elukey> reboot analytics-tool100* and an-tool100* for kernel upgrades [production]
07:21 <elukey> upload matomo_3.9.1-3 to stretch-wikimedia and upgrade matomo1001 [production]
07:06 <moritzm> disabling TCP selective acknowledgements on a number of internal test hosts [production]
07:04 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2039 from config T221533 (duration: 00m 46s) [production]
07:03 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2039 from config T221533 (duration: 00m 51s) [production]
06:56 <onimisionipe> pooling maps1001 - reimage is complete - T224395 [production]
06:19 <marostegui> Stop slave and mysql on db1112 to copy its content to dbprov1001:/srv/backups/tmp/db1112 - T225981 [production]
05:54 <marostegui> Stop slave and mysql on db1112 to copy its content to dbstore1001:/srv/tmp/db1112 - T225981 [production]
04:44 <marostegui> Deploy schema change on db1073 (labtestwiki and labswiki) - T225643 [production]
04:28 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool pc1008 after optimizing its tables T210725 (duration: 00m 47s) [production]
03:46 <kartik@deploy1001> scap-helm cxserver finished [production]
03:46 <kartik@deploy1001> scap-helm cxserver cluster eqiad completed [production]
03:46 <kartik@deploy1001> scap-helm cxserver upgrade -f cxserver-eqiad-values.yaml production stable/cxserver [namespace: cxserver, clusters: eqiad] [production]
03:45 <kartik@deploy1001> scap-helm cxserver finished [production]
03:45 <kartik@deploy1001> scap-helm cxserver cluster codfw completed [production]
03:45 <kartik@deploy1001> scap-helm cxserver upgrade -f cxserver-codfw-values.yaml production stable/cxserver [namespace: cxserver, clusters: codfw] [production]
03:42 <kartik@deploy1001> scap-helm cxserver finished [production]
03:42 <kartik@deploy1001> scap-helm cxserver cluster staging completed [production]
03:42 <kartik@deploy1001> scap-helm cxserver upgrade -f cxserver-staging-values.yaml staging stable/cxserver [namespace: cxserver, clusters: staging] [production]
00:22 <RoanKattouw> Running populateRevisionSha1.php on dewikivoyage for T219816 [production]
00:15 <RoanKattouw> Running populateRevisionSha1.php on testwiki for T219816 [production]
2019-06-17 §
23:40 <Krinkle> Repopulating lost "coal.*" data in Graphite from NavigationTiming for 2019-04-17, ref T221401 [production]