451-500 of 10000 results (74ms)
2019-07-05 §
11:05 <jmm@cumin2001> START - Cookbook sre.ganeti.makevm [production]
11:05 <jmm@cumin2001> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) [production]
11:04 <jmm@cumin2001> START - Cookbook sre.ganeti.makevm [production]
11:00 <ema> depool cp1088 and reimage as upload_ats T226638 [production]
10:55 <ema> pool cp1086 w/ ATS backend T226638 [production]
10:29 <moritzm> rebooting debug proxies to pick up MDS-enabled qemu [production]
10:29 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:29 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:23 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:23 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:23 <moritzm> rebooting seaborgium to pick up correct Stretch kernel [production]
10:15 <moritzm> rebooting serpens to pick up correct Stretch kernel [production]
10:14 <moritzm> fixed up kernel packages on serpens/seaborgium, these were dist-upgraded from jessie, but the correct kernel packages for Stretch were not setup, as such there were still stuck with an old jessie kernel [production]
10:06 <jiji@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:06 <jiji@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:02 <jijiki> Rolling rebood rdb* hosts - T227304 [production]
10:00 <moritzm> rebooting seaborgium to pick up MDS-enabled qemu [production]
09:51 <moritzm> rebooting serpens to pick up MDS-enabled qemu [production]
09:48 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:48 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:39 <ema> depool cp1086 and reimage as upload_ats T226638 [production]
09:31 <moritzm> rebooting LDAP replicas in eqiad [production]
09:28 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:28 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:15 <gehel@puppetmaster1001> conftool action : set/pooled=inactive; selector: dc=codfw,name=elastic2054.codfw.wmnet [production]
09:01 <moritzm> rebooting kraz (irc.wikimedia.org) to pick up MDS-enabled qemu [production]
08:54 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:54 <jmm@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:57 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1087 (duration: 00m 48s) [production]
07:35 <moritzm> installing imagemagick security updates on jessie [production]
07:23 <moritzm> installing wireshark security updates on jessie [production]
07:17 <marostegui> Compress small wikis on labsdb1009 T222978 [production]
07:13 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1087 (duration: 00m 52s) [production]
06:46 <jynus@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1109 with full weight (duration: 00m 49s) [production]
06:35 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove old comments (duration: 00m 50s) [production]
05:56 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool db1104 after upgrade (duration: 00m 49s) [production]
05:43 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1104 after upgrade (duration: 00m 49s) [production]
05:23 <marostegui> Upgrade db1104 T227062 [production]
05:22 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1104 for upgrade (duration: 00m 51s) [production]
05:18 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
05:17 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
05:09 <marostegui> Stop MySQL on db1069 for decommission T227166 [production]
05:08 <vgutierrez@cumin1001> START - Cookbook sre.ganeti.makevm [production]
05:08 <vgutierrez@cumin1001> START - Cookbook sre.ganeti.makevm [production]
05:02 <marostegui> Remove db1069 from tendril and zarcillo - T227166 [production]
2019-07-04 §
21:50 <volans@deploy1001> Finished deploy [debmonitor/deploy@0ee26a3]: Deploy Debmonitor v0.1.10 (duration: 00m 48s) [production]
21:50 <volans@deploy1001> Started deploy [debmonitor/deploy@0ee26a3]: Deploy Debmonitor v0.1.10 [production]
21:35 <volans> forcing reboot of elastic2054 from console, host unresponsive - T227298 [production]
17:03 <AndyRussG> re-enabled banner impressions loader job [production]
16:36 <ema> pool cp1084 w/ ATS backend T226638 [production]