1601-1650 of 10000 results (62ms)
2019-11-25 ยง
18:50 <bblack> cp[245]*: wipe daemon.log and restart syslog, again [production]
18:48 <mutante> mw1298 - pooling [production]
18:26 <bblack> cp[245]*: disk space exhausted, rm /var/log/daemon.log + restart rsyslog [production]
18:17 <bblack> cp4028: disk space exhausted, rm /var/log/daemon.log + restart rsyslog [production]
18:16 <effie> Restart php-fpm on mw* and wtp* servers in eqiad and codfw - T236963 [production]
18:07 <effie> Upgrade php-wikidiff2 to 1.10.0 to all servers - T236963 [production]
17:55 <gehel> restart wdqs-updater on all wdqs servers [production]
17:55 <onimisionipe@deploy1001> Finished deploy [wdqs/wdqs@4c5f503]: Revert New Blazegraph Build and WDQS Updates (duration: 10m 24s) [production]
17:50 <mobrovac@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Parsoid: Switch private wiki clients (Flow, VE) to Parsoid/PHP -- T229015 (duration: 00m 53s) [production]
17:45 <onimisionipe@deploy1001> Started deploy [wdqs/wdqs@4c5f503]: Revert New Blazegraph Build and WDQS Updates [production]
17:36 <marostegui> Upgrade kernel on db2125 T239042 [production]
17:25 <onimisionipe@deploy1001> Finished deploy [wdqs/wdqs@4c5f503]: New Blazegraph Build and WDQS Updates (duration: 12m 23s) [production]
17:19 <XioNoX> power down cr2-knams - T237030 [production]
17:14 <arlolra@deploy1001> Finished deploy [parsoid/deploy@e7faa19]: Updating Parsoid to a6bfdfa (duration: 08m 58s) [production]
17:12 <onimisionipe@deploy1001> Started deploy [wdqs/wdqs@4c5f503]: New Blazegraph Build and WDQS Updates [production]
17:05 <arlolra@deploy1001> Started deploy [parsoid/deploy@e7faa19]: Updating Parsoid to a6bfdfa [production]
16:48 <jynus> upgrading and restarting dbprov* hosts [production]
15:49 <ema> pool cp3064 with varnish-be T227432 [production]
15:36 <ema> cp3064 create filesystem on /dev/nvme0n1p1 (see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/552547/) and reboot T238494 [production]
15:22 <ema> cp3064 manual reboot after wmf-auto-reimage error: 'Unable to run wmf-auto-reimage-host: Failed to reboot_host' T238494 [production]
15:20 <ema> cp-ats: rolling ats-{tls,backend} restart to enable lua reload T233274 [production]
15:18 <gehel@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:14 <gehel@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:13 <mholloway-shell@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . [production]
15:11 <mholloway-shell@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . [production]
15:11 <ema@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:11 <ema> cp1075: ats-tls-restart to enable lua reload T233274 [production]
15:10 <mholloway-shell@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . [production]
15:09 <ema@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:03 <ema> cp1075: ats-backend-restart to enable lua reload T233274 [production]
15:02 <bblack@cumin1001> conftool action : set/pooled=yes; selector: name=cp3056.esams.wmnet [production]
15:00 <bblack@cumin1001> conftool action : set/weight=100; selector: name=cp3056.esams.wmnet,service=ats-be [production]
14:50 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
14:50 <XioNoX> enable cr3-esams:et-1/0/0 - T236767 [production]
14:45 <ema> depool cp3064 and reimage with varnish-be T227432 [production]
14:44 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
14:38 <marostegui> Remove triggers from archive table on s1 codfw sanitarium T234704 [production]
14:37 <marostegui> Deploy schema change on s1 codfw (this will generate lag on codfw) - T234066 T233135 [production]
14:23 <moritzm> upgrading OpenJDK 11 on an-conf* [production]
14:04 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
13:27 <elukey> set global read_only=1 on db1108's log database - T159170 [production]
13:16 <XioNoX> cleanup config on cr3-esams - T237031 [production]
13:15 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
13:11 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
13:06 <XioNoX> cleanup config on cr2-esams - T237031 [production]
13:02 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
12:59 <oblivian@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'blubberoid' for release 'staging' . [production]
12:48 <XioNoX> bundle esams-knams links on knams side - T237031 [production]
12:42 <XioNoX> bundle esams-knams links on esams side - T237031 [production]
12:27 <XioNoX> disable BGP to knams transits - T237031 [production]