3351-3400 of 10000 results (85ms)
2020-01-02 §
06:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2087:3316 - T239453', diff saved to https://phabricator.wikimedia.org/P10020 and previous config saved to /var/cache/conftool/dbconfig/20200102-062650-marostegui.json [production]
06:22 <marostegui> Depool labsdb1009 [production]
00:22 <ejegg> re-enabled fundraising cron jobs [production]
2020-01-01 §
21:13 <ejegg> stopped fundraising cron jobs to calculate EOY summaries [production]
04:57 <andrewbogott> depooling labweb1002 so I can hotfix labweb1001 for T240734 [production]
2019-12-31 §
16:57 <arturo> import more openstack packages into reprepro (T241347) thirdparty/openstack-pike-stretch [production]
11:48 <marostegui> Upgrade db2119 and db2106 [production]
11:35 <marostegui> Upgrade db2073 [production]
10:49 <marostegui> Upgrade db1108 with elukey [production]
10:34 <marostegui> Upgrade db2111 and db2123 [production]
10:23 <elukey> execute 'clear bfd session address fe80::5e5e:ab00:d3d:85ce' on cr3-knams - T240659 [production]
08:25 <marostegui> Upgrade db2128 [production]
08:24 <marostegui> Upgrade db1107 from 10.4.10 to 10.4.11 [production]
07:46 <marostegui> Upgrade db2126 db2125 db2108 db2107 [production]
07:11 <marostegui> Restart wikibugs - T241109 [production]
2019-12-30 §
16:00 <ema> cp1083: restart ats-tls and varnish-fe after crashes - T241593 [production]
11:45 <arturo> importing more packages into stretch-wikimedia/thirdparty/openstack-pike-stretch (T241347) [production]
2019-12-29 §
22:29 <gehel> repooling wdqs1007 [production]
17:58 <sbassett@deploy1001> Synchronized php-1.35.0-wmf.11/extensions/EventBus/includes/EventFactory.php: Security fix for T241410 (duration: 00m 56s) [production]
11:13 <ema> restarted wikibugs to fix phab irc notifications [production]
10:57 <ema> repool cp3061 T238305 [production]
10:07 <elukey> powercycle cp3061 - mgmt serial console not showing a working tty, no ssh [production]
10:06 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3061.esams.wmnet [production]
2019-12-28 §
22:54 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: cirrus: re-sync move more_like traffic to codfw (duration: 00m 54s) [production]
21:54 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: cirrus: Move more_like traffic to codfw (duration: 00m 55s) [production]
21:35 <ebernhardson@deploy1001> Synchronized wmf-config/CirrusSearch-common.php: Reduce query load on cirrus elastic clusters (duration: 00m 55s) [production]
21:13 <ebernhardson@deploy1001> Synchronized wmf-config/CirrusSearch-common.php: Reduce query load on cirrus elastic clusters (duration: 00m 58s) [production]
2019-12-27 §
19:51 <gehel> restarted blazegraph on wdqs1007 [production]
19:45 <gehel> depool wdqs1007 - checking what is wrong with updater [production]
19:26 <gehel> restarting elasticsearch on elastic1038 - high load [production]
19:23 <gehel> restarting elasticsearch on elastic1036 - high load [production]
15:35 <jynus> upgrade and restart dbprov1002 [production]
12:16 <jynus> upgrade and restart dbprov1001 [production]
11:05 <arturo> import openstack pike packages into thirdparty/openstack-pike-stretch in install1002 (T241347) [production]
10:55 <arturo> import gpg key 0x56056AB2FEE4EECB in install1002 for openstack packages (T241347) [production]
10:51 <jynus> upgrade and restart dbprov2002 [production]
09:30 <jynus> upgrade and restart dbprov2001 [production]
2019-12-25 §
10:12 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:12 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:12 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:11 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
2019-12-24 §
15:33 <gehel@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
15:13 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:13 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:38 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:38 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:13 <gehel> data reload from wdqs1008 to wdqs1006 - T241418 [production]
14:11 <gehel@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
13:58 <jynus> tune2fs -m 0 /dev/mapper/wdqs1006--vg-data T241418 [production]
12:21 <volans@deploy1001> Finished deploy [debmonitor/deploy@39ad186]: Release v0.2.2 - T241206 (duration: 00m 40s) [production]