1451-1500 of 10000 results (62ms)
2020-01-02 §
10:40 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:58 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:58 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2076 T241647', diff saved to https://phabricator.wikimedia.org/P10021 and previous config saved to /var/cache/conftool/dbconfig/20200102-085806-marostegui.json [production]
08:35 <marostegui> Upgrade db2090 [production]
08:26 <marostegui> Upgrade db2075 [production]
08:10 <marostegui> Deploy schema change on officewiki.flow_wiki_ref on s3 master (db1123) T241387 [production]
07:49 <marostegui> Deploy schema change on techconductwiki.flow_wiki_ref (empty table) on s3 master (db1123) T241387 [production]
07:26 <marostegui> Upgrade db2079 [production]
07:18 <marostegui> Deploy schema change on labswiki.flow_wiki_ref (empty table) T241387 [production]
06:46 <marostegui> Deploy schema change on db2131 - T241387 [production]
06:44 <marostegui> Repool labsdb1009 [production]
06:30 <marostegui> Upgrade labsdb1009 [production]
06:29 <marostegui> Remove revision partitions from db2087:3316 T239453 [production]
06:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2087:3316 - T239453', diff saved to https://phabricator.wikimedia.org/P10020 and previous config saved to /var/cache/conftool/dbconfig/20200102-062650-marostegui.json [production]
06:22 <marostegui> Depool labsdb1009 [production]
00:22 <ejegg> re-enabled fundraising cron jobs [production]
2020-01-01 §
21:13 <ejegg> stopped fundraising cron jobs to calculate EOY summaries [production]
04:57 <andrewbogott> depooling labweb1002 so I can hotfix labweb1001 for T240734 [production]
2019-12-31 §
16:57 <arturo> import more openstack packages into reprepro (T241347) thirdparty/openstack-pike-stretch [production]
11:48 <marostegui> Upgrade db2119 and db2106 [production]
11:35 <marostegui> Upgrade db2073 [production]
10:49 <marostegui> Upgrade db1108 with elukey [production]
10:34 <marostegui> Upgrade db2111 and db2123 [production]
10:23 <elukey> execute 'clear bfd session address fe80::5e5e:ab00:d3d:85ce' on cr3-knams - T240659 [production]
08:25 <marostegui> Upgrade db2128 [production]
08:24 <marostegui> Upgrade db1107 from 10.4.10 to 10.4.11 [production]
07:46 <marostegui> Upgrade db2126 db2125 db2108 db2107 [production]
07:11 <marostegui> Restart wikibugs - T241109 [production]
2019-12-30 §
16:00 <ema> cp1083: restart ats-tls and varnish-fe after crashes - T241593 [production]
11:45 <arturo> importing more packages into stretch-wikimedia/thirdparty/openstack-pike-stretch (T241347) [production]
2019-12-29 §
22:29 <gehel> repooling wdqs1007 [production]
17:58 <sbassett@deploy1001> Synchronized php-1.35.0-wmf.11/extensions/EventBus/includes/EventFactory.php: Security fix for T241410 (duration: 00m 56s) [production]
11:13 <ema> restarted wikibugs to fix phab irc notifications [production]
10:57 <ema> repool cp3061 T238305 [production]
10:07 <elukey> powercycle cp3061 - mgmt serial console not showing a working tty, no ssh [production]
10:06 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp3061.esams.wmnet [production]
2019-12-28 §
22:54 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: cirrus: re-sync move more_like traffic to codfw (duration: 00m 54s) [production]
21:54 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: cirrus: Move more_like traffic to codfw (duration: 00m 55s) [production]
21:35 <ebernhardson@deploy1001> Synchronized wmf-config/CirrusSearch-common.php: Reduce query load on cirrus elastic clusters (duration: 00m 55s) [production]
21:13 <ebernhardson@deploy1001> Synchronized wmf-config/CirrusSearch-common.php: Reduce query load on cirrus elastic clusters (duration: 00m 58s) [production]
2019-12-27 §
19:51 <gehel> restarted blazegraph on wdqs1007 [production]
19:45 <gehel> depool wdqs1007 - checking what is wrong with updater [production]
19:26 <gehel> restarting elasticsearch on elastic1038 - high load [production]
19:23 <gehel> restarting elasticsearch on elastic1036 - high load [production]
15:35 <jynus> upgrade and restart dbprov1002 [production]
12:16 <jynus> upgrade and restart dbprov1001 [production]
11:05 <arturo> import openstack pike packages into thirdparty/openstack-pike-stretch in install1002 (T241347) [production]
10:55 <arturo> import gpg key 0x56056AB2FEE4EECB in install1002 for openstack packages (T241347) [production]
10:51 <jynus> upgrade and restart dbprov2002 [production]