3551-3600 of 10000 results (77ms)
2020-02-10 §
08:32 <effie> update php-apcu on eqiad - T236800 [production]
08:29 <effie> rolling restart php-fpm on cloudweb2001-dev.wikimedia.org,mw[2135-2147,2150-2212,2214-2290].codfw.wmnet,wtp[2001-2020].codfw.wmnet [production]
08:23 <effie> update php-apcu on codfw - T236800 [production]
07:58 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:56 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:54 <moritzm> updating d-i netinst image for Stretch 9.12 point release (which bumped the kernel ABI) [production]
07:29 <moritzm> updating d-i netinst image for Buster 10.3 point release (which bumped the kernel ABI) [production]
07:09 <elukey> restore mw1347's mcrouter settings to its default (proxy threads 10 -> 5) [production]
07:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Place db1107 - MariaDB 10.4 on s1 with minimal weight - T242702', diff saved to https://phabricator.wikimedia.org/P10363 and previous config saved to /var/cache/conftool/dbconfig/20200210-070140-marostegui.json [production]
06:55 <vgutierrez> depool ncredir2002 and reimage as buster - T243391 [production]
06:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool es1019', diff saved to https://phabricator.wikimedia.org/P10362 and previous config saved to /var/cache/conftool/dbconfig/20200210-065326-marostegui.json [production]
06:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10361 and previous config saved to /var/cache/conftool/dbconfig/20200210-065135-marostegui.json [production]
06:47 <vgutierrez> depool cp4029 & reimage as buster - T242093 [production]
06:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es1019', diff saved to https://phabricator.wikimedia.org/P10360 and previous config saved to /var/cache/conftool/dbconfig/20200210-064553-marostegui.json [production]
06:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10359 and previous config saved to /var/cache/conftool/dbconfig/20200210-064458-marostegui.json [production]
06:39 <marostegui> Compress db1124:3318 - this will generate lag on s8 wiki replicas - T232446 [production]
06:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10358 and previous config saved to /var/cache/conftool/dbconfig/20200210-063716-marostegui.json [production]
06:23 <marostegui> Remove partitions from db1099:3311, db1099:3318 T239453 [production]
06:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1099:3318 T239453', diff saved to https://phabricator.wikimedia.org/P10357 and previous config saved to /var/cache/conftool/dbconfig/20200210-062112-marostegui.json [production]
06:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool repool db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10356 and previous config saved to /var/cache/conftool/dbconfig/20200210-061822-marostegui.json [production]
06:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10355 and previous config saved to /var/cache/conftool/dbconfig/20200210-061656-marostegui.json [production]
2020-02-09 §
05:11 <cdanis> T238305 hardreset cp3051 [production]
2020-02-08 §
19:12 <_joe_> set cpufreq governor to performance on mw1328 [production]
17:04 <_joe_> restarted php7.2-fpm on mw1332 [production]
16:53 <Urbanecm> mwscript resetAuthenticationThrottle.php --wiki=enwiki --signup --ip 12.24.27.50 [production]
16:47 <gjg@deploy1001> Synchronized wmf-config/throttle.php: SWAT: Editathon in Charolette (duration: 00m 58s) [production]
00:05 <Jeff_Green> switched payments.wikimedia.org to codfw datacenter due to T244610 [production]
2020-02-07 §
22:20 <jeh> ceph: round 2 OSD failover and recovery testing on cloudcephosd1003.wikimedia.org T240718 [production]
20:47 <mutante> OS install on new install_server VMs worked on second attempt, issues are gone. signed puppet certs for install1003.eqiad.wmnet, install2003.codfw.wmnet, initial puppet runs (T224576) [production]
20:42 <jeh> ceph: OSD failover and recovery testing on cloudcephosd1003.wikimedia.org T240718 [production]
20:32 <mutante> ganeti: attempting to reinstall install1003 which failed last time [production]
17:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es1019 after on-site maintenance T243963', diff saved to https://phabricator.wikimedia.org/P10350 and previous config saved to /var/cache/conftool/dbconfig/20200207-173850-marostegui.json [production]
17:36 <twentyafterfour@deploy1001> Synchronized wmf-config/InitialiseSettings.php: sync InitializeSettings again for lols refs T233866 (duration: 01m 03s) [production]
17:32 <twentyafterfour@deploy1001> Synchronized wmf-config/InitialiseSettings.php: sync https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/570929 refs T233866 (duration: 01m 02s) [production]
17:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es1019 after on-site maintenance T243963', diff saved to https://phabricator.wikimedia.org/P10349 and previous config saved to /var/cache/conftool/dbconfig/20200207-172541-marostegui.json [production]
17:22 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: roll back all wikis to 1.35.0-wmf.16 refs T233866 [production]
17:19 <marostegui> Start MySQL on es1019 after onsite maintenance T243963 [production]
16:46 <filippo@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
16:38 <filippo@cumin1001> START - Cookbook sre.ganeti.makevm [production]
16:13 <XioNoX> remove MSS clamping from eqiad/eqord/knams/esams [production]
16:05 <andrew@deploy1001> Finished deploy [horizon/deploy@bc777d6]: Fix for T243422 (duration: 03m 45s) [production]
16:04 <vgutierrez> pooling cp4030 with buster - T242093 [production]
16:03 <bblack> removing GRE MTU mitigations from cp[135]xxx - T232602 [production]
16:01 <andrew@deploy1001> Started deploy [horizon/deploy@bc777d6]: Fix for T243422 [production]
15:50 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
15:48 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:25 <vgutierrez> depool & reimage cp4030 as buster - T242093 [production]
15:21 <vgutierrez> pooling cp4031 with buster - T242093 [production]
15:20 <vgutierrez> pooling ncredir3001 running buster - T243391 [production]
15:18 <marostegui> Restart all instances on db1124 and db1125 to pick up a new replication filter - T240094 [production]