2851-2900 of 10000 results (84ms)
2020-02-10 §
11:57 <vgutierrez> pooling ncredir1002 with buster - T243391 [production]
11:43 <vgutierrez> pooling cp4027 with buster - T242093 [production]
11:38 <vgutierrez> depool ncredir1002 and reimage as buster - T243391 [production]
11:31 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:29 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:22 <vgutierrez> depooling cp5011 and cp5005 & reimage as buster - T242093 [production]
11:07 <vgutierrez> depool cp4027 & reimage as buster - T242093 [production]
11:07 <vgutierrez> pooling ncredir2001 with buster - T243391 [production]
11:03 <vgutierrez> pooling cp4028 with buster - T242093 [production]
10:50 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:48 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:47 <godog> remove old logs from /var/log/swift on swift hsots [production]
10:31 <vgutierrez> depool ncredir2001 and reimage as buster - T243391 [production]
10:26 <vgutierrez> depool cp4028 & reimage as buster - T242093 [production]
10:14 <moritzm> installing sudo security updates for buster [production]
08:53 <vgutierrez> pooling cp4029 with buster - T242093 [production]
08:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Increase weight from 1 to 5 for db1107 - T242702', diff saved to https://phabricator.wikimedia.org/P10364 and previous config saved to /var/cache/conftool/dbconfig/20200210-084446-marostegui.json [production]
08:43 <vgutierrez> pooling ncredir2002 with buster - T243391 [production]
08:34 <effie> rolling restart php-fpm on labweb[1001-1002].wikimedia.org,mw*.eqiad.wmnet,scandium.eqiad.wmnet, wtp[1025-1048].eqiad.wmnet [production]
08:32 <effie> update php-apcu on eqiad - T236800 [production]
08:29 <effie> rolling restart php-fpm on cloudweb2001-dev.wikimedia.org,mw[2135-2147,2150-2212,2214-2290].codfw.wmnet,wtp[2001-2020].codfw.wmnet [production]
08:23 <effie> update php-apcu on codfw - T236800 [production]
07:58 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:56 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:54 <moritzm> updating d-i netinst image for Stretch 9.12 point release (which bumped the kernel ABI) [production]
07:29 <moritzm> updating d-i netinst image for Buster 10.3 point release (which bumped the kernel ABI) [production]
07:09 <elukey> restore mw1347's mcrouter settings to its default (proxy threads 10 -> 5) [production]
07:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Place db1107 - MariaDB 10.4 on s1 with minimal weight - T242702', diff saved to https://phabricator.wikimedia.org/P10363 and previous config saved to /var/cache/conftool/dbconfig/20200210-070140-marostegui.json [production]
06:55 <vgutierrez> depool ncredir2002 and reimage as buster - T243391 [production]
06:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool es1019', diff saved to https://phabricator.wikimedia.org/P10362 and previous config saved to /var/cache/conftool/dbconfig/20200210-065326-marostegui.json [production]
06:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10361 and previous config saved to /var/cache/conftool/dbconfig/20200210-065135-marostegui.json [production]
06:47 <vgutierrez> depool cp4029 & reimage as buster - T242093 [production]
06:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es1019', diff saved to https://phabricator.wikimedia.org/P10360 and previous config saved to /var/cache/conftool/dbconfig/20200210-064553-marostegui.json [production]
06:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10359 and previous config saved to /var/cache/conftool/dbconfig/20200210-064458-marostegui.json [production]
06:39 <marostegui> Compress db1124:3318 - this will generate lag on s8 wiki replicas - T232446 [production]
06:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10358 and previous config saved to /var/cache/conftool/dbconfig/20200210-063716-marostegui.json [production]
06:23 <marostegui> Remove partitions from db1099:3311, db1099:3318 T239453 [production]
06:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1099:3318 T239453', diff saved to https://phabricator.wikimedia.org/P10357 and previous config saved to /var/cache/conftool/dbconfig/20200210-062112-marostegui.json [production]
06:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool repool db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10356 and previous config saved to /var/cache/conftool/dbconfig/20200210-061822-marostegui.json [production]
06:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10355 and previous config saved to /var/cache/conftool/dbconfig/20200210-061656-marostegui.json [production]
2020-02-09 §
05:11 <cdanis> T238305 hardreset cp3051 [production]
2020-02-08 §
19:12 <_joe_> set cpufreq governor to performance on mw1328 [production]
17:04 <_joe_> restarted php7.2-fpm on mw1332 [production]
16:53 <Urbanecm> mwscript resetAuthenticationThrottle.php --wiki=enwiki --signup --ip 12.24.27.50 [production]
16:47 <gjg@deploy1001> Synchronized wmf-config/throttle.php: SWAT: Editathon in Charolette (duration: 00m 58s) [production]
00:05 <Jeff_Green> switched payments.wikimedia.org to codfw datacenter due to T244610 [production]
2020-02-07 §
22:20 <jeh> ceph: round 2 OSD failover and recovery testing on cloudcephosd1003.wikimedia.org T240718 [production]
20:47 <mutante> OS install on new install_server VMs worked on second attempt, issues are gone. signed puppet certs for install1003.eqiad.wmnet, install2003.codfw.wmnet, initial puppet runs (T224576) [production]
20:42 <jeh> ceph: OSD failover and recovery testing on cloudcephosd1003.wikimedia.org T240718 [production]
20:32 <mutante> ganeti: attempting to reinstall install1003 which failed last time [production]