|
2020-02-10
§
|
| 08:29 |
<effie> |
rolling restart php-fpm on cloudweb2001-dev.wikimedia.org,mw[2135-2147,2150-2212,2214-2290].codfw.wmnet,wtp[2001-2020].codfw.wmnet |
[production] |
| 08:23 |
<effie> |
update php-apcu on codfw - T236800 |
[production] |
| 08:17 |
<legoktm> |
updating tox-mysqld docker-pkg image https://gerrit.wikimedia.org/r/571234 |
[releng] |
| 07:58 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 07:56 |
<vgutierrez@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:54 |
<moritzm> |
updating d-i netinst image for Stretch 9.12 point release (which bumped the kernel ABI) |
[production] |
| 07:43 |
<legoktm> |
updating tox docker-pkg image for gettext https://gerrit.wikimedia.org/r/571019 |
[releng] |
| 07:29 |
<moritzm> |
updating d-i netinst image for Buster 10.3 point release (which bumped the kernel ABI) |
[production] |
| 07:20 |
<legoktm> |
updating tox docker-pkg image (try #2) https://gerrit.wikimedia.org/r/571127 |
[releng] |
| 07:09 |
<elukey> |
restore mw1347's mcrouter settings to its default (proxy threads 10 -> 5) |
[production] |
| 07:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Place db1107 - MariaDB 10.4 on s1 with minimal weight - T242702', diff saved to https://phabricator.wikimedia.org/P10363 and previous config saved to /var/cache/conftool/dbconfig/20200210-070140-marostegui.json |
[production] |
| 07:00 |
<legoktm> |
updating docker-pkg for https://gerrit.wikimedia.org/r/571017 |
[releng] |
| 06:55 |
<vgutierrez> |
depool ncredir2002 and reimage as buster - T243391 |
[production] |
| 06:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1019', diff saved to https://phabricator.wikimedia.org/P10362 and previous config saved to /var/cache/conftool/dbconfig/20200210-065326-marostegui.json |
[production] |
| 06:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10361 and previous config saved to /var/cache/conftool/dbconfig/20200210-065135-marostegui.json |
[production] |
| 06:47 |
<vgutierrez> |
depool cp4029 & reimage as buster - T242093 |
[production] |
| 06:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1019', diff saved to https://phabricator.wikimedia.org/P10360 and previous config saved to /var/cache/conftool/dbconfig/20200210-064553-marostegui.json |
[production] |
| 06:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10359 and previous config saved to /var/cache/conftool/dbconfig/20200210-064458-marostegui.json |
[production] |
| 06:39 |
<marostegui> |
Compress db1124:3318 - this will generate lag on s8 wiki replicas - T232446 |
[production] |
| 06:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10358 and previous config saved to /var/cache/conftool/dbconfig/20200210-063716-marostegui.json |
[production] |
| 06:23 |
<marostegui> |
Remove partitions from db1099:3311, db1099:3318 T239453 |
[production] |
| 06:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1099:3318 T239453', diff saved to https://phabricator.wikimedia.org/P10357 and previous config saved to /var/cache/conftool/dbconfig/20200210-062112-marostegui.json |
[production] |
| 06:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool repool db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10356 and previous config saved to /var/cache/conftool/dbconfig/20200210-061822-marostegui.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10355 and previous config saved to /var/cache/conftool/dbconfig/20200210-061656-marostegui.json |
[production] |
|
2020-02-08
§
|
| 22:53 |
<Framawiki> |
qdeled job 2929463 maj_articles_recents stuck for one month |
[tools.totoazero] |
| 19:12 |
<_joe_> |
set cpufreq governor to performance on mw1328 |
[production] |
| 17:17 |
<wm-bot> |
<maurelio> Webservice updated to kubernetes backend running on php7.2 now |
[tools.stewardbots] |
| 17:14 |
<wm-bot> |
<maurelio> Shutting down webservice to upgrade it |
[tools.stewardbots] |
| 17:04 |
<_joe_> |
restarted php7.2-fpm on mw1332 |
[production] |
| 16:53 |
<Urbanecm> |
mwscript resetAuthenticationThrottle.php --wiki=enwiki --signup --ip 12.24.27.50 |
[production] |
| 16:47 |
<gjg@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: Editathon in Charolette (duration: 00m 58s) |
[production] |
| 09:35 |
<elukey> |
created /wmf/data/raw/wikidata/dumps/all_ttl on hdfs |
[analytics] |
| 09:35 |
<elukey> |
created /wmf/data/raw/wikidata/dumps/all_json on hdfs |
[analytics] |
| 00:05 |
<Jeff_Green> |
switched payments.wikimedia.org to codfw datacenter due to T244610 |
[production] |
|
2020-02-07
§
|
| 23:20 |
<brennen> |
Updating dev-images docker-pkg files on contint1001 for T244382 |
[releng] |
| 23:20 |
<dpifke> |
puppetdb on deployment-puppetdb03 was killed by kernel OOM at Feb 7 09:50:29, per syslog. I just ran `systemctl start puppetdb` on that host, to fix puppet issues in beta. |
[releng] |
| 23:07 |
<bstorm_> |
upgraded toollabs-webservice for stetch toolsbeta to 0.60 T244611 |
[toolsbeta] |
| 22:20 |
<jeh> |
ceph: round 2 OSD failover and recovery testing on cloudcephosd1003.wikimedia.org T240718 |
[production] |
| 21:09 |
<bstorm_> |
upgraded toollabs-webservice package for stretch toolsbeta to 0.59 T244293 T244289 T234617 T156626 |
[toolsbeta] |
| 20:47 |
<mutante> |
OS install on new install_server VMs worked on second attempt, issues are gone. signed puppet certs for install1003.eqiad.wmnet, install2003.codfw.wmnet, initial puppet runs (T224576) |
[production] |
| 20:42 |
<jeh> |
ceph: OSD failover and recovery testing on cloudcephosd1003.wikimedia.org T240718 |
[production] |
| 20:32 |
<mutante> |
ganeti: attempting to reinstall install1003 which failed last time |
[production] |
| 19:10 |
<James_F> |
Zuul: [labs/tools/video-cut-tool-back-end] Add node10 CI T244079 |
[releng] |
| 19:01 |
<James_F> |
Zuul: [labs/tools/VideoCutTool] Switch from tox to node T244079 |
[releng] |
| 18:51 |
<bd808> |
Updated Ingress default-route to redirect https://tools.wmflabs.org/ to https://tools.wmflabs.org/admin/ |
[tools.fourohfour] |
| 18:45 |
<wm-bot> |
<maurelio> Deploy 454b2d5 |
[tools.mabot] |
| 18:11 |
<jeh> |
shutdown cloudvirt1016 for hardware maintenance T241882 |
[admin] |