|
2020-02-10
§
|
| 07:00 |
<legoktm> |
updating docker-pkg for https://gerrit.wikimedia.org/r/571017 |
[releng] |
| 06:55 |
<vgutierrez> |
depool ncredir2002 and reimage as buster - T243391 |
[production] |
| 06:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool es1019', diff saved to https://phabricator.wikimedia.org/P10362 and previous config saved to /var/cache/conftool/dbconfig/20200210-065326-marostegui.json |
[production] |
| 06:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Fully repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10361 and previous config saved to /var/cache/conftool/dbconfig/20200210-065135-marostegui.json |
[production] |
| 06:47 |
<vgutierrez> |
depool cp4029 & reimage as buster - T242093 |
[production] |
| 06:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1019', diff saved to https://phabricator.wikimedia.org/P10360 and previous config saved to /var/cache/conftool/dbconfig/20200210-064553-marostegui.json |
[production] |
| 06:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10359 and previous config saved to /var/cache/conftool/dbconfig/20200210-064458-marostegui.json |
[production] |
| 06:39 |
<marostegui> |
Compress db1124:3318 - this will generate lag on s8 wiki replicas - T232446 |
[production] |
| 06:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10358 and previous config saved to /var/cache/conftool/dbconfig/20200210-063716-marostegui.json |
[production] |
| 06:23 |
<marostegui> |
Remove partitions from db1099:3311, db1099:3318 T239453 |
[production] |
| 06:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1099:3318 T239453', diff saved to https://phabricator.wikimedia.org/P10357 and previous config saved to /var/cache/conftool/dbconfig/20200210-062112-marostegui.json |
[production] |
| 06:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool repool db1099:3311 T239453', diff saved to https://phabricator.wikimedia.org/P10356 and previous config saved to /var/cache/conftool/dbconfig/20200210-061822-marostegui.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1091 T232446', diff saved to https://phabricator.wikimedia.org/P10355 and previous config saved to /var/cache/conftool/dbconfig/20200210-061656-marostegui.json |
[production] |
|
2020-02-08
§
|
| 22:53 |
<Framawiki> |
qdeled job 2929463 maj_articles_recents stuck for one month |
[tools.totoazero] |
| 19:12 |
<_joe_> |
set cpufreq governor to performance on mw1328 |
[production] |
| 17:17 |
<wm-bot> |
<maurelio> Webservice updated to kubernetes backend running on php7.2 now |
[tools.stewardbots] |
| 17:14 |
<wm-bot> |
<maurelio> Shutting down webservice to upgrade it |
[tools.stewardbots] |
| 17:04 |
<_joe_> |
restarted php7.2-fpm on mw1332 |
[production] |
| 16:53 |
<Urbanecm> |
mwscript resetAuthenticationThrottle.php --wiki=enwiki --signup --ip 12.24.27.50 |
[production] |
| 16:47 |
<gjg@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: Editathon in Charolette (duration: 00m 58s) |
[production] |
| 09:35 |
<elukey> |
created /wmf/data/raw/wikidata/dumps/all_ttl on hdfs |
[analytics] |
| 09:35 |
<elukey> |
created /wmf/data/raw/wikidata/dumps/all_json on hdfs |
[analytics] |
| 00:05 |
<Jeff_Green> |
switched payments.wikimedia.org to codfw datacenter due to T244610 |
[production] |
|
2020-02-07
§
|
| 23:20 |
<brennen> |
Updating dev-images docker-pkg files on contint1001 for T244382 |
[releng] |
| 23:20 |
<dpifke> |
puppetdb on deployment-puppetdb03 was killed by kernel OOM at Feb 7 09:50:29, per syslog. I just ran `systemctl start puppetdb` on that host, to fix puppet issues in beta. |
[releng] |
| 23:07 |
<bstorm_> |
upgraded toollabs-webservice for stetch toolsbeta to 0.60 T244611 |
[toolsbeta] |
| 22:20 |
<jeh> |
ceph: round 2 OSD failover and recovery testing on cloudcephosd1003.wikimedia.org T240718 |
[production] |
| 21:09 |
<bstorm_> |
upgraded toollabs-webservice package for stretch toolsbeta to 0.59 T244293 T244289 T234617 T156626 |
[toolsbeta] |
| 20:47 |
<mutante> |
OS install on new install_server VMs worked on second attempt, issues are gone. signed puppet certs for install1003.eqiad.wmnet, install2003.codfw.wmnet, initial puppet runs (T224576) |
[production] |
| 20:42 |
<jeh> |
ceph: OSD failover and recovery testing on cloudcephosd1003.wikimedia.org T240718 |
[production] |
| 20:32 |
<mutante> |
ganeti: attempting to reinstall install1003 which failed last time |
[production] |
| 19:10 |
<James_F> |
Zuul: [labs/tools/video-cut-tool-back-end] Add node10 CI T244079 |
[releng] |
| 19:01 |
<James_F> |
Zuul: [labs/tools/VideoCutTool] Switch from tox to node T244079 |
[releng] |
| 18:51 |
<bd808> |
Updated Ingress default-route to redirect https://tools.wmflabs.org/ to https://tools.wmflabs.org/admin/ |
[tools.fourohfour] |
| 18:45 |
<wm-bot> |
<maurelio> Deploy 454b2d5 |
[tools.mabot] |
| 18:11 |
<jeh> |
shutdown cloudvirt1016 for hardware maintenance T241882 |
[admin] |
| 17:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1019 after on-site maintenance T243963', diff saved to https://phabricator.wikimedia.org/P10350 and previous config saved to /var/cache/conftool/dbconfig/20200207-173850-marostegui.json |
[production] |
| 17:36 |
<twentyafterfour@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: sync InitializeSettings again for lols refs T233866 (duration: 01m 03s) |
[production] |
| 17:32 |
<twentyafterfour@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: sync https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/570929 refs T233866 (duration: 01m 02s) |
[production] |
| 17:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool es1019 after on-site maintenance T243963', diff saved to https://phabricator.wikimedia.org/P10349 and previous config saved to /var/cache/conftool/dbconfig/20200207-172541-marostegui.json |
[production] |
| 17:22 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: roll back all wikis to 1.35.0-wmf.16 refs T233866 |
[production] |
| 17:19 |
<marostegui> |
Start MySQL on es1019 after onsite maintenance T243963 |
[production] |
| 16:57 |
<halfak> |
deploying ores a6f4f14 |
[releng] |
| 16:57 |
<halfak> |
deploying ores a6f4f14 |
[deployment-prep] |
| 16:46 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 16:38 |
<filippo@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 16:13 |
<XioNoX> |
remove MSS clamping from eqiad/eqord/knams/esams |
[production] |