|
2020-01-13
§
|
| 20:37 |
<clarakosi@deploy1001> |
Finished deploy [restbase/deploy@bfdd342]: Use parsoid_uri, add ngwiki. T241756, T240771 (duration: 15m 41s) |
[production] |
| 20:29 |
<bd808> |
Restarted webservice to fix broken registration with the front proxy (T242538) |
[tools.ato] |
| 20:21 |
<clarakosi@deploy1001> |
Started deploy [restbase/deploy@bfdd342]: Use parsoid_uri, add ngwiki. T241756, T240771 |
[production] |
| 19:39 |
<tgr> |
ran disableOATHAuthForUser.php for T242543 |
[production] |
| 19:34 |
<James_F> |
Zuul: [mediawiki/extensions/NativeSvgHandler] Add basic CI via quibble |
[releng] |
| 19:22 |
<tgr@deploy1001> |
Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:509914|Revert a temporary CommonsMetadata cache validation hook that has been unneeded for a long time]] (duration: 00m 56s) |
[production] |
| 18:54 |
<bd808> |
Moved redirect only webservice from grid engine to 2020 Kubernetes cluster using technique from https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Make_a_tool_redirect_to_another_tool_WITHOUT_running_a_webservice (T242538) |
[tools.quarry] |
| 18:12 |
<bd808> |
`kubectl delete pod versions-57b986d64f-7fdmd` to test latest php7.3 image |
[tools.versions] |
| 17:48 |
<bd808> |
Running `puppet ca destroy` for each unsigned cert on tools-puppetmaster-01 (T242642) |
[tools] |
| 16:50 |
<James_F> |
Docker: Publishing java7-sonar-scanner:0.5.3 on contint1001 |
[releng] |
| 16:42 |
<bd808> |
Cordoned and fixed puppet on tools-k8s-worker-12. Rebooting now. T242559 |
[tools] |
| 16:41 |
<bstorm_> |
There was a filesystem unclean and other problems on the "old cluster" worker node 1001. Rebooting it in case that helps. |
[toolsbeta] |
| 16:33 |
<bd808> |
Cordoned and fixed puppet on tools-k8s-worker-11. Rebooting now. T242559 |
[tools] |
| 16:31 |
<bd808> |
Cordoned and fixed puppet on tools-k8s-worker-10. Rebooting now. T242559 |
[tools] |
| 16:26 |
<bd808> |
Cordoned and fixed puppet on tools-k8s-worker-9. Rebooting now. T242559 |
[tools] |
| 15:56 |
<moritzm> |
installing cyrus-sasl security updates |
[production] |
| 15:19 |
<moritzm> |
remove hassium in Ganeti T224567 |
[production] |
| 15:19 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 15:18 |
<jmm@cumin2001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 15:18 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'rbac-deploy-clusterrole' . |
[production] |
| 15:18 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 15:00 |
<joal> |
Deploy hdfs-tools 0.0.3 using scap |
[analytics] |
| 15:00 |
<joal@deploy1001> |
Finished deploy [analytics/hdfs-tools/deploy@a1b4d34]: Deploy hdfs-rsync bug correction (duration: 00m 08s) |
[production] |
| 15:00 |
<joal@deploy1001> |
Started deploy [analytics/hdfs-tools/deploy@a1b4d34]: Deploy hdfs-rsync bug correction |
[production] |
| 14:58 |
<jmm@cumin2001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 14:57 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 14:55 |
<moritzm> |
remove hassaleh in Ganeti T224567 |
[production] |
| 14:47 |
<jeh> |
finished testing ceph on cloudvirt1022, revert to local disk puppet role and recreate canary instance T225320 |
[openstack] |
| 14:24 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:558017| Bumping portals to master (563985)]] (duration: 00m 55s) |
[production] |
| 14:24 |
<joal> |
Releasing hdfs-tools 0.0.3 to archiva |
[analytics] |
| 14:24 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:558017| Bumping portals to master (563985)]] (duration: 00m 56s) |
[production] |
| 13:34 |
<arturo> |
[¢odfw1dev] prevent neutron from allocating floating IPs from the wrong subnet by doing `neutron subnet-update --allocation-pool start=208.80.153.190,end=208.80.153.190 cloud-instances-transport1-b-codfw` (T242594) |
[admin] |
| 13:11 |
<moritzm> |
upgrade mw canaries to PHP 7.2.26 T241222 |
[production] |
| 12:54 |
<elukey> |
restart hue to re-apply user hive limits (again) |
[analytics] |
| 12:08 |
<Urbanecm> |
EU SWAT done |
[production] |
| 12:08 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: c7cf53c: Deploy partial blocks on enwiki (T242569) (duration: 00m 55s) |
[production] |
| 11:58 |
<jdrewniak@deploy1001> |
Synchronized portals: Wikimedia Portals Update: [[gerrit:558017| Bumping portals to master (563985)]] (duration: 00m 55s) |
[production] |
| 11:57 |
<jdrewniak@deploy1001> |
Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:558017| Bumping portals to master (563985)]] (duration: 00m 55s) |
[production] |
| 11:42 |
<moritzm> |
upgrading remaining mwdebug* servers and mw1261 to PHP 7.2.26 T241222 |
[production] |
| 11:04 |
<volans@deploy1001> |
Finished deploy [debmonitor/deploy@265059b]: Release v0.2.3 (duration: 01m 10s) |
[production] |
| 11:03 |
<volans@deploy1001> |
Started deploy [debmonitor/deploy@265059b]: Release v0.2.3 |
[production] |
| 10:51 |
<vgutierrez> |
pooling esams for ncredir - T242321 |
[production] |
| 09:38 |
<moritzm> |
rename Ganeti group in ulsfo from "default" to "row_1" |
[production] |
| 09:16 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:16 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 07:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1112', diff saved to https://phabricator.wikimedia.org/P10134 and previous config saved to /var/cache/conftool/dbconfig/20200113-075334-marostegui.json |
[production] |
| 07:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1112', diff saved to https://phabricator.wikimedia.org/P10133 and previous config saved to /var/cache/conftool/dbconfig/20200113-073656-marostegui.json |
[production] |
| 07:30 |
<XioNoX> |
cr3-knams> clear bfd session fe80::5e5e:ab00:d3d:85c - T240659 |
[production] |
| 07:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1112', diff saved to https://phabricator.wikimedia.org/P10132 and previous config saved to /var/cache/conftool/dbconfig/20200113-072611-marostegui.json |
[production] |
| 06:45 |
<marostegui> |
Upgrade db1112 |
[production] |