The Monasca-transform DevStack plugin is tested only on Ubuntu 16.04 (Xenial).
A short cut to running monasca-transform in devstack is implemented with vagrant.
and deploys using that. Changes made by the user need to be committed in order to be used in the devstack instance. It is important therefore that changes should not be pushed from the vm as the unevaluated commit would be pushed.
Once the deploy is up use the following commands to set up tox.
This mostly re-does the work of the devstack plugin, updating the code from the shared directory, regenerating the venv and the zip that is passed to spark during the spark-submit call. The configuration and the transform and pre transform specs in the database are updated with fresh copies, along with driver and service python code.
If refresh_monasca_transform.sh script completes successfully you should see a message like the following in the console.
Here are the normal steps a developer can take to make any code changes. It is essential that the developer runs all tests in functional tests in a devstack environment before submitting any changes for review/merge.
4. tools/vagrant/refresh_monasca_transform.sh (See "Updating the code for dev"
Note: It is mandatory to run functional unit tests before submitting any changes for review/merge. These can be currently be run only in a devstack VM since tests need access to Apache Spark libraries. This is accomplished by setting SPARK_HOME environment variable which is being done in tox.ini.
How to find and fix test failures ?
in test or in code where you want to start python debugger.
In a devstack environment ports on which Spark Streaming UI (4040), Spark Master(18080) and Spark Worker (18081) UI are available are forwarded to the host and are accessible from the host machine.
Monasca-transform requires supporting services, such as Kafka and Zookeeper, also are set up. So just adding “enable_plugin monasca-transform” to a default DevStack local.conf is not sufficient to configure a working DevStack deployment unless these services are also added.
Please reference the devstack/settings file for an example of a working list of plugins and services as used by the Vagrant deployment.
This is a work in progress. There are a number of improvements necessary to improve value as a development tool.
Shorten initial deploy Currently the services deployed are the default set plus all of monasca. It’s quite possible that not all of this is necessary to develop monasca-transform. So some services may be dropped in order to shorten the deploy.