OpenStack Nova Compute Cannot Launch Instance after Upgrade to Fedora 21

I updated my system to Fedora 21 beta and then found that I could not use OpenStack. Every effort to launch an instance failed:

libvirtError: operation failed: filter 'nova-no-nd-reflection' already exists with uuid 728f13cf-f1a9-4e0e-add7-357b93d052ee

Removing the openstack packages, clearing the data and reinstalling openstack did not help.

The problem seems to be an incompatibility  between nova-compute and the libvirt libraries.

Upgrading the libvirt libraries to rawhide did not help.

Finally, I used rpm to downgrade libvirt using downloaded packages from the fedora 20 repository, ignoring the dependency problems.

$ sudo rpm -U --oldpackage --nodeps libvirt*1.1.3*rpm

All seems well for now.

Openstack Nova Compute fails to start after updating to 2014.1.3-2.fc21

OpenStack nova compute service failed to start after the update to th e openstack-icehouse version 2014.1.3-2.fc21  - the reason appeared to be:

"Failed to add interface: can't add lo to bridge br100: Invalid argument"

This bug report, though marked as invalid, helped, The problem was a change in the file nova/network/linux_net.py.

The following exception had been in the wrong place earlier and was being ignored:

if (err and err != "device %s is already a member of a bridge; "
              "can't enslave it to bridge %s.\n" % (interface, bridge)):
       msg = _('Failed to add interface: %s') % err
       raise exception.NovaException(msg)

The interface for the flat network is set to 'lo' even if the nova.conf file does not define it. network['bridge_interface'] seems to be set to 'lo' in the following line:

iface = CONF.flat_interface or network['bridge_interface']

One workaround was to change the following line in ensure_bridge method:
<         if interface:
---
>         if interface and interface != 'lo':

Subsequently, I came across another workaround in a bug report by creating a dummy interface.

The virtual machines were stuck in 'powering on' state after the update. This time solving it was easy!

Experimenting with open source big data applications on a personal desktop

I decided to learn OpenStack, Hadoop, Elasticsearch and related open source software products using my quad core desktop. I had to raise the RAM to 16GB and add a 1 terabyte disk.

However, after the hardware upgrade, it has been remarkably simple to experiment and learn these tools using Fedora 20 repositories.

I am writing about my experiences in a monthly column in Open Source for You and will continue to post them on my site after a month or more of the publication.

I hope learning new tools is as effective as learning new languages to keep senility away after retirement :)

Csound and x86_64 machines

I could not understand why Sugar TamTam activities wouldn't play any sound on x86_64 even though the problem seemed resolved! It took a while to realize that the problem was with x86_64 architecture. I could get the sound to play on an i686 virtual machine.

Thanks to realtime Csound programming using Python, I could trace the fault. The issue is illustrated by the following code in csoundSession.py:

        if csnd.csoundGetSizeOfMYFLT() == 8:
            pfields = csnd.doubleArray(n)
        else:
            pfields = csnd.floatArray(n)

The csound client code in Python and C used by tamtam activities assumes a float array.

The following line in csound.spec from rpm source makes the issue pretty clear:

# Csound is really dumb about 64-bit

Rebuilding csound on x86_64 with "%define useDouble 0" even for x86_64 resolved the problem with TamTam activities.

However, it is entirely possible that this workaround may cause some other applications which use csound to fail on x86_64.

OpenStack - Virtual Machine stuck in task state 'powering-on'

Today for the second time I had to search for the solution in http://dachary.org/?p=2939. Last time, I seemed to have selected the right keywords but it took much more effort today.

I just needed to reset task_state in instances table of the nova database:

mysql -e "update instances set task_state = NULL \
           where deleted = 0 and hostname = 'master'" nova

The problem may be caused by error in nova compute starting up as I use virtual machine manager as well.

Virtual machine manager creates images in /var/lib/libvirt/images which are readable by root only.

For some reason, nova-compute looks in this directory as well and gets errors like:

Stderr: "qemu-img: Could not open '/var/lib/libvirt/images/fedora20.qcow2': Permission denied\n"

I manually change the permissions to readable by all to overcome the problem.

Running TamTam Activities on Fedora 20

While preparing an old machine to give to the children of our helper, I had to struggle a bit to get TamTam activities to run. I was using the Sugar environment on Fedora 20. The version of TamTam activities in the Fedora repository was old and did not work.

The solutions were as follows:

• The tamtamorc.cfd file in tamtam/common/Resources/ needs an additional comma in line 62:
  
  $ diff tamtamorc.csd tamtamorc.csd.orig
  62c62
  < aindex, atrans, ifreq, iphase, itable, itabdur xin
  ---
  > aindex, atrans, ifreq, iphase itable, itabdur xin
  
• On the x86_64 systems, in common/Util/Clooper, I needed to run make to make sure that aclient.so is built for the installed version of csound.

Getting started with Erlang Webmachine on Fedora 20

The problems I am encountering in learning erlang on Fedora 20 is helping me understand erlang better :)

I had to install erlang-ibrowse and erlang-meck even for starting the skeleton application. They should probably be included in the dependencies of erlang-webmachine rpm.

The template for creating the skeleton application is included in
/usr/lib64/erlang/lib/webmachine-1.10.1/priv/templates.

To create a new skeleton app:
$ cd /usr/lib64/erlang/lib/webmachine-1.10.1/priv
$ rebar create template=wmskel prefix= appid=

$ cd
$ rebar compile

The wmskel.template needed to be modified to comment out the following lines related to rebar:
%{file, "{{webmachine}}/rebar", "{{prefix}}/rebar"}.
%{chmod, 8#744, "{{prefix}}/rebar"}.

The version number in rebar.config also needed to be changed from 1.9.* to 1.10.*

However, execution of start.sh fails with the error:
 init terminating in do_boot ()

The issue was that in .erl, the following statement fails:
   ensure_started(mochiweb),

It appears that some of the dependencies of mochiweb are not implicitly started. I am not sure if I am missing something, this is a fedora specific issue or a current mochiweb version issue. A work around was to modify the start function in src/.erl adding the lines in bold:

start() ->
    ensure_started(inets),
    ensure_started(crypto),
    ensure_started(asn1),
    ensure_started(public_key),
    ensure_started(ssl),
    ensure_started(xmerl),
    ensure_started(compiler),
    ensure_started(syntax_tools),
    ensure_started(mochiweb),
    application:set_env(webmachine, webmachine_logger_module,
                        webmachine_logger),
    ensure_started(webmachine),
    application:start(webmachine_demo).

For diagnosing, modifying ensure_started function in the same file helped.

ensure_started(App) ->
    case application:start(App) of
        ok ->
            ok;
        {error, {already_started, App}} ->
            ok;
        X -> io:format("App failed ~w ~w~n",[App, X])
    end.


Now,
$ rebar compile
$ ./start.sh



http://localhost:8000  should show "Hello, new world".

It may be preferable to modify the start function in templates/src/wmskel.erl.

Getting started with Erlang Mochiweb on Fedora 20

It is simple to use this tutorial for creating a minimal Erlang Mochiweb application. Clone the git repository and run make twice.

However, trying to do the same on Fedora 20 using the packages erlang-mochiweb and erlang-rebar wasn't so simple.

While the templates are included in the /usr/lib64/erlang/lib/mochiweb-2.4.2/support directory, it is missing the Makefile. We need to execute the rebar commands instead.

Several lines need to be commented in support/templates/mochiwebapp.template. These correspond to files not included in the erlang-mochiweb rpm package -

%%{file, "../../.gitignore", "{{dest}}/.gitignore"}.
%%{file, "../../Makefile", "{{dest}}/Makefile"}.
%%{file, "../../rebar", "{{dest}}/rebar"}.
%%{chmod, 8#755, "{{dest}}/rebar"}.

The script start-dev.sh fails. Solution is to replace '\\' by '\'. The script becomes:

exec erl -pa ebin edit deps/*/ebin -boot start_sasl \
    -sname {{appid}}_dev \
    -s {{appid}} \
    -s reloader

Now,
$ cd /usr/lib64/erlang/lib/mochiweb- 2.4.2/
$ rebar create template=mochiwebapp dest= appid=
$ cd
$ rebar compile
$ ./start-dev.sh

If successful, browsing localhost:8080 shows the success page.