Redhat Sale Evokes Painful Memories

It has been a dozen years since I got a diskless lab created for an engineering college, with a Linux server, of course. It was not just to save money as we used the surplus to set up a multi-media lab of Macs.

The result was a depressing failure. I have consistently had problems with bosses. I can't sell an idea to even my wife much less to anyone else :) Actually, doubts make me doubt myself. I have no real idea of what the future will be! And I have been "that way since 1956"

Within a year, I was no longer handling the department. The new head added disks to the diskless workstations. The Linux server continued to work but only as a mail server. The reason was the future prospects for the students. The market is for Windows. Who needs open source engineers. And, of course, the Macs were not allowed to be used for exploring multi-media as that was not a part of the syllabus.

All this came back to me with the news of IBM buying Redhat for more money than I can imagine.

I need to see A Wonderful Life again, may be many many times :)

Frustrating Experience with BSNL 3G Data SIM - and a workaround

I took a BSNL SIM while travelling expecting that their services were more likely to be available in most states. Unfortunately, I did not test it well in Chandigarh before leaving.

I was using a Huawei 3G E8231 dongle and it just did not connect. I tried on cell phone as well but the data connection did not happen whether 2G or 3G.

I needed BSNL connection in Rajasthan where Tata Docomo did not have their own data service. Once I reached Pune, I did not need the BSNL connection but it worked using the APN bsnllive!

However, my relief was not long. It did not connect in Goa either, except for some reason for 1 day, when it worked very well for the whole day and using the APN bsnlnet as expected.

Since then again no connection. It is frustrating to contact BSNL support on their call centre, especially while roaming.

I sent emails to the addresses mentioned on BSNL site. The messages did not bounce; however, I got no feedback at all. Curiously, the email addresses were on gmail.com :)

I got a call yesterday (may be because I had visited the BSNL office in Mapusa who said that they will contact Chandigarh for help). I was told that I should try now as the problem is resolved.

However, no change in status and I cannot call back the person who had called :(

Finally, I got a person on 1503 of the Maharashtra circle who gave me the help number for Punjab. The person from the Punjab circle wasn't helpful at all and probably disconnected in the middle. I tried once again and it got disconnected again. No point in making any more effort.

I am bewildered by why did it work on two days and not on other days. Since it worked twice, it does not seem like a compatibility problem with my devices. Oh, well, fortunately, there are many more options now, which work AND who really do try to help!

I will dump the sim. A lesson learnt for a thousand rupees - a mere US$ 15 :)

Update 7th Feb 2015:

There had to be a reason why the BSNL connection worked twice. I kept thinking about the sequence of operations which might have been the reason.

I think I now know.

The default APN for bsnl is 'bsnlnet'. It did not work.
I changed the profile to 'bsnllive'.
Waited for a while.
State is now connected. Some data is sent but nothing is received.
I changed the profile back to 'bsnlnet'.
State remains connected and the 3G data connection works :)

The above steps have worked on two days now.
Why? That remains a mystery!

OpenStack Nova Compute Cannot Launch Instance after Upgrade to Fedora 21

I updated my system to Fedora 21 beta and then found that I could not use OpenStack. Every effort to launch an instance failed:

libvirtError: operation failed: filter 'nova-no-nd-reflection' already exists with uuid 728f13cf-f1a9-4e0e-add7-357b93d052ee

Removing the openstack packages, clearing the data and reinstalling openstack did not help.

The problem seems to be an incompatibility  between nova-compute and the libvirt libraries.

Upgrading the libvirt libraries to rawhide did not help.

Finally, I used rpm to downgrade libvirt using downloaded packages from the fedora 20 repository, ignoring the dependency problems.

$ sudo rpm -U --oldpackage --nodeps libvirt*1.1.3*rpm

All seems well for now.

Openstack Nova Compute fails to start after updating to 2014.1.3-2.fc21

OpenStack nova compute service failed to start after the update to th e openstack-icehouse version 2014.1.3-2.fc21  - the reason appeared to be:

"Failed to add interface: can't add lo to bridge br100: Invalid argument"

This bug report, though marked as invalid, helped, The problem was a change in the file nova/network/linux_net.py.

The following exception had been in the wrong place earlier and was being ignored:

if (err and err != "device %s is already a member of a bridge; "
              "can't enslave it to bridge %s.\n" % (interface, bridge)):
       msg = _('Failed to add interface: %s') % err
       raise exception.NovaException(msg)

The interface for the flat network is set to 'lo' even if the nova.conf file does not define it. network['bridge_interface'] seems to be set to 'lo' in the following line:

iface = CONF.flat_interface or network['bridge_interface']

One workaround was to change the following line in ensure_bridge method:
<         if interface:
---
>         if interface and interface != 'lo':

Subsequently, I came across another workaround in a bug report by creating a dummy interface.

The virtual machines were stuck in 'powering on' state after the update. This time solving it was easy!

Experimenting with open source big data applications on a personal desktop

I decided to learn OpenStack, Hadoop, Elasticsearch and related open source software products using my quad core desktop. I had to raise the RAM to 16GB and add a 1 terabyte disk.

However, after the hardware upgrade, it has been remarkably simple to experiment and learn these tools using Fedora 20 repositories.

I am writing about my experiences in a monthly column in Open Source for You and will continue to post them on my site after a month or more of the publication.

I hope learning new tools is as effective as learning new languages to keep senility away after retirement :)

Csound and x86_64 machines

I could not understand why Sugar TamTam activities wouldn't play any sound on x86_64 even though the problem seemed resolved! It took a while to realize that the problem was with x86_64 architecture. I could get the sound to play on an i686 virtual machine.

Thanks to realtime Csound programming using Python, I could trace the fault. The issue is illustrated by the following code in csoundSession.py:

        if csnd.csoundGetSizeOfMYFLT() == 8:
            pfields = csnd.doubleArray(n)
        else:
            pfields = csnd.floatArray(n)

The csound client code in Python and C used by tamtam activities assumes a float array.

The following line in csound.spec from rpm source makes the issue pretty clear:

# Csound is really dumb about 64-bit

Rebuilding csound on x86_64 with "%define useDouble 0" even for x86_64 resolved the problem with TamTam activities.

However, it is entirely possible that this workaround may cause some other applications which use csound to fail on x86_64.

OpenStack - Virtual Machine stuck in task state 'powering-on'

Today for the second time I had to search for the solution in http://dachary.org/?p=2939. Last time, I seemed to have selected the right keywords but it took much more effort today.

I just needed to reset task_state in instances table of the nova database:

mysql -e "update instances set task_state = NULL \
           where deleted = 0 and hostname = 'master'" nova

The problem may be caused by error in nova compute starting up as I use virtual machine manager as well.

Virtual machine manager creates images in /var/lib/libvirt/images which are readable by root only.

For some reason, nova-compute looks in this directory as well and gets errors like:

Stderr: "qemu-img: Could not open '/var/lib/libvirt/images/fedora20.qcow2': Permission denied\n"

I manually change the permissions to readable by all to overcome the problem.

Running TamTam Activities on Fedora 20

While preparing an old machine to give to the children of our helper, I had to struggle a bit to get TamTam activities to run. I was using the Sugar environment on Fedora 20. The version of TamTam activities in the Fedora repository was old and did not work.

The solutions were as follows:

• The tamtamorc.cfd file in tamtam/common/Resources/ needs an additional comma in line 62:
  
  $ diff tamtamorc.csd tamtamorc.csd.orig
  62c62
  < aindex, atrans, ifreq, iphase, itable, itabdur xin
  ---
  > aindex, atrans, ifreq, iphase itable, itabdur xin
  
• On the x86_64 systems, in common/Util/Clooper, I needed to run make to make sure that aclient.so is built for the installed version of csound.

Getting started with Erlang Webmachine on Fedora 20

The problems I am encountering in learning erlang on Fedora 20 is helping me understand erlang better :)

I had to install erlang-ibrowse and erlang-meck even for starting the skeleton application. They should probably be included in the dependencies of erlang-webmachine rpm.

The template for creating the skeleton application is included in
/usr/lib64/erlang/lib/webmachine-1.10.1/priv/templates.

To create a new skeleton app:
$ cd /usr/lib64/erlang/lib/webmachine-1.10.1/priv
$ rebar create template=wmskel prefix= appid=

$ cd
$ rebar compile

The wmskel.template needed to be modified to comment out the following lines related to rebar:
%{file, "{{webmachine}}/rebar", "{{prefix}}/rebar"}.
%{chmod, 8#744, "{{prefix}}/rebar"}.

The version number in rebar.config also needed to be changed from 1.9.* to 1.10.*

However, execution of start.sh fails with the error:
 init terminating in do_boot ()

The issue was that in .erl, the following statement fails:
   ensure_started(mochiweb),

It appears that some of the dependencies of mochiweb are not implicitly started. I am not sure if I am missing something, this is a fedora specific issue or a current mochiweb version issue. A work around was to modify the start function in src/.erl adding the lines in bold:

start() ->
    ensure_started(inets),
    ensure_started(crypto),
    ensure_started(asn1),
    ensure_started(public_key),
    ensure_started(ssl),
    ensure_started(xmerl),
    ensure_started(compiler),
    ensure_started(syntax_tools),
    ensure_started(mochiweb),
    application:set_env(webmachine, webmachine_logger_module,
                        webmachine_logger),
    ensure_started(webmachine),
    application:start(webmachine_demo).

For diagnosing, modifying ensure_started function in the same file helped.

ensure_started(App) ->
    case application:start(App) of
        ok ->
            ok;
        {error, {already_started, App}} ->
            ok;
        X -> io:format("App failed ~w ~w~n",[App, X])
    end.


Now,
$ rebar compile
$ ./start.sh



http://localhost:8000  should show "Hello, new world".

It may be preferable to modify the start function in templates/src/wmskel.erl.

Getting started with Erlang Mochiweb on Fedora 20

It is simple to use this tutorial for creating a minimal Erlang Mochiweb application. Clone the git repository and run make twice.

However, trying to do the same on Fedora 20 using the packages erlang-mochiweb and erlang-rebar wasn't so simple.

While the templates are included in the /usr/lib64/erlang/lib/mochiweb-2.4.2/support directory, it is missing the Makefile. We need to execute the rebar commands instead.

Several lines need to be commented in support/templates/mochiwebapp.template. These correspond to files not included in the erlang-mochiweb rpm package -

%%{file, "../../.gitignore", "{{dest}}/.gitignore"}.
%%{file, "../../Makefile", "{{dest}}/Makefile"}.
%%{file, "../../rebar", "{{dest}}/rebar"}.
%%{chmod, 8#755, "{{dest}}/rebar"}.

The script start-dev.sh fails. Solution is to replace '\\' by '\'. The script becomes:

exec erl -pa ebin edit deps/*/ebin -boot start_sasl \
    -sname {{appid}}_dev \
    -s {{appid}} \
    -s reloader

Now,
$ cd /usr/lib64/erlang/lib/mochiweb- 2.4.2/
$ rebar create template=mochiwebapp dest= appid=
$ cd
$ rebar compile
$ ./start-dev.sh

If successful, browsing localhost:8080 shows the success page.

Relying more on Desktop Client for Emails

After switching to Akregator from Google Reader and learning to live with a little inconvenience, I was struck by a comment about the danger of email accounts being broken into by the criminal elements.

I now have setup an imap account and am using Kmail. Contents of any folder/label which need not available for easy access online, I plan to move to a local folder. I will miss out on the power of Google search, but I think, along with consuming less energy, learning to live with some technological inconveniences is the preferred/imperative option.

Update - Came across an interesting news of excessive reliance of technology -  When technology fails a news anchor, there are no words.

Life after Google Reader - Akregator

Web based services are convenient. It is far too easy to get used to them. I tend to look at my rss feeds from at least two systems. So, it seemed that I will use one of the online services which are trying to replace Google Reader and make the transition as painless as possible.

In view of the recent disclosures about Prism, opting for a little inconvenience is not a bad idea.

I am now quite comfortable with Akregator. One advantage is that I can see and access the entries I have already read as well. The responsiveness is very good.

To switch between machines, I take a tar backup of  .kde/share/apps/akregator from one machine and restore it on the other. Since it is a bit of a nuisance, I am organizing my net activities to normally use one system only for reading rss feeds and switch rarely.

It is not a very serious constraint and probable saves me distractions.

The mail services of our ISP are very unreliable otherwise I might have seriously considered at least partially moving from gmail.

As long as software read the content, I was unaffected; hence, no impact of Microsoft's criticisms of Google. This is probably more common  than just  for me as stories about Eliza illustrate -

Another story tells of the author's secretary using Eliza when he entered the room. The secretary asked her boss to leave until she had finished her conversation.

The author also considered putting in a module that recorded peoples conversations. This was greeted with accusations that Weizenbaum was spying on their secrets and innermost thoughts.

So, humans reading our content is another issue! 

Robots, however, guide and save humanity :)

Inactive logical volume after upgrading to Fedora 19

I upgraded my netbook to Fedora 19 beta version using yum. The process was smooth and went through without a hitch.

The problem came up on rebooting. Systemd waited and finally failed to mount home, which was a logical volume. This seemed strange as root and swap were also on the same volume group and were being used.

Finally, lvscan showed that the home logical volume was inactive. The command

lvchange --activate y 

worked and it was a relief to realize that the home logical partition was perfectly fine.

Unfortunately, rebooting the system showed the same problem. The cause of the problem was that all the lvm2 services were disabled.

Enabling lvm2-monitor.service resolved the issue :)

NFS root unexpectedly squashed

While experimenting with setting up ltsp on Fedora 18, I came across a strange NFS behaviour which took quite some time to iron out.

I was already exporting /opt as a read only file system; however, with root squashed. LTSP added another export for /opt/ltsp with no_root_squash option. 

/opt *(ro,sync,no_subtree_check)
# export for LTSP version 5
/opt/ltsp   *(ro,no_root_squash,async,no_subtree_check)

Everything seemed to be sort of working with a few strange effects, e.g. I could not use passwords and some ltsp scripts were failing.
The problem was that root seemed to be still squashed:

[anil@localhost ~]$ sudo mount amd:/opt/ltsp/x86_64 /mnt
[anil@localhost ~]$ cd /mnt/home
[anil@localhost home]$ ls -l
total 8
drwxrwx--- 2 anil  anil  4096 Apr  6 13:04 anil
drwx------ 2 guest guest 4096 Apr 10 12:35 guest
[anil@localhost home]$ sudo su
[root@localhost home]# cd anil
bash: cd: anil: Permission denied

A workaround was to use fsid=0 although whatever documentation I came across seemed to suggest that it was no longer needed:

/opt/ltsp   *(ro,fsid=0,no_root_squash,sync,no_subtree_check)

Unfortunately, I had forgotten that /opt was also being exported and noticed it only after finding the work-around and getting ltsp to work on Fedora 18.

Arch Linux installation using ipxe and a Fedora server

Arch Linux is great way to learn, e.g. installing Arch Linux booting over the network. As is usual with Arch Linux, the documentation for the process is excellent. Surprisingly, a mirror in India gave an excellent speed and low latency so even the first stage went well.

Since I was experimenting with ipxe, I decided to install Arch Linux using the iso and ipxe. Using dnsmasq was no problem. However, since I use Fedora and the default is dhcpd, I wanted to make it work with dhcpd. The problem  was finding the equivalent options of dhcp-option-force.

The issue was that the client would try to search for cfg file as per its rules and ignore the specification in the dhcpd configuration file. The solution was available on the syslinux wiki in response to

Can I send information to PXELINUX via special options in the DHCP response?

It is much harder syntax than the dnsmasq's dhcp-option-force but worked.

Install an OS from a DVD iso without booting

While experimenting with LTSP on Fedora18, I realized that I would have to have a supported environment in order to be able to understand the failure on Fedora. I had an iso file of Scientific Linux 6.3 and decided to use that.

Since ltsp-build-client installs an OS in a chroot environment while the system is operational, I decided to see if I could use a LTSP script to install the Scientific Linux on a free partition.

A python program, chroot-creator, seemed a likely prospect.

1. Commented a few lines in chroot-creator meant specifically for LTSP clients. 
2. Chose a suitable kickstart file 
3. Loop mounted the iso image and used it as the yum repository
4. Started installation on on the free partition which failed.

Installation was not possible as the partition needed to be mounted and the chroot-creator installs the OS in a directory which should not be existing. So, the installation was successfully done in a directory 'Install' under the mount point.

After the installation:

1. In Install:
   # mv * ..
2. Modify fstab file for Scientific Linux
3. Run grub2-mkconfig. It should find and include the Scientific Linux partition.
4. Chroot to the SL partition
5. Change password for root. Create additional users if needed.

Scientific Linux should now be a usable boot option.

This process should work with any distribution which is compatible with yum on Fedora.

Note: the experimental version of LTSP scripts now uses mock and chroot-creator program is not included.

LTSP On Fedora 18 using client created on Ubuntu 12.10

Exploration of LTSP turned out to be a very interesting challenge. My motivation was to write a monthly column for Linux For You.

My primary system runs Fedora 18. I thought that I would be able to get the scripts to run on Fedora 18 in spite of the explicit caution that Fedora 15 onward is not supported. I haven't succeeded so far but I did succeed in getting an client image created on Ubuntu 12.10 to run on Fedora 18.

I installed the server scripts from the repository for experimental version of LTSP 5 for Fedora 18. This ensured that that the ldminfod version was compatible with the ldm version on Ubuntu 12.10 client.

Ubuntu uses nbd by default. Unionfs root was not working with NFS root on Ubuntu as well. So, nbd version 3.1.1 needed to be installed on Fedora. The version in Fedora repository is 2.9.20.

LTSP scripts run nbd server using xinetd while Ubuntu 12.10 runs nbd-server as a system service. Since the xinetd option did not work, nbd-server was started using a system service script. The key extract from nbd-server script is:

DAEMON="/usr/bin/nbd-server"
test -x $DAEMON || exit 0
case "$1" in
    start)
    daemon $DAEMON 10809 -C /etc/nbd-server/config

The port has to be 10809 to be compatible with the -N option used by ndb-client on Ubuntu client.

/opt/ltsp/amd64, /etc/nbd-server and  /var/lib/tftpboot/ltsp/amd64 directories was copied from Ubuntu to Fedora. The following were the final steps:

1. Run ltsp-update-sshkeys
2. Run ltsp-update-kernels
3. Run ltsp-update-image amd64 -f
4. Change the dhcpd configuration files to use the amd64 directory in tftpboot instead of i386. I modified the /etc/dhcp/dhcpd.conf file and did not use the ltsp-dhcpd script.

If all has gone well, you will get a LDM login screen with Ubuntu theme. The sessions available will be the ones on Fedora 18 and you should be able to login to a session using your Fedora 18 credentials!

Booting Porteus over the LAN from Fedora 18 server

Continuing my experiments with diskless options, I was going through RAM resident distributions on Wikipedia. Porteus appealed to me as it was able to run KDE desktop from RAM. The newly released version is packaged to boot off the network from a Porteus system with no additional effort.

I wanted to boot from my existing Fedora 18 system and I wanted to avoid using tftp. The recipe I used is as follows:

In order to make sure that both tftp and html could be used:

$ sudo ln -s /var/lib/tftpboot /var/www/html/tftpboot

Obviously, httpd and dhcpd are enabled. There is no need to enable tftp in /etc/xinetd.d/tftp. The following steps were needed in the tftpboot directory:

$ sudo mount -o loop Downloads/Porteus-v2.0-x86_64.iso /mnt
$ sudo mkdir /var/lib/tftpboot/porteus
$ sudo cp -r /mnt/boot/syslinux/* /var/lib/tftpboot/porteus

Now, we need to edit /etc/dhcp/dhcpd.conf. It's content (for my network) will be:

# dhcpd.conf
#
# configuration for pxe from http://docs.fedoraproject.org/en-US/Fedora/17/html/Installation_Guide/s1-netboot-pxe-config.html
#
  option space pxelinux;
  option domain-name-servers 192.168.1.1, 8.8.8.8;
  option pxelinux.magic code 208 = string;
  option pxelinux.configfile code 209 = text;
  option pxelinux.pathprefix code 210  = text;
  option pxelinux.reboottime code 211 = unsigned integer 32;

  subnet 192.168.2.0 netmask 255.255.255.0 {
          option routers 192.168.2.1;
          range 192.168.2.200 192.168.2.210;

          class "pxeclients" {
                  match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
                  next-server 192.168.2.1;
                  option pxelinux.pathprefix "http://192.168.2.1/tftpboot/porteus/";
                  filename "http://192.168.2.1/tftpboot/porteus/pxelinux.0";
          }
  }

 Finally, we need to copy the base os files into the http root, i.e.

$ sudo cp /mnt/porteus/base/* /var/www/html

We need to disable the welcome screen for Porteus to successfully boot from the apache server. So, comment all lines in
/etc/httpd/conf.d/welcome.conf.

Now, restart the httpd and dhcpd servers.

On the client machine, I am using grub2. The client is configured as follows:

1. Download ipxe.iso
2. Extract ipxe.krn from the iso and copy it in /boot of the client.
3. Edit /etc/grub.d/40_custom and add the following lines:
   menuentry "ipxe"{
         linux16 /boot/ipxe.krn
   }
4. Create a new grub.cfg
   Fedora:  /boot/grub2/grub.cfg using grub2-mkconfig
   Ubuntu: /boot/grub/grub.cfg using grub-mkconfig

Reboot the client. Choosing the ipxe option should bring up the Porteus boot menu and it should boot into kde or lxde depending upon the option selected.

Booting over network - Use ipxe and forget etherboot/gpxe

I was playing around with diskless options and decided to see if I could boot Puppy Linux over the network. As usual, it was easy to find a recipe on the net. I got the gpxe boot linux kernel image for forcedeth driver.

The surprise was the terrible performance I got on a 100mbps network. I did not wait for it to finish booting! On a 1GB network, the performance was 'merely' bad. I did not expect tftp to be so bad!

Searching the net, I realized that gpxe allows html as the protocol instead. There was no change if I used pxelinux.0 and it was even worse if I used gpxelinux.0! I tried pxelinux.0 and gpxelinux.0 from version 5.01 of Syslinux with no improvement. However, the performance was better on my lenovo netbook which needed the r8169 driver. The performance using html protocol was still not any better.

The search for an answer led to the ipxe project. Mercifully, the ipxe.krn from the iso file worked for both the cards. Now, the performance over html was significantly better. It took half the time to boot Puppy Linux using html protocol.

Upgrading to Fedora 18 using Yum - Nouveau causes system freeze?

The documentation for upgrading fedora using yum seems to be much better than before. A very nice script which automates the instructions was terrific. The upgrades of my desktop and the netbook were very smooth. After the upgrade, I installed the Cinnamon desktop as well. It is very nice though I will stick to KDE as I am now comfortable with it.

There was, however, a surprise on the desktop. On booting into Fedora 18, the display vanished and the system hung. The system came up in muti-user mode but not in graphical interface. Gdm was the problem. I switched to kdm. The login screen came and was fine in KDE. However, the system again froze when I tried Cinnamon or Gnome desktops. LXDE worked but the system froze as soon as I started Firefox!

KDE desktop was working with desktop effects disabled. If I enabled them, the desktop froze.

The desktop worked fine with the nomodeset option for booting the kernel. However, it was using the vesa driver. So, the problem is with nouveau driver on "GeForce 6150SE nForce 430" hardware.

Another curious result. The system is working fine with the kernel 3.6.11-1.fc17.x86_64 from fedora 17, which has made it possible to continue with Fedora 18 on the desktop.

Need to reconfirm the problems I faced on the desktop as I did not find a bug report with similar issues.

On Lenovo S10-3, I have so far not faced any issues related to the upgrade.