Welcome to this year's second issue of DistroWatch Weekly! Today we have something slightly different on the menu. While most of the time this publication caters to home users, this week's feature story is targeted at enterprise Linux system administrators as we take a first look at LucidWorks Enterprise, a Java-based, cross-platform search engine. Read on to find out how it fared in our test. In the news section we learn about several subtle signs that indicate a proximity of the Debian GNU/Linux 6.0 release before we link to the latest update from the Mageia project and to an article by a well-known distro reviewer who picks the best Linux distribution of 2010. Also in this issue, something for fans and users of genealogy software and a Question and Answers section that gives a few hints to those who think about building their own custom Linux distributions. Happy reading!
In mid-December the Lucid Imagination team released a new product called LucidWorks Enterprise. It's a solution based upon the Apache Lucene/Solr search technology. So you might be wondering - is it a search engine, a desktop search tool, a library? In the documentation for LucidWorks Enterprise (LWE) we receive the following introduction: "LucidWorks Enterprise has been designed to provide you with the search capabilities and benefits of Solr while still providing the ease of use you need to work efficiently in an environment in which data is everywhere, and you need to get a handle on it. While it does provide some great opportunities for programmers to take control and build powerful search applications using those capabilities, it's also been designed to take much of the pain out of using such a complex system."
The above is a good mission statement, but it doesn't really tell us what LWE does or why it's an attractive icing on top of the Solr cake. Fortunately the good folks over at Lucid Imagination were kind enough to provide me with a demo of their latest and greatest so I could answer those questions. The download package, which takes the form of a Java archive, is approximately 86 MB in size. Being based on Java technology, LWE is cross-platform and is supported on Linux, Mac OS X and Windows. For the purpose of testing LWE I set up a VirtualBox virtual machine running Linux Mint 9 (which is based on Ubuntu's 10.04 LTS release, appropriately named "Lucid Lynx"). The people at Lucid Imagination also provided me with their "Getting Started With LucidWorks Enterprise" document, a 41-page PDF file.
Before jumping into the technical details of installing, configuring and using LWE, I want to say that the Getting Started guide is quite good. It's a bit of a cliché that open-source documentation is sparse and/or cryptic and that is not the case here. The first two pages read suspiciously like a press release, but past that point the guide is spot on. The PDF gives a short blurb on how LWE works and then jumps into the installation requirements, the steps needed to perform the installation (either via a GUI or the command line) and provides occasional screen shots. We're up and running around page twelve and the remainder of the document covers customizing, more exotic use cases and optimizing LWE to suit your users' needs. Most of the instructions are provided in a straight forward manner and I suspect anyone who is able to install a Linux distribution will have no trouble setting up an install of LWE. The PDF also gives us further insight into what LWE is, so what is it?
In short, we can say LucidWorks Enterprise is a search engine, one which can be customized to suit your business's needs. It's flexible in that LWE will search web sites (remote or local) and index databases and file systems. Imagine your company having its own Google, if you will, where users can search for documents, web pages and database entries via a simple web interface. Further, most of the administrator functions are also provided in a clean web interface. LWE claims to be able to quickly handle millions of documents and can assist searchers by breaking down documents into categories, such as file type, author and source. We'll cover a bit more of that later, but first let's look at the install process.
Launching the Java archive kicks off the installer and shows us the system requirements (2GB of memory or higher), then asks us to accept the product's license agreement. We're then asked which components we want to install on this machine and which network ports LWE should use. I chose to install everything and kept the default ports of 8888 and 8989. We then select where we want to install the package, it will install just about anywhere, so I plunked it down into the home directory of a user I had created for this purpose. A nice side-effect of this approach is that we don't need administrator access to install or run LWE. From there the installer unpacks the required files and, optionally, launches LWE in the background. We're then given a message letting us know the service is available to be accessed. With the installation completed, we can get to work configuring LWE using our web browser. Any modern GUI browser should do the trick, I opted to use Firefox.
We connect to LWE by pointing our web browser to the server, on port 8989. In my case, this meant using the URL "http://localhost:8989". What we see at first is pretty standard search engine material, a query box and a search button. Over in the top-right corner we find a link to login. Logging in takes us to the administrator console where LWE guides us through adding a new data source. We can select the type of data source we want (local file system, website, database or Solr resource), give the source a name and choose when to index the data. As an example, I chose to index a local directory of documents and told LWE to index the directory immediately. Once the indexing begins, we're taken to a status page which tells us about current tasks in progress, statistics on recent queries and our disk space usage. At this point I went back to the initial search page. Once again we see the query box, but now with a new addition. Over on the right side of the screen is a series of links categorizing indexed data by source, by author and by file type. These are, basically, quick links and I found them quite helpful. The query box works just as it does in any other search engine -- we put in key words and the system returns relevant results with a brief preview.
Something I found interesting about the way LWE is set up is that if we select an item we want, we're then taken to a screen with more information on the item and provided a link to the item. If the item we selected is a part of a website, the link we're given takes us directly to the appropriate web page. However, by default, if the resource was indexed on a local file system LWE will not send us a copy of the file. I suspect this is for security reasons. We're told where the item is, but it is not duplicated for us. The LWE documentation explains how to change this setting so locally indexed items can be copied and sent to people performing searches.
On the topic of searches, LWE does a good job of implementing common search features. The search page includes auto-complete and spell-check. When results are found, the user is provided with a link to find similar matches. While users can search for documents based on key words and file names they are additionally able to search for documents based on file type and length. For instance, we can find PDF files which contain 40 to 50 pages. The feature I think I enjoyed the most is the e-mail alert. When a user is logged into LWE and they perform a search, let's say for "Linux", they have the option of setting an alert. When the alert is set and any new document matching the word "Linux" is found in the system, LWE can e-mail the user a notification. Fields can be set to limit the number of notifications and how often the system should check for new matches. It's a handy way to automatically keep track of documents of a particular type or subject.
The defaults LWE comes with will probably suite most organizations. I certainly found the initial configuration to be useful. But, for people who want to fine-tune the system, there are some interesting admin tools provided in the GUI. We're able to change the way LWE indexes files, changing the fields the search software examines. We can also add search synonyms, making one search word act like another. For instance, we can make sure anyone searching for "airplane" also sees results for "aeroplane". There's a reverse version of the synonym feature where we can set a list of words we want to ignore. These ignored words are called "stop words" and they usually include "a", "the" and "of", but can be extended to block out other unwanted (or overly common) terms. The administrator is able to schedule updates to the various indexes, setting a unique update interval for each data source. It's useful to be able to update a website's index every hour, or a folder once a day, depending on how dynamic and large the data source is.
Though my experience with LWE was bug-free, I did run into one quirk. Once my install was completed and I had configured my data sources, I thought it would be a good idea to change the administrator password. I didn't find any information on changing the default password in the Getting Started guide, nor did I locate the proper function in the GUI. Upon contacting Lucid Imagination, I was directed to a collection of scripts kept in the LWE directory structure. There are scripts included to help administrators gather information on their users, change settings, configure accounts and reset passwords. There was one more hurdle and that was these scripts require JSON to be installed in the LWE directory. Again, the LWE documentation is a help here and provides directions for obtaining JSON. Once I had the proper script it was a straight forward process to create new accounts and change the default password.
One might wonder how much it costs to provide an organization with their own personalized search engine. Lucid Imagination provides trials free of charge, allowing system administrators to try before they buy. I'm told that developers wanting to work with the LucidWorks Enterprise API can also get copies of the product gratis. Support contracts, for production use and for developers who need more than the free documentation provided, start at US$36,000. I find it interesting to note licensing is based on the number of servers an organization has and not on the number of users or by the number of data sources being indexed. During my trial I contacted Lucid Imagination a few times to ask questions and get advice on my configuration and I always received quick and courteous replies.
A note on system requirements -- Linux users should find that LWE will run on just about any computer that can support Java 1.6 (and higher). Though it's recommended LWE be run on a server with a few gigabytes of RAM, requirements will depend a lot on the amount of data the organization is indexing and how many users need to simultaneously make use of the search function. The virtual machine I created for LWE made use of just 1 GB of memory and featured Linux Mint running KDE. With this environment I was able to index thousands of documents in a few seconds over the network and perform parallel searches almost instantly. Obviously resources will have to scale up with an organization's requirements.
I've spent several days playing with LucidWorks Enterprise and I have to say I am impressed with what I have seen thus far. The system requirements are low, the system is flexible and fast and there are lots of example scripts for developers who wish to expand on the functionality. The documentation is well put together, the admin GUI is easy to use and the end-user interface is familiar. Aside from stepping out of the GUI to change the admin password, I found the whole experience to be smooth and friendly. LWE is built from open source components and, in my opinion, offers an excellent solution for organizations who need to keep track of large numbers of documents in a wide range of formats. If you have any interest in search technology, I recommend you give it a try.
The much anticipated release of Debian GNU/Linux 6.0 "Squeeze" did not arrive before the end of the year 2010, as some have speculated, so the wait is still on. But as Susan Linton writes in Is the Debian 6.0 "Squeeze" Release Upon Us?, there are several signs, some more tangible than others, that it won't be long now: "Release critical bugs concerning the next release have been markedly decreasing in recent weeks with persons of major concern dropping to its lowest numbers this release cycle. The total number of all bugs remains relatively high at 1555 (down from over 2,600 this time last year), the real number of bugs affecting this release is down to just 84. This number reflects all bugs minus ignored bugs, bugs in packages not in testing, and bugs whose tags indicate not in testing. This number has been very low and decreasing since the beginning of the new year. But for persons who like more tangible proof, there is little from actual developers at this moment, but there are small clues. One to surface today was mentioned on the Debian forum by a 'Sid' user. He noticed during updates that the latest base-files package changes the codename from 'Debian GNU Linux squeeze/sid' to 'Debian GNU/Linux 6.0.'"
One other interesting news item from the Debian world which is getting ready for a big release. According to Steve McIntyre, the project has recently started building hybrid, dual-architecture disk images containing packages for the two most popular processors, i386 and amd64. Furthermore, the ISO images are now transferable to USB disk drives with a simple "dd" command. Steve McIntyre writes: "We've had a wish-list bug (#551951) open against debian-cd for some time, asking for the creation of 'isohybrid' CD images for i386 and amd64. These are special in that, as well as the normal CD-based ISO9660 file system, they also contain a valid-looking DOS-style partition table. Thus, if you simply copy one of these images raw to a USB stick a normal PC BIOS will boot the image directly. This would be a neat feature, making it much easier for people to use standard Debian installer images on their USB sticks without having to follow a lot more instructions. ... We've worked together in the last few months on porting my old JTE code (which creates our jigdo images) from cdrkit to xorriso and, after a lot of testing and debugging, we now have things working fully. I've added a small amount of code in debian-cd to use the new xorriso features, and for the last couple of weeks all of the i386, amd64 and i386-amd64 multi-arch CDs and DVDs have been built as hybrids."
Mageia, the Linux distribution created by the former developers and contributors to Mandriva Linux, has published an update on the current work taking place around the project's infrastructure. Perhaps the most interesting item on the update list is information about the package building process which is about to start: "During past Wednesday's packagers' meeting, Mageia packaging tasks have been launched. First packages will be imported in the coming days by about 40 packagers. This goes together with starting the mentoring of new packagers so that every proposal of contributions we received since the beginning can turn into a positive effort. You can join the #mageia-mentoring channel on the Freenode IRC network to discuss all aspects of this subject. In the mean time, 2 representatives have been elected for the packaging team leadership. Our buildsystem, which is a corner stone of the project, is being updated/wrapped up by the sysadmin team and with the great help of Pascal Terjan. We will give you more details about the buildsystem deployment later on." According to the same article, the team is on target for the initial alpha release later this month.
Many of our readers have probably come across Dedoimedo, a neat website whose publisher, Igor Ljubuncic, has been writing reviews of free operating systems for some time. As the year 2010 came to a close, the author published a grand summary of his testing experiences which he called And the best distro of 2010 is.... The winner? Linux Mint: "You haven't seen it listed above, but it would feature under the glorious title of best all-rounder. And that would have to be Linux Mint. While it did underperform in the first quarter, the third quarter release is just splendid. It's a perfect 10 for the tenth release. Linux Mint 'Julia' has the best overall combination of ingredients. The best desktop theme and menu, the best combination of programs, the best package management. It's the most usable distribution out there, and it's just a pleasure to run. Linux Mint is not afraid to be different. While it is based on Ubuntu, it uses its own set of programs, sticking to goodies like Thunderbird, Pidgin, GIMP, and others, without following the herd mentality of the social integration and application dumbefication. Everything works out of the box, every little detail is carefully placed and designed, there's practically nothing bad you can think of. Upping this achievement is going to be really hard. But the title is well deserved." Those are some superlatives, but with Linux Mint now comfortably on the second spot in our Page Hit Ranking statistics, it's clear that the project is doing many things right.
Finally, something for those users who are interested in genealogy and in tracing their family lineage. Dale Athanasias has emailed us about the recently released Linux Genealogy CD, an Ubuntu-based live CD with pre-installed GRAMPS (an open-source genealogy program) and GraphViz (an application that draws charts and graphs in GRAMPS): "You might be interested in helping to promote GRAMPS by showing that a live CD exists and that some of the other open source genealogy developers may want to add their programs to the CD." The project's web site offers the following introduction to the Linux Genealogy CD: "The Linux Genealogy Desktop CD 6.1 includes the following pre-installed genealogy software: GRAMPS 3.2.5 as well as GraphViz, a program that draws pretty charts and graphs in GRAMPS. This Live CD is based on Ubuntu 10.10 Desktop CD. In addition to the live session, this disk also allows permanent installation of Linux and genealogical software on your computer's hard drive. This way you achieve adequate speed and the ability to save your data, and can do real work with your Linux software. Everything is similar to the live session, except that this is a permanent setup."
Building-something asks: How do I go about creating my own distribution?
DistroWatch answers: There are a few different ways to go about making a distribution, depending on whether you're looking at creating your own spin of an existing distro to share among friends, making a re-spin to share with the general public, or making your own distro from the ground up.
Let's start with the easy one, making a spin of an existing distribution to share with a handful of people. This can be handy if you enjoy using a particular distribution, but find yourself always performing the same configuration actions with every install. Here we don't need much, just a computer with a fresh install of your favourite Linux distribution and a copy of a program like remastersys. The remastersys program allows you to make an ISO image of your system, with or without your personal files. Really, it does all the work and the only thing you need to do is burn the resulting ISO file to disc. While remastersys runs on Debian and the Ubuntu family of distros, there are corresponding scripts for Mandriva, PCLinuxOS and Fedora.
If you're happy with your creation and want to share it with the world as a publicly available distribution, you'll need a few other things. You'll want a website with a lot of bandwidth, for starters. You'll also want to set up a forum and probably a bug tracker (or a web page to point people to the upstream tracker). You'll probably need some coding experience to help you fix bugs. Your project will probably need a package repository too. Most importantly you will need quite a bit of free time. If any of these items sound daunting it's probably a good idea to join an existing project, rather than branch out on your own. More on that in a moment.
In case the second approach isn't challenging enough, you can attempt your own distribution from scratch. Before doing this, I recommend spending some time working with an existing distribution, probably a small one, where you can learn the ropes and get involved in a lot of different aspects of the building, trouble-shooting and upkeep processes. Some projects which have few members, but lots going on in the community, include (among many others) Linux Mint, PC-BSD and Zenwalk. Assuming you like the experience and thrive in the distro-creating environment, you'll have picked up the skills needed to make your own project. Hopefully working on an established distribution will also show you what can be done to improve the computing experience for others.
Shane Remington has announced the release of Peppermint OS One-01042011, an Ubuntu-based distribution featuring the LXDE desktop and many integrated web-based applications: "We are proud to announce the availability of Peppermint OS One-01042011 being the latest respin of our original release. This version offers a fully updated system as of January 4, 2011 and comes with some bug fixes as well as some new features. The default kernel has been updated to 2.6.35 in order to stay more current regarding hardware support and to match the kernel in Peppermint Ice. The Hardware Abstraction Layer (HAL) has been completely removed from this version in order to help with performance and to increase application modularity. The default screenshot application has been replaced with PyShot, a simple Python/GTK+ application." Read the rest of the release announcement for more details.
Larry Short has announced the release of Puppy Linux 5.2, an independent, minimalist Linux distribution for the desktop. From the release notes: "To create Lucid Puppy 5.2 we began with the popular Lucid Puppy 5.1.1. We then upgraded, updated, and/or improved all of the main programs as well as many of the other programs in the menu and system. We have incorporated numerous improvements from the latest version of Barry Kauler's Puppy builder, Woof. We have refined operation throughout. The first thing up is a very tidy Quickset dialog to accept or change video resolution, time zone, language, locale and keyboard. Next up could be the browser installer with browser default allowing you to change which one you use as the default. And then there's Quickpet with as much good stuff as a Swiss army knife."
A new version of Superb Mini Server (SMS), a Slackware-based Linux distribution for servers, has been released: "Superb Mini Server version 1.5.5 released (Linux kernel 2.6.35.7). This release upgrades packages to Slackware 'Current' and brings the latest stable versions of several packages including PHP 5.3.5 and CUPS 1.4.6. New packages in SMS 1.5.5 are Terminator (a tool for arranging terminals which can also broadcast the same output to several terminals), lsscsi (listing SCSI devices utility) and perl-modules which split from Perl package. If you are upgrading don't forget to install new packages and especially the perl-modules package as otherwise you will have problems with Perl modules and SpamAssassin. SMS live CD now has RAID support, which means that it will automatically find, assemble and mount existing RAID arrays while booting." See the release announcement which is followed by a complete changelog.
Jeff Bilyk has announced the release of Alpine Linux 2.1.4, a community-developed operating system designed for x86 routers, firewalls, VPNs, VoIP and servers: "We are pleased to announce the Alpine Linux 2.1.4 release. This release includes an update to 2.6.35.10-based kernels. Hyper-V fixes have been applied to these kernels. Boot scripts have been updated so that if at least one network interface starts successfully, then services that depend on networking will be started. In previous releases (since Alpine 1.9.0) all network interfaces had to start successfully for this to happen. Also, BusyBox contains a fix for a long-standing bug that caused crontab files to mysteriously disappear." Read the rest of the release notes for further details and upgrade instructions.
Ron Ropp has announced the release of wattOS R3, a lightweight, Ubuntu-based distribution featuring the LXDE desktop: "wattOS R3 is released - loads of new improvements, programs, and new fun. wattOS is a lightweight Linux operating system remastered from the core Ubuntu build. It focuses on a small footprint, low power, and a simple quick interface. A brief list of updates: based upon Ubuntu 10.10; autologin works on live CD and as an option for install to hard drive; lightweight Linux running LXDE; new music player and music/video search tool (foobnix); new simple and fast photo editor (Fotoxx); updated file manager PCManFM with integrated support for network share browsing, trash, applications listed, many improvements...." Read the release announcement on the project's forums to learn more.
FIDOSlax. FIDOSlax is a Russian distribution and live CD based on Slax.
DoudouLinux. DoudouLinux is a distribution specially designed for children to make computer use as easy and pleasant as possible. DoudouLinux provides tens of applications that suit children from two years on and tries to give them an environment as easy to use as a gaming console.
This concludes this week's issue of DistroWatch Weekly. The next instalment will be published on Monday, 17 January 2011.