Rebayct

[This is a post related with a piece of software I released some time ago]

ReBayCT ('Redes Bayesianas para Clasificación en Tesauros', literally in Spanish "Bayesian networks for classification from a Thesaurus") is a console-based tool for performing experiments in Thesaurus-based indexing, that is to say, Text Categorization over the set of descriptors of a thesaurus. For more information in this problem, see [1]. It's written in Java (JDK 5.0 or higher required). The code of project is located here and it is free software (see below).

There are several classifiers implemented in this software. Two baseline (VSM and hierarchical VSM) and one algorithm based in Bayesian networks with versions for unsupervised classification and also supervised. If you use them, please consider citing [2] and [3].

[1] L. M. de Campos, J. M. Fernández-Luna, J. F. Huete, A. E. Romero, Thesaurus Based Automatic Indexing, book chapter in Handbook of Research on Text and Web Mining Technologies. Ed. Idea Group, Inc. USA, 2009, ISBN: 978-1-59904-990-8. Available online at http://www.cs.rhul.ac.uk/~aeromero/pdf/thesaurus.pdf.

[2] L. M. de Campos, A. E. Romero, Bayesian Network Models for Hierarchical Text Classification from a Thesaurus, Int. J. Approx. Reasoning 50(7): 932-944 (2009). Available online at http://www.cs.rhul.ac.uk/~aeromero/pdf/ijar09-thesaurus.pdf.

[3] L. M. de Campos, J. M. Fernández-Luna, J. F. Huete, A. E. Romero, Automatic Indexing from a Thesaurus Using Bayesian Networks: Application to the Classification of Parliamentary Initiatives. ECSQARU 2007: 865-877. In: Lecture Notes in Computer Science 4724 Springer 2007, ISBN 978-3-540-75255-4. Available online at http://www.cs.rhul.ac.uk/~aeromero/pdf/lncs07-ecsqaru-thesaurus.pdf.

The license of the software package is GNU GPL v3. Please check http://www.gnu.org/licenses/gpl.html for more details.

Note (a) to possible users: I am not maintaining this software (except for small bugs) and I'm not working in this research topic now. So, don't wait for a new "major release", because it's never going to come out. If you have any doubts about how to extend or use it, please ask writing a comment to this post or to my gmail account (alfonsoeromero). I'll be glad to answer it and helping with your project. I must recall that derivative works should also be free software, as specified by the GPL license (it should have a compatible license).

Note (b) to possible users: to run this software you need a collection and the EUROVOC (or other) thesaurus in XML. I cannot distribute the EUROVOC, so please try to get a copy yourself (in XML). The dataset I used for experimentation in [2] is not entirely public (parliamentary initiatives of the Parliament of Andalusia), and I prefer to have some "control" about it, due to the fact that I don't have the real ownership of the data (and it's not very clear whether I should be able to distribute it), although it could be obtained by parsing public documents occuring in the Parliament of Andalusia webpage. Anyway, if you need the set of documents, please ask them to me.

Liblinear is amazingly fast!

I decided to give up using LibSVM for the linear case because it was not optimized for that. Then, I had a look at liblinear, developed on the same team than LivSVM. Liblinear is recommended for document classification because it removes lots of unuseful operations for the linear case. It has also a (very recent) port to Java, which is located here.

Now, working with 50 categories and 0.5GB of data takes only less than 10 seconds on a Core 2 Duo 2GHz laptop. Those timings are impressive! The interface for this library is very similar to the LibSVM one, so it is very easy to migrate from one library to other. Of course, if you made a good design, all you have to do is to update/change your corresponding facade class.

BTW: Happy new year!

Novice problems with LibSVM

If you are dealing with LibSVM, you mus remember the following:

• When building sparse vectors using datatype svm_node, be careful with allocating keys in ascending order. This is clearly specified in the documentation, but sometimes we are too lazy to read it before.
• By default, the outputs of LibSVM, when doing classification are one of {-1,1}. So, do not wait to get real outputs (for instance, distance to the hyperplane), unless you hack the code yourself. If you are doing text categorization, this is good to measure (macro/micro) F1, but not to get a good accuracy.
• You must first preprocess your feature vectors! Joachims proposes using a tf * idf, followed by a L2 normalization (classical Euclidean norm). This is valid for text classification, translating every coordinate value to the interval [0,1]. Other normalization schemes are valid for "classic" classification problems like iris and so (in those cases, the different atributes are scaled independently to [0,1]).
• There is a nasty bug (lack of feature?) in the Java version, at the method "svm_save_model", that makes very slow that procedure, because the output is not buffered. To solve it, find this line:
  DataOutputStream fp = new DataOutputStream(new FileOutputStream(model_file_name));
  And change it by the following:
  DataOutputStream fp = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(model_file_name)));

Using a SVM library for text categorization

In text categorization, one of my research fields, is impossible to ignore the great power of Support Vector Machines (SVM). Almost everyone agrees that, even in its more primitive form (Linear SVM), they are the killer algorithm to do this task (the one that gets more accuracy). And of course, that implies that any new presented approach to solve this problem should be tested against SVMs.

But, ¿what (open source/free) software packages are available for Support Vector Machines? In fact, the list is very reduced, being the two most popular ones the following:

• SVM^light, a C implementation written by Thorsten Joachims.
• LibSVM, a C (and Java) implementation written by Chih-Jen Lin.

Both of them are suitable for tasks of text categorization, as they are lightweight implementations, and relatively fasts. Besides, they both include Platt's SMO algorithm in order to make training procedure faster. So, ¿which one can be chosen?

I have tested both of them. In my most recent paper, we have used SVM^light to make a comparison against a Bayesian Network model to classify in a thesaurus environment (and of course, we beat linear SVMs!). That package is amazingly fast, not only due to the language it is written in (C), but due to the great job of Joachims in doing heuristics and other tricks.

In this moment, I am using LibSVM in my Java environment for text categorization (which I expect to release soon as free software), using directly the Java implementation. Althought it is written in a very "C-style" (arrays instead of containers, static methods, no exceptions,...), it is not so bad at speed (obviously it is several times slower than SVM^light, but it is Java, avoiding linking with a non portable library, and keeping the entire system in one language).

From the point of view of software licenses, LibSVM is released under the modified BSD license (a GPL compatible license). This is good, because it allows yo to use this package, even in a non free software environment (I must admit the last point is not really so good). SVM^light , on the other hand, is not free software. The license note claims that:

The program is free for scientific use. Please contact me, if you are planning to use the software for commercial purposes. The software must not be further distributed without prior permission of the author.

This is an important fact for me, and that is why I prefer LibSVM. I must admit that Joachims' work is impressive, and SVM^light is probably a faster and more complete environment, but I can cope with the lack of functionality and speed of LibSVM, because it is free software.

What do you think about this?