Apertium: Free open source language translator

Came across a very cool "free" and "open source" tool to translate text from one language to another.

It's "Apertium" => "A free/open-source machine translation platform".

https://www.apertium.org

Excerpt from the site:

Apertium is a free/open-source machine translation platform, initially aimed at related-language pairs but expanded to deal with more divergent language pairs (such as English-Catalan). The platform provides a language-independent machine translation engine tools to manage the linguistic data necessary to build a machine translation system for a given language pair and linguistic data for a growing number of language pairs.

You can try the UI at https://www.apertium.org to translate sample text from one language to another.

Wiki and documentation related to it can be found at http://wiki.apertium.org/wiki/Main_Page

It also has a variety of tools for users / translators and developers: http://wiki.apertium.org/wiki/Tools

Apertium tools for users / translators

Apertium tools for developers

The installation steps are very easy. Just follow the one suitable for your OS at: http://wiki.apertium.org/wiki/Installation

Here is a video explaining the download, install process and command line usage of it for Ubuntu: https://www.youtube.com/watch?v=vy7rWy2u_m0

Below are the steps to install and use in a fresh new Ubuntu system:

1. Download stable-release sh script and run it to add Apertium package key to Ubuntu and to update the packages list.

$ wget https://apertium.projectjj.com/apt/install-release.sh -O - | sudo bash

2. To install all the core Apertium tools run:

$ sudo apt-get install apertium-all-dev

3. Download a language pair for your conversion use. You can search for your language pair here at http://wiki.apertium.org/wiki/List_of_language_pairs. For this example case, we will be downloading spanish-english language pair for conversion:

"apertium-en-es"  => English <-> Spanish

$ sudo apt-get install apertium-en-es

4. Now finally time to test it:

The command syntax is apertium <language-of-given-text>-<language-to-be-translated>

In our case, we will be translating from Spanish to English:

$ echo 'gracias' | apertium es-en

o/p: thank you

Now we will try to convert this spanish sentence "Tengo Un Mes Estudiando Español" which in english means "I've been learning Spanish for 1 month".

$ echo 'Tengo Un Mes Estudiando Español' | apertium es-en

o/p: Have A Month Studying Spanish

Same translation from google translate results in:

I have a month studying Spanish

Seems like Apertium translation result is quite similar to google translate result. This is great, coming from a free and open source tool. Hope you find use for it in your applications :)

Using git pre-commit hook for php and js syntax check

This is a followup from my two previous posts on php and js git pre-commit syntax check where I had mentioned how to check for php and js syntax independently using pre-commit with git.



But what if we wanted to check for both php and js syntax at same time while committing. So, I used the scripts used for both and combined them. Below is the final result.

#!/bin/bash

commit_error=false

ROOT_DIR="$(pwd)/"

LIST=$(git diff-index --cached --name-only --diff-filter=ACMR HEAD)

ERRORS_BUFFER=""

for file in $LIST

do

    EXTENSION=$(echo "$file" | grep ".php$")

    if [ "$EXTENSION" != "" ]; then

        ERRORS=$(php -l "$ROOT_DIR$file" 2>&1 | grep "Parse error")

        if [ "$ERRORS" != "" ]; then

            if [ "$ERRORS_BUFFER" != "" ]; then

                ERRORS_BUFFER="$ERRORS_BUFFER\n$ERRORS"

            else

                ERRORS_BUFFER="$ERRORS"

            fi

            echo "Syntax errors found in file: $file "

        fi

        # Check for xdebug statments

        ERRORS=$(grep -nH xdebug_ "$ROOT_DIR$file" | \

                 sed -e 's/^/Found XDebug Statment : /')

        if [ "$ERRORS" != "" ]; then

            if [ "$ERRORS_BUFFER" != "" ]; then

                ERRORS_BUFFER="$ERRORS_BUFFER\n$ERRORS"

            else

                ERRORS_BUFFER="$ERRORS"

            fi

        fi

    fi

done

if [ "$ERRORS_BUFFER" != "" ]; then

    echo

    echo "Found PHP parse errors: "

    echo -e $ERRORS_BUFFER

    echo

    echo "PHP parse errors found. Fix errors and commit again."

    commit_error=true

else

    echo "No PHP parse errors found. Committed successfully."

fi

files=$(git diff --cached --name-only --diff-filter=ACM | grep "\.js$")

if [ "$files" = "" ]; then

    exit 0

fi

pass=true

JS_ERRORS_BUFFER=""

echo -e "\nValidating JavaScript:\n"

for file in ${files}; do

    result=$(jslint ${file} | grep "${file} is OK")

    if [ "$result" != "" ]; then

        echo -e "\t\033[32mJSLint Passed: ${file}\033[0m"

    else

        JS_ERRORS=$(jslint ${file})

        JS_ERRORS_BUFFER="$JS_ERRORS_BUFFER\n$JS_ERRORS"

        echo -e "\t\033[31mJSLint Failed: ${file}\033[0m"

        pass=false

    fi

done

echo -e "\nJavaScript validation complete\n"

if ! $pass; then

    echo -e "\033[41mCOMMIT FAILED:\033[0m Your commit contains files that should pass JSLint but do not. Please fix the JSLint errors and try again."

    echo -e $JS_ERRORS_BUFFER

    echo

    commit_error=true

else

    echo -e "\033[42mCOMMIT SUCCEEDED\033[0m\n"

fi

if $commit_error; then

    exit 1

fi

Store it in .git/hooks/pre-commit

Give it execute permission: $ chmod +x .git/hooks/pre-commit and it should be good to go :)

Sample output when I try to commit two php files and two js files with syntax errors:

Filenames with code content:

error.php

<?php
$array = ;

another_error.php

<?php

var_dump(');

error.js

i =

another_error.js

i = k;

$ git commit -m 'Error Commit' error.php another_error.php error.js another_error.js

Output from above error commit:

Syntax errors found in file: another_error.php 

Syntax errors found in file: error.php 

Found PHP parse errors: 

PHP Parse error: syntax error, unexpected ';' in /var/www/html/error.php on line 2

PHP Parse error: syntax error, unexpected '');' (T_ENCAPSED_AND_WHITESPACE) in /var/www/html/another_error.php on line 2

PHP parse errors found. Fix errors and commit again.

Validating JavaScript:

JSLint Failed: error.js

JSLint Failed: another_error.js

JavaScript validation complete

COMMIT FAILED: Your commit contains files that should pass JSLint but do not. Please fix the JSLint errors and try again.

error.js #1 'i' was used before it was defined. i = // Line 1, Pos 1 #2 Unexpected character '(space)'. i = // Line 1, Pos 4 #3 Unexpected '(end)'. i = // Line 1, Pos 3 #4 Stopping. (50% scanned). // Line 1, Pos 3

another_error.js #1 'i' was used before it was defined. i = k; // Line 1, Pos 1 #2 'k' was used before it was defined. i = k; // Line 1, Pos 5

Above script can be further modified to format the output better but is helpful for basic php and js syntax issues and js standard checks in code :)

Associating git hooks with js syntax check

This is a followup article from my previous post.

Here we will be using git pre-commit to check for syntax error while committing in JavaScript files.

We will need nodejs in our system. Please check https://nodejs.org/en/ for downloading and install steps.

Note: I am using ubuntu 14.04 for this setup.

Steps to add pre-commit JavaScript syntax check with git:

1. Use npm to install jslint package globally in your system.

https://www.npmjs.com/package/jslint

$ npm install -g jslint

This should globally install jslint in your system and can be referenced as below to check syntax error for any js file.

$ jslint sample.js

for (i = 0; i < 10; i++)

{

        sum += i;

}

Sample output from above command:

sample.js

 #1 'i' was used before it was defined.

    for (i = 0; i < 10; i++) // Line 1, Pos 6

 #2 'i' was used before it was defined.

    for (i = 0; i < 10; i++) // Line 1, Pos 13

 #3 'i' was used before it was defined.

    for (i = 0; i < 10; i++) // Line 1, Pos 21

 #4 Unexpected '++'.

    for (i = 0; i < 10; i++) // Line 1, Pos 22

 #5 Expected exactly one space between ')' and '{'.

    { // Line 2, Pos 1

 #6 Use spaces, not tabs.

    sum += i; // Line 3, Pos 1

 #7 Expected 'sum' at column 9, not column 2.

    sum += i; // Line 3, Pos 2

 #8 'i' was used before it was defined.

    sum += i; // Line 3, Pos 9

 #9 Expected '}' at column 5, not column 1.

    } // Line 4, Pos 1

Above info is quite informative and helpful to fix basic syntax errors, missing semi-colons, indentation etc.

Correcting the above errors to below code and running jslint again:

var i, sum = 0;

for (i = 0; i < 10; i = i + 1) {

    sum += i;

}

Output from jslint:

sample.js is OK.

Note: Please also refer rules and standards being followed by jslint for checking here. Makes a nice read. :)

2. Now that jslint is globally available, let us use it and create a pre-commit file.
I found below links which highlight the use of it in a script:

http://stackoverflow.com/questions/15703065/setup-pre-commit-hook-jshint
http://dev.venntro.com/2012/11/maintaining-consistent-javascript-with-jslint/

The only disadvantage I see in the script is that it only highlights passed and failed JavaScript files with names and does not specify the error in each.

So, I made a few changes to the script as below (highlighted in blue):

#!/bin/sh

files=$(git diff --cached --name-only --diff-filter=ACM | grep "\.js$")

if [ "$files" = "" ]; then

    exit 0

fi

pass=true

ERRORS_BUFFER=""

echo -e "\nValidating JavaScript:\n"

for file in ${files}; do

    result=$(jslint ${file} | grep "${file} is OK")

    if [ "$result" != "" ]; then

        echo -e "\t\033[32mJSLint Passed: ${file}\033[0m"

    else

        ERRORS=$(jslint ${file})

        ERRORS_BUFFER="$ERRORS_BUFFER\n$ERRORS"

        echo -e "\t\033[31mJSLint Failed: ${file}\033[0m"

        pass=false

    fi

done

echo -e "\nJavaScript validation complete\n"

if ! $pass; then

    echo -e "\033[41mCOMMIT FAILED:\033[0m Your commit contains files that should pass JSLint but do not. Please fix the JSLint errors and try again."

    echo -e $ERRORS_BUFFER

    echo 

    exit 1

else

    echo -e "\033[42mCOMMIT SUCCEEDED\033[0m\n"

fi

3. Save the above script in your git pre-commit file as below:

$ cd /project/directory/
$ vim .git/hooks/pre-commit

Paste the above content and save it. Then give execute permission to it.

$ chmod +x .git/hooks/pre-commit

And we are done and ready :)

Trying a sample case:

I have two js files named sample.js and another.js with below content:

sample.js:

var i = j;

for (key in names) {

    echo names[key];

}

another.js:

var i = []

Lets try to commit them:

$ git commit -m 'Error Commit' sample.js another.js

Below is the pre-commit check output:

Validating JavaScript:

JSLint Failed: sample.js

JSLint Failed: true.js

JavaScript validation complete

COMMIT FAILED: Your commit contains files that should pass JSLint but do not. Please fix the JSLint errors and try again.

 sample.js #1 'j' was used before it was defined. var i = j; // Line 1, Pos 9 #2 'key' was used before it was defined. for (key in names) { // Line 3, Pos 6 #3 Cannot read property 'kind' of undefined // Line 3, Pos 10

 true.js #1 Expected ';' and instead saw '(end)'. var i = [] // Line 1, Pos 11

The output can be formatted a little bit better to place each line error in a new line of its own. But this simple tool is pretty handy for checking js errors while committing :)

Getting started with Redis

As per redis home page,

"Redis is an open source (BSD licensed), in-memory data structure store, used as database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries."

Redis Installation Steps:

I am trying the below steps in an ubuntu 14.04 system. Open the terminal and issue the below commands:

1. Get the latest tar zip for redis (http://redis.io/download)

$ cd /path/to/download/directory/
$ wget http://download.redis.io/releases/redis-3.0.7.tar.gz

2. Untar it and issue below commands:

$ tar xf redis-3.0.7.tar.gz

$ cd redis-3.0.7

$ make

$ sudo make install

Getting started with Redis:

1. Run redis server in one console:

$ redis-server

or as a background process:

$ redis-server &

2. You can interact with redis now using the built-in client in console:

$ redis-cli

redis> set name Satej

OK

redis> get name

"Satej"

If you are a newbie at Redis like me, give a try at the online interactive tutorial: http://try.redis.io/. It's real fun and easy.

I learnt some basic data types and their related operations from the tutorial:
- integer
- string
- list
- set
- sorted sets
- hashes

Importance of MySQL cache

My test environment is:
Ubuntu 14.04 Trusty Tahr
MySQL Server version: 5.5.44-0ubuntu0.14.04.1 (Ubuntu)

MySQL uses sql cache to store results of queries that have been executed so that when the same query is executed again it retrieves the result data set from the cache instead of getting it again from db. So it is faster data access.

It is by default enabled in MySQL.

This is interesting since there is one question we ought to ask here whether we should use it or disable it or just leave it as it is who cares :).

Ok, moving forward today's session goals are:

1.     How useful is MySQL cache?
2.     When to use it and when not to use it?
3.     What to do if you do not want to use it?

There are some catchy areas here too like not all your queries will be stored in cache. For example, a query executed inside a stored function will not be stored in cache. Below doc link from MySQL further details out the specifics.

https://dev.mysql.com/doc/refman/5.5/en/query-cache-operation.html

Starting with our goals:

1. How useful is MySQL cache?

It is very useful indeed as once you have the result data set, if the same query is executed again, the data is now fetched from the cache and it is much faster.

2. When to use it and when not to use it?

As per MySQL doc "If a table changes, all cached queries that use the table become invalid and are removed from the cache." https://dev.mysql.com/doc/refman/5.5/en/query-cache-operation.html .

So if you have executed a query and any content in any of the tables used in the query has changed (whether related or not to the dataset for the query), all the cache entries using those tables will be flushed and executing the same query again will fetch the data from the database.

So is this bad or is fine? Depends :).

If the query you are executing is very often executed lets say 5-6 times every minute and the related tables in database are altered very often lets say at a similar rate 5-6 times a second, so we even if all the queries are stored in cache but are ultimately flushed when related table data is changed, leading to fetching once again from database and storing in cache, but which we are never able to utilize.

So in this case if the table data is too often changed then it is better not to use cache for related queries since using cache adds a certain overhead (13% as stated below) to MySQL.

https://dev.mysql.com/doc/refman/5.5/en/query-cache.html

"If all the queries you are performing are simple (such as selecting a row from a table with one row), but still differ so that the queries cannot be cached, the overhead for having the query cache active is 13%. This could be regarded as the worst case scenario. In real life, queries tend to be much more complicated, so the overhead normally is significantly lower."

Otherwise if the table is less fairly changed and the query fetching the data is more frequent then it is more efficient to use cache.

3. What to do if you do not want to use it?

As discussed above, if you do not want to use it for a query which is not able to benefit the cache storage use SQL_NO_CACHE option while executing the query
which tells MySQL not to store the result set in cache.

Eg. SELECT SQL_NO_CACHE id, name FROM customer;

https://dev.mysql.com/doc/refman/5.5/en/query-cache-in-select.html

How to convert audio files from one format to another using "Sound Converter"?

Sound Converter is an open source tool available from Ubuntu Software Center that can be used to convert audio files from one format to another.

Example usage is to generate different formats of same audio file to be used in case of HTML5 <audio> element.

Install it from Ubuntu Software Center

Add a file to be converted, from the browser dialog after clicking on "Add File"

Select the format, the audio file is to be converted to and the place to be saved and then click the "Convert" button in the main window to begin the process.