Pragmatic Development Notes

Software development stuff

Make Ruby Tests Greener

The Ruby Greener Test Challenge is here. It is attempt to make Ruby on Windows better and more stable and direct goal of this challenge is to fix all tests and to reach 0F0E (0 failures, 0 errors) state. If you are Ruby developer on Windows you should consider taking a role in the challenge.

If you decide to try to fix some failing tests or existing errors it is easy to set up complete environment with the help of Ruby Challenge Pack. I will not write about details here since you have all you need in the initial challenge post given in the link on top of the article. Instead I will give you few suggestions about participating in the challenge.

You should be aware of three things when you run tests:

  1. Set up console code page to 1252 or 65001. Some tests may fail due to the console code page, so before running tests you should execute chcp 1252 or chcp 65001.
  2. Option -j2 may lead to errors. In case you want to run tests faster you can use make test-al TESTS="-qv -j2" but be aware that this can also lead to fals failures or errors.
  3. Anti-virus software. If you see Errno::EACCES: Permission denied error in test, most likely the cause is anti-virus which locked the file during tests.

With these things on your mind you can proceed to fixing. Here is simple workflow you can use. The very first thing you should do is to clone Ruby source from GitHub:

1
git clone git://github.com/ruby/ruby.git

After deciding what you want to fix you should make new local branch in which you work:

1
git checkout -b fix-win32ole-fso-encoding-misuse

After you find a source of failures or errors you will make fixes and commit changes to your local branch. I would suggest making small commits. If commits contain too many changes, patches are harder to apply and are more likely to cause conflicts. Finally after finishing all fixes you just have to make a patch (es).

1
git checkout -b fix-win32ole-fso-encoding-misuse

Adjust last number in above command to the number of commits you made on your local branch. This will create one or more patch files (one file per commit). In the case of already submitted fix single file was created: 0001-FileSystemObject-does-not-support-UTF-8.patch. At the end open a ticket, write small description, attach your patch file (s) and assign ticket to Luis Lavena.

Exefy ‘Em All

Recently Luis Lavena started thread Idea: executable stubs to replace batch files on RubyInstaller mailing list. In short, the idea is to use command line applications – executable stubs – instead of batch files in RubyInstaller Ruby versions. Using command line applications instead of batch files has several benefits. First one, maybe not so important, is to avoid annoying

Terminate batch job (Y/N)?

question when execution is interrupted with Ctrl-C key combination. The second is to get meaningful list of processes in the system. With batch files we can only see bunch of ruby.exe processes in the list. This gives us possibility to define firewall rules for these applications which will not be applied globally for all Ruby scripts. Finally, installing Ruby applications as services, with the help of some service wrapper, is usually easier if we use executable file. These are reasons why new gem-exefy gem was made.

Gem-exefy Internals

Gem-exefy mimics behavior of batch files installed by RubyGems and to see how it works we must know how existing batch files work. First step is to install gem that has executable value defined in gem specification. Example of such gem is Bundler. After installing it on Windows, RubyGems will create bundle.bat file in <path_to_ruby_installation>/bin folder. Content of that file is:

1
2
3
4
5
6
@ECHO OFF
IF NOT "%~f0" == "~f0" GOTO :WinNT
@"ruby.exe" "c:/path/to/ruby/installation/bin/bundle" %1 %2 %3 %4 %5 %6 %7 %8 %9
GOTO :EOF
:WinNT
@"ruby.exe" "%~dpn0" %*

Without digging too much into all details we can see that batch file starts ruby.exe passing it full path to Ruby script (bundle, in this case) using all arguments passed to batch file as arguments of Ruby script. So all we have to do is to make application which will be able to execute Ruby script and will accept arguments passed in the command line. Sounds familiar, isn’t it? Exactly! We already have ruby.exe and in Ruby code we can find almost everything we need. Here is a slightly simplified version of Ruby’s main.c file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#include "ruby.h"

#ifdef HAVE_LOCALE_H
#include <locale.h>
#endif

int
main(int argc, char **argv)
{
#ifdef HAVE_LOCALE_H
  setlocale(LC_CTYPE, "");
#endif

  ruby_sysinit(&argc, &argv);
  {
    RUBY_INIT_STACK;
    ruby_init();
    return ruby_run_node(ruby_options(argc, argv));
  }
}

If we build application from this source we will get the same application as ruby.exe. This means if we want to execute some Ruby script we will have to pass path to it as a first argument with optional arguments following it. But our goal is not to invoke application with the path to Ruby script. Instead we want to invoke predefined script. In order to achieve that, we obviously have to alter the list of arguments (argv) and to insert path to target Ruby script. But there is a catch (I spent almost whole day to figure it out). We must change the list of arguments after the call

1
ruby_sysinit(&argc, &argv);

Ruby performs system dependent arguments initialization in the above method and the list of arguments will be reverted back if we change it before this initialization. Of course there are some additional details that we must take care of, but you can figure them out directly from “the source”:https://github.com/bosko/gem-exefy/blob/master/templates/gem_exe.c. After we change the list of arguments, Ruby will execute script we passed it and that’s all the magic gem-exefy does.

Let’s Exefy

We are now ready to exefy existing and new gems on our RubyInstaller version. gem-exefy is made as RubyGems plugin. After installing it with:

1
gem install gem-exefy

new gem command will be available – exefy. This command is used for replacing batch files for single or all installed gems. Replacing batch files with executable stubs for single gem is performed by passing name of the targeted gem to the exefy command.

1
gem exefy bundler

Exefying all installed gems is simple – just pass --all to exefy command

1
gem exefy --all

If you are not satisfied and still want to use batch files – don’t worry. You can always revert old batch files for single or all gems with --revert argument

1
2
gem exefy bundler --revert
gem exefy --all --revert

After gem-exefy is installed it will, by default, install executable stubs instead of batch files for all gems installed after it.

It is important to mention that gem-exefy will not replace batch files for commands installed with RubyInstaller (irb, rake…) and are part of Ruby core. Will support for converting these batch files be implemented is yet to be seen.

Acknowledgements

I want to thanks @luislavena,@azolo and @jonforums for helping me out making gem-exefy.

Ruby on Windows Guides - the Book

I use Windows on my everyday work. First thing I do after installing Windows is to set up complete environment for Ruby. In “ancient” days (measuring in computer years, of course) I have used One Click Installer. Later, RubyInstaller came on the scene.

With much better foundations, RubyInstaller simplified Ruby usage and made it look like on Linux, Ruby’s native platform. Key benefit of RubyInstaller over One Click installer was possibility to install Ruby native gems written in C. Switching from commercial VisualStudio, used in One Click Installer, to MinGW build tool-chain in RubyInstaller was crucial decision made by Luis Lavena, great author of RubyInstaller.

After RubyInstaller, one more project appeared with the goal to simplify Ruby on Rails installation and usage on Windows. RailsInstaller. I haven’t tried it, but I’m sure it will help increasing number of Rails users on Windows.

But do you really need installer for Ruby on Rails on Windows? It depends of your attitude. If you are not interested what is going on behind the scenes and just want to be able to start using Rails after few mouse clicks RailsInstaller is definitely for you.

If, on the other hand, you want to know how things work, you should try to install Ruby on Rails without installer. With RubyInstaller installing Ruby on Rails on Windows is, actually, just a matter of issuing

1
2
gem install rails
gem install sqlite3

Of course you might face problems building and installing some gems, primarily because gem authors don’t want to bother with Windows support or simply because gems rely of third party libraries that are not ported to Windows. In any case you can ask for help on RubyInstaller mailing list.

People wonder if Ruby and Ruby on Rails are really usable on Windows. My opinion is that they are. Of course, there are some problems mainly with speed, but I would dare to say that solutions are on the way and hopefully very soon this will not be problem any more. I guess main reason why they are not so widespread on Windows platforms is scepticism and lack of experience.

As one of contributors to RubyInstaller project I noticed same questions are repeated on RubyInstaller mailing list. Mainly because of lack of familiarity with Ruby and RoR on Windows. That’s why I decided to write a book where I tried to give deeper insight on all details about installing and using Ruby and Ruby on Rails on Windows. The Ruby on Windows Guides is still in beta. However it already covers the most important issues and I think all novices and advanced developers can learn something new from it. Work on the book is still in progress so if you have any suggestions, ideas and wishes you can send them as an issue on the GitHub.

If you want to get deeper understanding of Ruby and Ruby on Rails on Windows, find out how to build native gems in several ways or just want to get an idea for what you can use Ruby on Windows go ahead and read the book. Of course I expect your comments, here or on GitHub.

Dep_walker Gem

Common problem that Ruby developers face on Windows is missing dll message box that appears when they try to use gem that has extension library. Usually these Gems are packed with pre-built binary\ extensions for windows and, even though, installation passes without any error or warning, when they try to use them they realize that dll these Gems depend on are missing on the system.

Maybe most common Gem for which this happens is sqlite3, Ruby binding for the SQlite3 embedded database. If sqlite3.dll is missing from the system, after installation of sqlite3 Gem any attempt to use it causes following message box to appear.

If two versions of a required dll exist on the system Gem might use the wrong one and strange crashes can appear.

These errors are not limited to the Gems that have pre-built binary extensions but can also happen if extension library was built with the RubyInstaller’s DevKit. If development files (header and library files) needed for the extension library to be built exist on the system while target dll is missing gem usage\ will cause the same message box to appear.

Frequently novice Ruby users on Windows ask question on mailing lists why Gem is not working even if it was installed without any error. That’s why I made dep_walker, small utility Gem, that can be used to check whether all dependencies for extension libraries used by installed Gem (s) are met or not. Source of the Gem can be found on GitHub.

Usage is very simple. If you want to check all installed Gems just invoke dep_walker with the ’-a’ switch.

1
dep_walker -a

And for particular gem swith -c can be used.

1
dep_walker -c sqlite3

More verbose output is obtained via -t and colour with --color swith. Happy dependencies walking!

Using Selenium With Cucumber Through Webrat or Capybara. Which One to Choose?

Introduction

Testing is (or should be) important part of every software development. Over time various testing strategies and supporting tools and frameworks have been developed. Regarding Web development biggest advance has been made in Behavior Driven Development. Consequently many tools for BDD are published and used.

Ruby on Rails framework had great built-in support for testing from the very beginning. As it usually happens, lot of specialised testing tools appeared aside of it and among all of them my favourites are RSpec and Cucumber. The first one for unit and the second for functional tests. Both of them are well integrated with Ruby on Rails and are very easy to set up and start with. Moreover there is no need to use real browsers which results in fast tests execution. Perfect way for BDD.

But what if you have to perform functional tests on non Ruby on Rails applications or your application relies heavily on JavaScript (no matter in which framework it is written)? Luckily Cucumber can be used in that case too. Since Cucumber supports Rails out of the box there is basically no need for some special configuration. On the other hand if a real browser must be used in tests, or functional testing must be done outside of the Rails environment setting up Cucumber can be little tricky but still simple enough. In this article I will focus on this scenario – testing non Rails applications with Cucumber and Selenium.

Using Selenium in Cucumber tests is done through Webrat or Capybara. First we must set up complete environment, and in the first step all necessary gems must be installed:

1
2
3
4
5
6
7
gem install launchy
gem install rspec
gem install cucumber
gem install webrat
gem install capybara --pre
gem install selenium-client
gem install selenium-webdriver

Option —pre is used to install Capybara 0.4.0 rc

Important notice for MS Windows users: Webrat depends on Json gem which installs binaries compiled against Rubyinstaller Ruby 1.8.x version. If you are using 1.9.2 Ruby you must uninstall Json gem and install it again but with --platform=ruby option:

1
2
gem uninstall json
gem install json --platform=ruby

Since article focuses on the functional testing outside of Rails we should manually create folder structure that Cucumber expects.

tests
 |- features
     |- support
     |- step_definitions

All .feature files go in the features folder. In the support folder env.rb file should be created and within it all set up must be made. Finally steps are implemented in Ruby files in step_definitions folder.

Webrat

Webrat controls Selenium through Selenium RC (remote control) and selenium-client gem. In order to use Selenium through Webrat put following code in your env.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
require 'cucumber/formatter/unicode'

require 'webrat'
require 'webrat/core/matchers'

Webrat.configure do |config|
  config.mode = :selenium
  config.application_framework = :external
  config.selenium_server_address = '127.0.0.1'
    if RbConfig::CONFIG['host_os'] =~ /mingw|mswin/
  config.selenium_browser_startup_timeout = 60
  config.application_address = 'localhost'
  config.application_port = '3000'
end

World do
  session = Webrat::Session.new
  session.extend(Webrat::Methods)
  session.extend(Webrat::Selenium::Methods)
  session.extend(Webrat::Selenium::Matchers)
  session
end

That’s all if you are running Linux based system. On Windows a little bit more effort must be made. First of all, Webrat usess 0.0.0.0 IP address when it starts Selenium and MS Windows does not like it at all. Secondly it uses /dev/null stream which is not available on MS Windows. Patch is already submitted and you can follow a ticket at Webrat Lighthouse. But until fix is accepted and new version is released, you can take a patch from Github gitst and apply it to Webrat sources.

Besides this patch few more things must be done. Line:

1
config.selenium_server_address = '127.0.0.1' if RbConfig::CONFIG['host_os'] =~ /mingw|mswin/

must be added to the config block as is already shown in the above snippet. Unfortunately selenium-client gem does not recognize Rubyinstaller since it is built using MinGW tools. Therefore one more tiny patch must be made in the selenium-client-1.2.18/lib/nautilus/shell.rb file. Function windows? must be replaced with:

1
2
3
def windows?
  ::RbConfig::CONFIG['host_os'] =~ /mswin|mingw/
end

You are ready for application testing. By default Selenium will use Firefox and if you want to use other browser (in the example Internet Explorer is set) add following line to config block:

1
config.selenium_browser_key = '*iexplore'

Capybara

Although Capybara can use Selenium RC, it primarily uses Selenium WebDriver which is still in beta phase but is working good. Since we already installed all necessary gems we can go on with configuring our testing environment. File env.rb should look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
require 'rbconfig'
require 'cucumber/formatter/unicode'

require 'capybara'
require 'capybara/dsl'
require "capybara/cucumber"

Capybara.default_driver = :selenium
Capybara.app_host = "http://127.0.0.1:8000/"
Capybara.register_driver :selenium do |app|
  Capybara::Driver::Selenium.new(app, :browser => :firefox)
end

World(Capybara)

Setting up Capybara is definitely much easier. But on MS Windows systems, if you want to use Internet Explorer, you still have to patch sources. Authors are already notified about required patch and I believe that new version of selenium-webdriver gem will be released with it. In the meantime you just have to change definition of initialize method in selenium-webdriver-0.0.28/lib/selenium/webdriver/ie/bridge.rb from:

1
def initialize()

to

1
def initialize(opts = {})

Changing browser is as easy as changing :firefox to :ie or :chrome. Instead of :firefox you can also use :ff and for Internet Explorer :internet_explorer. One more notice about differences if you are switching from Webrat to Capybara. Capybara will reset session after each step. If you do not want that (for example you log in to your application in the first scenario, and do not want to repeat it in each succeeding one) just add:

1
2
After do
end

in env.rb file.

With Capybara you are not limited to Selenium WebDriver. If you want to use Selenium RC you just need to configure it in env.rb file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
require 'rbconfig'
require 'cucumber/formatter/unicode'

require 'capybara'
require 'capybara/dsl'
require "capybara/cucumber"

Capybara.default_driver = :selenium
Capybara.app_host = "http://127.0.0.1:9000/"
Capybara.register_driver :selenium do |app|
  # This way we are using Selenium-RC
  Capybara::Driver::Selenium.new(app,
                                 :browser => :remote,
                                 :url => "http://127.0.0.1:4444/wd/hub",
                                 :desired_capabilities => :internet_explorer)
end

World(Capybara)

Conclusion

Both gems for running Selenium as a base for functional tests – Webrat and Capybara are easy to use. Although Webrat needs more patching to work under Windows it has one advantage. It can be used with Mechanize if you do not need real browser and you still want to test non Rails application. But as much as it is advantage for “classic” Web application Mechanize cannot interpret JavaScript. So if you want to include JavaScript testing you either have to use real browser or switch to Capybara.

Capybara, on the other hand, needs significantly less patching on MS Windows systems and it cannot use Mechanize as far as I know. But, from my point of view, it is easier to use then Webrat. Currently it cannot use Mechanize, but it can use Culerity and Celerity for JavaScript testing. Moreover capybara-envjs driver can be used to interpret JavaScript outside of the browser.

Although I’m still not sure which one is better to use, I switched from Webrat to Capybara and I think that tests that use Selenium WebDriver are running faster. There is an initiative for merging these projects into one but I do not know if it will happen and when. I would like to hear what you think. What is your choice: Webrat or Capybara?

ActiveRecord SchemaDumper and MySQL Problem

After finishing first version of Rmre and issuing fix gem dependency in version 0.0.2, I got an idea for additional functionality. Why not use Rmre for dumping complete schema with all foreign keys data? What would be possible scenario for using this, one might ask? We have possibility to create ActiveRecord models in order to move to Ruby on Rails where main premise is to keep logic out of database and maintain it in application. Therefore we do not need foreign keys since we already have constraints defined in models.

But what if you cannot move to Ruby on Rails and you only have to change DBE, i.e. instead of MS SQL you must use Oracle? In that case you still have to work with legacy database from PHP or Hibernate in Java and “only” thing you have to do is to make create script for all tables but for another DBE. When database has hundreds of tables with lots of relations this can turn into nightmare, especially if you have to maintain both versions.

Rmre should simplify this. First you use Rmre to dump schema to some file and later you can use ActiveRecord’s capabilities to load it on different DBE. Since loading schema in ActiveRecord is DBE agnostic it should correctly create tables, indices and foreign keys on any database engine. That’s theory and, as usual, practice is a little bit different. On a very first step I’ve faced problem in MySQL database.

Let’s examine database with just a two tables – city and country. Create script would look like (example from Sakila database):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
CREATE TABLE city (
  city_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT,
  city VARCHAR(50) NOT NULL,
  country_id SMALLINT UNSIGNED NOT NULL,
  last_update TIMESTAMP NOT NULL DEFAULT
    CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY  (city_id),
  KEY idx_fk_country_id (country_id),
  CONSTRAINT `fk_city_country`
    FOREIGN KEY (country_id)
    REFERENCES country (country_id)
    ON DELETE RESTRICT ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE country (
  country_id SMALLINT UNSIGNED NOT NULL AUTO_INCREMENT,
  country VARCHAR(50) NOT NULL,
  last_update TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
    ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY  (country_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

As can be seen from above script table city has foreign key on table country. Now let’s see what is result of a dump:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
ActiveRecord::Schema.define(:version => 0) do

  create_table "actor", :primary_key => "actor_id", :force => true do |t|
  create_table "city", :primary_key => "city_id", :force => true do |t|
    t.string    "city",        :limit => 50, :null => false
    t.integer   "country_id",  :limit => 2,  :null => false
    t.timestamp "last_update",               :null => false
  end

  add_index "city", ["country_id"], :name => "idx_fk_country_id"

  create_table "country", :primary_key => "country_id", :force => true do |t|
    t.string    "country",     :limit => 50, :null => false
    t.timestamp "last_update",               :null => false
  end

  execute "ALTER TABLE city ADD CONSTRAINT fk_city_country FOREIGN KEY (country_id) REFERENCES country(country_id)"
end

At the first glance this looks good but unfortunately doesn’t work. Problem is that loading this schema through ActiveRecord will create columns city_id in table city and country_id in table country as integer type but column country_id in table city is created as smallint. Defining constraint on columns which are not of same type is not allowed so last statement for altering table fails. At the moment I have no idea how to fix this and any suggestion is very welcome. I still have to check what happens on other DBEs: PostgreSQL, Oracle and MS SQL.

RMRE - Rails Models Reverse Engineering Gem

Very often I have to work on databases which do not follow ActiveRecord convention and making ActiveRecord models, if number of tables is large, is very slow and boring task. In order to speed up and simplify it I’ve created Rmre gem. Gem is quite simple yet you might find it useful if you want to create fixtures, migrations or simply port application to Ruby on Rails.

So how it works? For each table in the database, gem creates model. Name of the model is created using Rails classify method. Moreover, if table’s primary key is not column named “id” gem sets primary key by adding set_primary_key "primaryKeyColumnName" line to the model. In addition for MySQL, PostgreSQL, Oracle or MS SQL foreign keys are analyzed and for each constraint gem generates belongs_to or has_many lines. Here is model created for table store in Sakila MySQL test database:

1
2
3
4
5
6
7
8
9
class Store < ActiveRecord::Base
  set_primary_key :store_id
  set_table_name 'store'
  has_many :customers, :class_name => 'Customer'
  has_many :inventories, :class_name => 'Inventory'
  has_many :staffs, :class_name => 'Staff'
  belongs_to :address, :class_name => 'Addres', :foreign_key => :address_id
  belongs_to :staff, :class_name => 'Staff', :foreign_key => :manager_staff_id
end

Ruby 1.9.2 on Windows - Coming Soon

I guess all Ruby and Windows users will be happy to hear that RubyInstaller team is about to release Ruby 1.9.2p0 very soon. See details in this thread. It will take some time to update documentation so be patient and keep an eye on the project home page.

Upgrade to Lucid Lynx Problems

After several successful updates of Ubuntu (starting from 8.10) last one was, more or less, complete mess. I was waiting impatiently for new Lucid Lynx to be released and started update as soon as I downloaded and burned DVD iso image. It seems everything goes smoothly till first reboot. Emacs was completely removed and new version was not installed. On my Compaq 8710w laptop with Quadro FX 1600M graphics card splash screen was displayed in low resolution and looked quite ugly. Moreover I was not able to install Emacs again.

I’ve decided to reinstall everything, so I’ve made backup of my home folder and start installation on a newly formatted partitions. Situation was a little bit better. I was able to install Emacs and all applications I need but problem with graphics card was still there as well as lousy splash screen. I’ve found few suggestions how to fix that by altering grub files and it really solved resolution problem but, at the same time, whole splash screen was shifted left and basically I was again on the beginning.

There is one more thing that annoyed me. I like Shere Khan Black Hand mouse pointers so I’ve installed that theme, but arrow pointer was still white, although some other pointers were changed (hand is small and black). Whatever I’ve tried arrow was keep staying white.

Then, after a few days, I saw black arrow and was really surprised. It turned out that it was unpleasant surprise because compiz stopped working and Nvidia driver was not enabled. Obviously it was time to try latest Nvidia drivers so I’ve downloaded and installed them. I must say that situation is now a little bit better – compiz is working, but I still have no luck with mouse pointers, nor with splash screen.

If Canonical’s goal is to increase number of Ubuntu users, they should really try to avoid such problems, since average user will certainly not be able to fix them and will switch to some other operating system.

Ruby, Rails and MS SQL Server

Setting up Rails and Ruby to use MS SQL server was always painful task. Fortunately things have changed – a lot! With new Rails SQL Server 2000, 2005 and 2008 Adapter and Christian Werner’s ruby-odbc gem you can do it in a few minutes.

If you want to use these gems on Windows grab Ruby installation from RubyInstaller site and be sure to install DevKit prior to installing ruby-odbc.

Versions of ruby-odbc before 0.9999 do not work on mingw based (RubyInstaller) Ruby. Luckily author was very fast and made new version very quickly after I sent him a patch. Thanks Christian!

Both gems work well on Ruby 1.8.6 and 1.9.1 Ruby versions on Windows with old ActiveRecords, but I hope rails adapter will be ported to ActiveRecords 3 soon.