Replace S3 with DigitalOcean as asset server for Spree commerce

In this post I will show steps need to setup DigitalOcean as asset server for Spree projects. So before going into any detail, I want to explain a bit the reason I did it. S3 was great for me. It’s simple to setup and use. Until one day my bill for Amazon S3 go up to ~7k USD/month. I was freak out. And the reason was, one of our dev guys have use aws key for testing purpose and he was not careful enough. And the rest of the story you already know what was going on. Then I decided to setup the asset server on DigitalOcean with only 5$/month and I got 20Gb of SSD storage. Which I think pretty good.

Ok. So here is what you need to make it happen.

First, add the following gem to your Gemfile `paperclip-storage-ftp` at https://github.com/xing/paperclip-storage-ftp. Second, you gonna need to do the config for Spree. You can add a file called `spree_ftp.rb` under `initializers` folder

attachment_config = {
  storage: :ftp,
  path: "/home/your_project_path/files/:attachment/:id/:style/:filename",
  url: "http://X.X.X.X/:attachment/:id/:style/:filename",
  styles: {
    mini: "48x48>",
    small: "100x100>",
    product: "240x240>",
    large: "600x600>"
  },
  default_style: "product",
    ftp_servers: [
      {
        host: "X.X.X.X",
        user: "your-ftp-user-name",
        password: "your-ftp-password"
      }
    ],
  ftp_connect_timeout: 5,
  ftp_ignore_failing_connections: true,
  ftp_keep_empty_directories: true
}

attachment_config.each do |key, value|
  Spree::Image.attachment_definitions[:attachment][key.to_sym] = value
end

That is all for Spree part. Now for DigitalOcean, you gonna need 2 things.

  • 1 is a nginx server – to serve the static content only (png, jpg,..)
  • 2 ftp server – for your Spree project upload asset to

Setup an nginx server on DigitalOcean quite easy and straight forward. One thing to keep in mind that you need to config Nginx to serve only static content (nothing else) and only authorize public users to access to the asset folder which you defined in the Spree config above.

For FTP server setup, you can follow great tutorial here https://www.digitalocean.com/community/tutorials/how-to-set-up-vsftpd-on-ubuntu-12-04 – This can be easily setup within 15 – 30 minutes. Then one last thing you need to do the config for FTP server also. By edit the following file `/etc/vsftpd.conf`. Here is my config as an example

Screen Shot 2016-03-15 at 9.15.46 PM

That’s all. Now I can upload all of my assets into DigitalOcean. The result was pretty great for me. Nginx handle static images very fast. I happy with it and now I gonna move all images from S3 to DigitalOcean.

Replace S3 with DigitalOcean as asset server for Spree commerce

Great text editor for Golang

You might think this is another blog post about Vim, Emacs, Atom, or Sublime. No, it’s not. All of those editor are super great. I have used all of them, and still using Atom for all of my Rails projects, and using Vim for all Linux related tasks. I’m happy with them. However when it come to Golang, I kind of feel not comfortable very much with those tools. I don’t need all of those configs for other stuff from Atom or Vim. Then I dis some research and found this amazing little small editor but still powerful, and it’s Textadept. http://foicica.com/textadept

Textadept is an text editor written in about 200o lines of Lua. And it’s super fast!!!

Here is a quick screenshot of how it look with my Golang code:

Screen Shot 2016-03-14 at 11.17.27 PM

Here is the reasons why I love it so much

  1. It’s very fast
  2. It support Mac/Linux/Window
  3. It’s light weight
  4. It got all handy shortcuts from other editors (Atom/Sublime) which still allow me coding very fast. Search, multi-cursor edit, moving cursors around, …
  5. It support Golang out of the box. I don’t need to do any config at all and it just work. Stuff like syntax highlight, go format … (similar to go-plus for Atom). And it’s even got very handy shortcut like: `command+R` -> then it gonna build and run the go program for me. So I don’t have to go to terminal and type `go run main.go` anymore.      Screen Shot 2016-03-14 at 11.29.49 PM
  6. It’s configurable. Like Vim got .vimrc – Textadept got it’s init file also where you can do all kind of customization that fit your need. Stuff like custom functions, themes…

Textadept is exact text editor I want. It’s small, simple, but still powerful. I highly recommend you give it a try.

Great text editor for Golang

Setup unit test for Swift command line app and CI with Travis

1. Adding test bundle for command line app

Writing unit tests for iOS or Cocoa App is way more straight forward rather than command line app. When you create iOS/Cocoa App, then you already got a test bundle come with free. So you don’t have to do setup or anything. You just simply add more test cases.

There’re couple of suggestions on StackOverFLow, but didn’t work for me. And here is the setup that would allow me write test cases for command line app.

  • Step 1: Open Xcode, go to `Editor` -> `Add Target` -> then choose `OSX Unit Testing Bundle`, then just next, next ..
  • Step 2: Edit scheme, click on `Test` -> `+` button to add

That’s all. Now all of your code would be available for test.

**TIP**:

  • Normally any command line app would require some arguments. And your build would fail if you don’t provide some default argument for the purpose of build. You can do so by `Edit Scheme` -> go to `Run` and add `arguments passed on launch`
  • One more tip, in order to use all stuff from you code you gonna need to use `@testable` to import the module contain you code

2. Adding CI (Travis)

Implement a CI for an Xcode project might not very clear for many people. The reason is because there’re not many resouce around it. Normally it would require many failed experiments to make it work with your cases. And here is the setup that work with mine.

I used the Swift command line app as a part of ruby web server. And here is it (`.travis.yml`)

# .travis.yml
matrix:
  include:
    - os: osx
      language: objective-c
      xcode_project: YourProjectName.xcodeproj
      xcode_scheme: YourProjectName
      script: cd YourProjectName && xcodebuild -scheme YourProjectName test
      osx_image: xcode7.2
      before_install:
        - rvm install 2.2.2
        - rvm use 2.2.2 --default
        - gem install bundler
        - bundle config build.eventmachine --with-cppflags=-I/usr/local/opt/openssl/include

Look pretty simple right. From the config above the most important line is this one `cd YourProjectName && xcodebuild -scheme YourProjectName test` -> this mean go into your Swift project and do the build from therer and run the test. For the rest you can look into travis tutorial, they’re pretty easy to understand. Again your project would be different but the structure gonna be quite similar.

You just got the project setup and CI ready. Now time to add some meaningful unit tests.

Setup unit test for Swift command line app and CI with Travis

Top Postgresql queries to checkup on database performance

These are the list of PostgreSQL queries which help me a lot over the time to troubleshoot problems with the database.

1. Find out which queries running currently

SELECT 
  pid, client_addr, query, state 
FROM pg_stat_activity;

2. Table access stat
This query is really helpful when you want to find out which table is most frequent accessed and by which way. Ex: sequence scan or index scan.

SELECT 
  schemaname,
  relname,
  seq_scan,
  idx_scan,
  cast(idx_scan AS numeric) / (idx_scan + seq_scan) AS idx_scan_pct
FROM pg_stat_user_tables WHERE (idx_scan + seq_scan) > 0
ORDER BY idx_scan_pct;

3. Find out missing indexes
Well, index is one of the most crucial thing which make the database fast or slow. So check for missing indexes are super important. And we can doing that with simple query like below.

SELECT 
  relname, 
  seq_scan-idx_scan AS too_much_seq, 
  case when seq_scan-idx_scan> 0 THEN 'Missing Index?' ELSE 'OK' END,    
  pg_relation_size(relname::regclass) AS rel_size, 
  seq_scan, 
  idx_scan
FROM pg_stat_all_tables
WHERE schemaname='public' AND pg_relation_size(relname::regclass) > 80000 
ORDER BY too_much_seq DESC;

4. Find out unused indexes
Well, indexes are great however over indexing will slow the database down. So how we know that we’re over indexing. Ok, we can use the following query.

SELECT
  relid::regclass AS table,
  indexrelid::regclass AS index,
  pg_size_pretty(pg_relation_size(indexrelid::regclass)) AS index_size,
  idx_tup_read,
  idx_tup_fetch,
  idx_scan
FROM pg_stat_user_indexes
JOIN pg_index USING (indexrelid)
WHERE idx_scan = 0;

5. Find out duplicate indexes
Over indexing is one of the problem, and duplicate index also another problem. We can find that out with the following query.

SELECT 
  pg_size_pretty(sum(pg_relation_size(idx))::bigint) AS size,
  (array_agg(idx))[1] AS idx1, (array_agg(idx))[2] AS idx2,
  (array_agg(idx))[3] AS idx3, (array_agg(idx))[4] AS idx4
FROM (SELECT 
        indexrelid::regclass AS idx, 
        (indrelid::text ||E'\n' || indclass::text ||E'\n'|| indkey::text ||E'\n'||
         coalesce(indexprs::text,'') || E'\n' || coalesce(indpred::text,'')) AS KEY
      FROM pg_index) sub
GROUP BY KEY HAVING count(*)>1
ORDER BY sum(pg_relation_size(idx)) DESC;

6. Find out tables size
Checkup on table size is also important so we can know which table is largest and we can take some action such as: archiving or truncating the table.

SELECT 
  relname as "Table",
  pg_size_pretty(pg_relation_size(relid)) As " Table Size",
  pg_size_pretty(pg_total_relation_size(relid) -
  pg_relation_size(relid)) as "Index Size"
FROM pg_catalog.pg_statio_user_tables 
ORDER BY pg_total_relation_size(relid) DESC;

Actually there’re more ways help to monitor the performance of the database. However those queries I find most useful for my case. So hope it’s also helpful to you.

Top Postgresql queries to checkup on database performance

How did I upgrade Rails 4.1.7 to Rails 4.2.4

I recently upgraded our Rails 4.1.7 codebase to the latest stable version of Rails 4.2.4. Since our codebase is quite large. We use lot of different gems. So the upgrade would definitely break lot of things. So I decide to do it step by step.

1. Upgrade to Rails 4.1.9

This step was quite easy for me. By changing the Rails version only in the Gemfile. And fix any dependencies conflict if any. And all specs was green for me. So all good, now ready for Rails 4.2.0

2. Upgrade to Rails 4.2.0

This is the important part. And I followed all the notes from here http://edgeguides.rubyonrails.org/upgrading_ruby_on_rails.html#upgrading-from-rails-4-1-to-rails-4-2. And yeah, it failed nearly of the unit tests (controllers/presenters/models/libs/..). In this step I only fix dependencies conflict. And got the bundle install done. I just ignored all those failed tests. I got a feeling that it might break again when I upgrade to 4.2.4. So I hold on the that until 4.2.4

3. Upgrade to Rails 4.2.4

Ok. So this is the final step. I planed to remove all the deprecated warnings. Of course after resolving all the dependencies conflicts. Updating all the gems which require >= Rails 4.2.x. Rails 4.2.4 actually changes many thing that not mentioned in the official documentation. Here’re couple of them:

  • exists?(object) -> deprecated. Solution exists?(id)
  • Arel::SqlLiteral.new -> undefined object. Solution Arel::Nodes::SqlLiteral.new
  • belongs_to :objects, class: AnotherObject -> doesn’t exist class option anymore. Solution belongs_to :objects, class_name: ‘AnotherObject’
  • where_values -> doesn’t work as expected like in the old version. This method not recommend to use. Solution: in my case I choose to use Arel instead to achieve the same result. 
  • deliver -> This method will be deprecated in Rails 5. However I resolve it anyway by using deliver_now or deliver_later
  • And there’re many more small things come up. All I needed to do is solve them one by one. Until I got all the unit tests pass. In case you don’t have the unit tests. Then you pretty mess up.

Final words: Upgrading Rails 4.2.4 was not so difficult (in my case). Just require little of patient and debugging skill. The errors or breaks might different from each codebase. It depends on the gems you uses and the code also. So keep calm and upgrading Rails.

4. Couple of links that I found useful

How did I upgrade Rails 4.1.7 to Rails 4.2.4

Why I think Elixir is one of the future programming language?

I recently has very much interested in a programming language called Elixir. I heard about Elixir more than a year ago. But I never took a chance to look detail into it until recently. My impression to it is small, simple, and powerful language.

So I decided to give it a chance by solving Elixir programming challenging on Exercism. By solving those problems I gradually learn Elixir syntax, building block, and feel the language. I actually like it alot. Even though it’s functional programming language but the syntax really friendly, not like Haskell or Clojure. Elixir actually like Ruby a lot. Since I got strong Ruby background so this one big thumbup for me.

Anyway, here is what I think Elixir can be one of the future programming language.

1. It’s powerful language

It’s a concurrent language built on top of Erlang Virtual Machine. Which mean it will carries all the power from Erlang. Even though I haven’t use any of those thing in real project yet. But this is a huge promise.

2. The syntax is feasible 

Unlike many other functional programming languages the syntax really hard to wrap in mind. Elixir provides nice, clean, and really compact syntax. Which super great for me. The simpler the better.

3. Great toolbox

Great language is not enough for any real projects. It must come with extra tools for developing and debugging. And yes, Elixir got a lot of those kind of tools. Many of them come with standard libraries. For example: `iex` the interactive console, very much similar to `irb` in Ruby. And `Elixir Pry` for debugging also similar to `Pry` in Ruby/Rails, and so much more.

4. Great community

This is one of the most important thing for any open source project – Community. Without the support of the community no software can exist for long time. Elixir community is getting more momentum everyday. By looking at the open source projects written in Elixir on Github. And the Elixir meetup around the world.

Final thought, I got very positive feeling about the future of Elixir. And decided it become one of my main programming languages arsenal.

# My solution for Elixir problems on Exercism.io https://github.com/kimquy/elixir-exercies

Why I think Elixir is one of the future programming language?

Rails – Table partitioning with Postgresql

Sometime you have to deal with a huge table with dozen of million rows or hundred of million rows. And that’s also the time the performance of the database go down. The most common technique to help improve the db performance is do the indexing properly. However in this case, there’s another technique that will become very powerful which is table partitioning.

Here is the short definition of table partitioning: “Partitioning allows a table, index, or index-organized table to be subdivided into smaller pieces, where each piece of such a database object is called a partition. Each partition has its own name, and may optionally have its own storage characteristics.”

In this post, I will present a way that we could employ this idea in a ruby on rails application.

# run this to generate an empty Rails app
rails new demo-partition -d postgresql
# Add this to your Gemfile
# This gem help to populate some fake data
gem 'faker'

After having project setup, we gonna need to create a user model. And here is the migration file I got:

class CreateUsers < ActiveRecord::Migration
  def change
    create_table :users, id: :uuid do |t|
      t.string :name, null: false
      t.string :email, null: false
      t.string :address
      t.string :birthday

      t.timestamps
    end

    add_index :users, :id, unique: true
    add_index :users, :created_at
  end
end

Here is come to the interesting part. I will create 5 different partitions for the users table.

create table users_1 ( like users including all ) inherits (users);
alter table users_1 add constraint partition_check check (created_at >= '2013-09-01' and created_at < '2014-04-01'); 

create table users_2 ( like users including all ) inherits (users); 
alter table users_2 add constraint partition_check check (created_at >= '2014-04-01' and created_at < '2014-07-01'); 

create table users_3 ( like users including all ) inherits (users); 
alter table users_3 add constraint partition_check check (created_at >= '2014-07-01' and created_at < '2014-10-01'); 

create table users_4 ( like users including all ) inherits (users); 
alter table users_4 add constraint partition_check check (created_at >= '2014-10-01' and created_at < '2015-03-01'); 

create table users_5 ( like users including all ) inherits (users); 
alter table users_5 add constraint partition_check check (created_at > '2015-03-01');

Let me explain a bit what the code above does. It will create 5 different sub-table which will inherit all the fields and indexes from the users table (master) and add a constraint to each sub-tables. As you can see the constraint is the time when user created. So each sub-table will hold data for each period of time.

And the question is how do we insert into the correct partition. That’s gonna be another step which require us to do a trigger on Postgresql insertion.

CREATE OR REPLACE FUNCTION on_users_insert() RETURNS TRIGGER AS $$
BEGIN
   IF (NEW.created_at >= '2013-09-01' AND NEW.created_at < '2014-04-01') THEN 
      INSERT INTO users_1 VALUES (NEW.*); 
   ELSIF (NEW.created_at >= '2014-04-01' AND NEW.created_at < '2014-07-01') THEN 
      INSERT INTO users_2 VALUES (NEW.*); 
   ELSIF (NEW.created_at >= '2014-07-01' AND NEW.created_at < '2014-10-01') THEN 
      INSERT INTO users_3 VALUES (NEW.*); 
   ELSIF (NEW.created_at >= '2014-10-01' AND NEW.created_at < '2015-03-01') THEN 
      INSERT INTO users_4 VALUES (NEW.*); 
   ELSIF (NEW.created_at > '2015-03-01') THEN
      INSERT INTO users_5 VALUES (NEW.*);
   ELSE
   END IF;

   RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER insert_trigger
BEFORE INSERT ON users
FOR EACH ROW EXECUTE PROCEDURE on_users_insert()

One last thing we need to do is to turn on the Constraint Exclusion, otherwise all of the above will not work.

SET constraint_exclusion = on;

Cool, now we can put all the sql above into one single migration file like below

class CreatePartition1ForUser < ActiveRecord::Migration def change execute %Q{ create table users_1 ( like users including all ) inherits (users); alter table users_1 add constraint partition_check check (created_at >= '2013-09-01' and created_at < '2014-04-01'); create table users_2 ( like users including all ) inherits (users); alter table users_2 add constraint partition_check check (created_at >= '2014-04-01' and created_at < '2014-07-01'); create table users_3 ( like users including all ) inherits (users); alter table users_3 add constraint partition_check check (created_at >= '2014-07-01' and created_at < '2014-10-01'); create table users_4 ( like users including all ) inherits (users); alter table users_4 add constraint partition_check check (created_at >= '2014-10-01' and created_at < '2015-03-01'); create table users_5 ( like users including all ) inherits (users); alter table users_5 add constraint partition_check check (created_at > '2015-03-01');
    }

    execute %Q{
      CREATE OR REPLACE FUNCTION on_users_insert() RETURNS TRIGGER AS $$
      BEGIN
          IF (NEW.created_at >= '2013-09-01' AND NEW.created_at < '2014-04-01') THEN INSERT INTO users_1 VALUES (NEW.*); ELSIF (NEW.created_at >= '2014-04-01' AND NEW.created_at < '2014-07-01') THEN INSERT INTO users_2 VALUES (NEW.*); ELSIF (NEW.created_at >= '2014-07-01' AND NEW.created_at < '2014-10-01') THEN INSERT INTO users_3 VALUES (NEW.*); ELSIF (NEW.created_at >= '2014-10-01' AND NEW.created_at < '2015-03-01') THEN INSERT INTO users_4 VALUES (NEW.*); ELSIF (NEW.created_at > '2015-03-01') THEN
              INSERT INTO users_5 VALUES (NEW.*);
          ELSE
          END IF;

          RETURN NULL;
      END;
      $$ LANGUAGE plpgsql;
    }

    # Insert trigger
    execute %Q{
      CREATE TRIGGER insert_trigger
      BEFORE INSERT ON users
      FOR EACH ROW EXECUTE PROCEDURE on_users_insert()
    }

    # Constraint Exclusion
    execute %Q{
      SET constraint_exclusion = on;
    }
  end
end

Now let generate some sample data, so we can test the result of what we did above. I got the following rake file. It will populate 100000 users at random time and insert into the correct partition as we defined above.

require 'faker'

desc "Populate sample data"
task :sample => :environment do

  ### Insert into master users ###
  inserts = []

  100_000.times do
    name    = Faker::Name.name.gsub('\'',' ')
    address = Faker::Address.street_address.gsub('\'', ' ')
    time    = rand(2.year.ago..Time.now).to_formatted_s('db')

    inserts.push("('#{name}', '#{Faker::Internet.email}', '#{address}', '#{time}', '#{time}')")
  end

  sql = "INSERT INTO users (name, email, address, created_at, updated_at) VALUES #{inserts.join(", ")};"
  User.connection.execute(sql)
  ### END ###
end

Now you can run the following to create, migrate, and populate the data.

rake db:create db:migrate sample

Great! It’s time to check out the result:

rails c

# Under your rails console
t = User.last.created_at
User.where(created_at: t).explain
User.where(id: "00005f11-66d1-4a2e-92d1-3b5973b54b33").explain

And here is what I got:

Screen Shot 2015-08-22 at 5.19.06 PM

Screen Shot 2015-08-22 at 5.22.59 PM

In the first case, it’s only look at the right partition. This will improve the db performance significantly. Especially for the big table. And in the second case, it look at all partitions. In this example, I partitioned the table into 5 sub smaller tables by timestamp. However we can partition by id, username, or whatever that we think it suitable for our case. We just need to add the proper constraint for the partitions. Again, table partitioning is a great technique that can help us to improve the database performance.

Rails – Table partitioning with Postgresql