Prod-to-dev with Mnesia

综合编程 ikura

Back in my dark days of Rails development
, it was essential to have a way to populate one’s local database, say SQLite
, with production data. And for some reason, half a decade has gone by as an OTP
engineer before I needed to perform this task in Erlang-land.

After looking into the matter, it was easy enough, but the documentation was scattered. So, I thought I would share my process for getting a production node’s db-data onto my local development machine.

Ramping up

First off, I want to define a few things for the sake of this guide. I will refer to our single production box via the name ‘jimbo@prod’ while our development name will be ‘jimbo@dev’ — these, of course, are the VM
’s sname
values, or short names, and in your case, you’ll probably be using fully qualified long names.

Also, we will assume our Unix user-name on production and development is ‘jimbo’ for both.

Lastly, we will assume you have a preferred strategy for attaching to production nodes — either via Erlang’s remote shell strategy, or via erl_pipes
.

Dumping production

The first order of business, is to make a copy of the production database. We will want to output it somewhere, so follow along and we will take care of that. On our production box, let’s create some directories to aid us along:

cd ~
mkdir tmp

Next, attach to your production system Erlang shell and perform the following:

1> mnesia:backup("/home/jimbo/tmp/mnesia.prod.backup").

Done. You can now detach from the production Erlang shell.

Now, change into that dump directory and tar
up the database contents for quick transfer to your development machine. Perform the following to do just that:

cd ~/tmp
tar cvfJ mnesia.prod.txz mnesia.prod.backup

And we can close our Unix session on production. We are all finished with it.

Dev prep

Our OTP
release could be anything at all. From a simple server, to something quite complicated. But for the sake of this guide, we need to make an assumption that our system has an API
resource whereby we can prime our Mnesia database for fresh installs of a target system.

There is no standard way of doing this, but to create the requisite database schema, I usually have some code that does this all in a deterministic way, such as foo_lib:init(db).
which can be called once, manually.

Here is some example code on what this could look like. Here is a snippet from a fictional module called ‘foo_lib.erl’ :

...

init(db) ->
    Nodes = [node()],
    mnesia:stop(),
    mnesia:create_schema(Nodes),
    mnesia:start(),
    try
        mnesia:table_info(type, user)
    catch
        exit: _ ->
            mnesia:create_table(user, [
              {disc_copies, Nodes},
              {attributes, record_info(fields, user)}
            ])
    end,
    try
        mnesia:table_info(type, pets)
    catch
        exit: _ ->
            mnesia:create_table(pets, [
              {disc_copies, Nodes},
              {attributes, record_info(fields, pets)}
            ])
    end,
    Tables = [user, pets],
    Reply  = mnesia:wait_for_tables(Tables, 15000),
    {status, Reply}.

...

This code is just to give you a clue about how to set up your database schema. Be opinionated about how you
like to do it, for sure. We will assume for the rest of this guide, a fresh Mnesia database can be primed after we start the OTP
system, then call foo_lib:init(db).

With that lengthy aside out of the way, let’s get ready to transfer our production data to our local box. First, let’s create the familiar working space with mkdir ~/tmp
and copy over the compressed database to there now:

scp jimbo@prod-box.net:~/tmp/mnesia.prod.txz /home/jimbo/tmp/

Let’s assume all went well with scp
, ssh access and such, and we will proceed with un-tarring the dumped database:

cd ~/tmp
tar xvf mnesia.prod.txz

There. A good deal of the nitty-gritty is out of the way.

Introducing ‘convert.erl’

Now’s the time to write some Erlang. We are going to create a new module that reads the production Mnesia database and converts the naming of the production node, to that of our development one, wherever it occurs.

Recall, that our production sname
is ‘jimbo@prod,’ but locally, it’s ‘jimbo@dev.’ That’s why we need to do this.

Here is the contents of the new ‘convert.erl’ source-file; this is on our development box & placed in ‘/home/jimbo/tmp’ :

-module(convert).

-export([new/0]).

new() ->
    From = 'jimbo@prod',
    To   = 'jimbo@dev',
    Path = "/home/jimbo/tmp",
    change_node_name(
      mnesia_backup, From, To, 
      Path ++ "/mnesia.prod.backup",
      Path ++ "/mnesia.dev.restorable").

change_node_name(Mod, From, To, Source, Target) ->
    Switch =
      fun(Node) when Node == From -> To;
         (Node) when Node == To -> throw({error, already_exists});
         (Node) -> Node
      end,
    Convert =
      fun({schema, db_nodes, Nodes}, Acc) ->
          {[{schema, db_nodes, lists:map(Switch,Nodes)}], Acc};
         ({schema, version, Version}, Acc) ->
          {[{schema, version, Version}], Acc};
         ({schema, cookie, Cookie}, Acc) ->
          {[{schema, cookie, Cookie}], Acc};
         ({schema, Tab, CreateList}, Acc) ->
          Keys = [ram_copies, disc_copies, disc_only_copies],
          OptSwitch =
            fun({Key, Val}) ->
                case lists:member(Key, Keys) of
                    true -> {Key, lists:map(Switch, Val)};
                    false-> {Key, Val}
                end
            end,
          {[{schema, Tab, lists:map(OptSwitch, CreateList)}], Acc};
         (Other, Acc) -> {[Other], Acc}
      end,
    mnesia:traverse_backup(Source, Mod, Target, Mod, Convert, switched).

Now, all the while working in ‘/home/jimbo/tmp’ we compile the new module using the humble erlc
command. We do that as follows:

erlc convert.erl

Next, let’s put convert
to work and get our database dump in a state where our local machine can work with. In that same directory, do as follows:

erl

1> l(convert).
2> convert:new().
3> init:stop().

Done. We now have a copy of our database ready to be loaded into the development version of our OTP
project. If you ls -lhart
the directory, you can see the new file convert
created.

Pulling it in

Go ahead and change directories to where your OTP
project lives.

It’s time to fire up our project locally, but first we make sure any old Mnesia development data is blown away so not to interfere. Assuming that the location of these old Mnesia database files are in the project’s root directory, you can tar
those up and place them aside, or simply rm -fr ./Mnesia.*
if you won’t be missing any development data.

Now go ahead and start up your OTP
release locally, and attach to it in your favorite manner.

Recall that we now have a missing database schema, and need to create one afresh. We will want to do that right away, followed by a Mnesia routine that populates our local database with our production data. We do both as follows:

1> foo_lib:init(db). %%%% for example
2> A = "/home/jimbo/tmp/mnesia.dev.restorable".
3> mnesia:restore(A, []).

And there you have it: your local Mnesia database now holds all the goodies from production. If this is something you need to do often, then the lion’s share of the above ought to be automated. But there’s no shame in bookmarking this post & coming back here when you need to do it all again.

what’s ikura?

follow on twitter

稿源:ikura (源链) | 关于 | 阅读提示

本站遵循[CC BY-NC-SA 4.0]。如您有版权、意见投诉等问题,请通过eMail联系我们处理。
酷辣虫 » 综合编程 » Prod-to-dev with Mnesia

喜欢 (0)or分享给?

专业 x 专注 x 聚合 x 分享 CC BY-NC-SA 4.0

使用声明 | 英豪名录