||= (or equal operator) in bash?

If you’re used to writing code in ruby, perl, or other languages that provide an ||= (or equal) type operator, you may someday find yourself wanting to do the same in a shell script. While this isn’t directly provided in bash, for example (at least not that I could find), you can emulate the same functionality.

The answer lies in using a little bit of of tomfoolery from the Advanced Bash Scripting Guide, specifically under the Parameter Substitution section.

The part we’re interested in at this point is the ability to use a default parameter. These constructs in bash look like so:

${parameter-default}
and
${parameter:-default}

Note that as pointed out in the documentation, the difference between the two is that the former will not evaluate if parameter is null whereas the latter will.

So just how do we accomplish something like:

$foo ||= $bar ||= $baz

Like so:

FOO=${FOO:-$BAR}
FOO=${FOO:-$BAZ}
FOO=${FOO:-foo}

So this short circuits just like the ||= type operator above. So say, for example, you wanted to spit out a diff of some code before confirmation of a submit or something. Ideally you want to fire up the diff program the user prefers. So why not do something like:

DIFF=${DIFF:-$MYOTHERDIFF}
DIFF=${DIFF:-diff}

In this example, we use what is in $DIFF first. If there’s nothing in $DIFF, we use $MYOTHERDIFF. Finally, if there is nothing in either, we use straight up diff. This way if certain diff variables are set to programs like gvimdiff, tkdiff, etc., they will be invoked.

ruby Net::SFTP put / upload file hang

There are quite a few posts around regarding problems with upload of files using the ruby Net::SFTP library written by Jamis Buck. You can see a smattering by googling for terms like ruby net::sftp upload hang. Several of them are in the context of problems with uploads under Capistrano as well as a couple of blog posts about potential workarounds, such as this one.

I decided to spend some time with my head in the Netbeans ruby debugger to see if I could get a glimpse more at the root of the cause. I didn’t set out to solve this or anything… because I figure if the author of the libraries himself hasn’t figured it out (or been able to make it consistently repro) then I probably don’t have much hope doing so. Not only that, but I can’t spend a week learning all the underlying stuffs to attempt to fix this bug in our production application. So a work around we shall have.

I did, however, make some interesting observations which may or may not be of use to someone else looking into the problem. First, some repro oddities, in no particular order:

  • I have been unable to repro in a primitive case with a toy script to simply upload a file.
  • I can consistently repro with “larger” files; works like a charm for smaller files. Unfortunately, no pattern consistently around CHUNK_SIZE seems to be evident. (See next item.)
  • I can make it work everytime if I reduce CHUNK_SIZE in your-net-sftp-code-dir/lib/net/sftp/operations/write.rb from 32 * 1024 to “something smaller.” I’ve made it work with 1024, 4096, 16384, etc. I mucked with this value after finding this thread on cpanforum about a potentially similar sounding issue with Perl’s Net::SFTP library.
  • The hosts in question don’t seem to matter. Sometimes remote, sometimes more proximal, sometimes localhost.

So here’s some code to look at (if I can make this fucking wordpress editor not destroy it). This is from your-net-sftp-code-dir/lib/net/sftp/operations/abstract.rb:

    # Execute the operation. If a callback is given, the operation will be
    # performed asynchronously with the callback being invoked when the
    # operation completes. If a callback is not given, the operation will
    # be performed synchronously, with the expected value being returned.
    def execute( *args, &callback )
      @log.debug "executing" if @log.debug?
.
      unless block_given?
        status = result = nil
        callback = Proc.new do |status, *pargs|
          status = status
          result = case
            when pargs.empty? then @session.status
            when pargs.length == 1 then pargs.first
            else pargs
          end
        end
      end
.
      @callback = callback
      @id = perform *args
      @log.debug "received request id #{@id}"
      @session.register( @id, self )
.
      unless block_given?
        i = 0
        @session.loop { status.nil? }
        @session.status = status
        return result
      end
    end

What I’ve discovered via the debugger is that, in a case where the problem repros, the callback is never invoked and it hangs inside of @session.loop(). The callback should actually be invoked inside of the do_status() function of the operation being performed. In this case, from write.rb:

    # Invoked when the server sends a status packet. If the status is FX_OK,
    # then the callback is invoked (if all data has been written), or the
    # next chunk is written to the server (if more data remains). Other
    # status codes are handled by the superclass.
    def do_status( code, message, language )
      if code == FX_OK
        @log.debug "[#{@id}] chunk written” if @log.debug?
        @pos += CHUNK_SIZE
.
        if @pos > @data.length
          @callback[ OK ]
          return
        end
.
        @driver.write @id, @handle, @offset + @pos, @data[@pos,CHUNK_SIZE]
        @session.register @id, self
      else
        super
      end
    end

I’m guessing that the FX_OK status packet is indeed making it back, but something is happening in the libraries such that the callback from the lower layer is not being invoked. When the callback isn’t invoked, the status is always nil and so we sleep in the @session.loop(). That’s our hang.

Ok… now someone fix it.

Using the Ruby LDAP extension library

So I was recently working on a ruby/rails project which had to interact with OpenLDAP. This was easy enough using this Ruby/LDAP extension library found on Sourceforge (LNUX). As a quick aside, there is also the whole ActiveLDAP RubyForge project which treats LDAP as another backend just like a SQL database with different adapters, etc etc. For now, however, I will discuss the extension library and will perhaps go into ActiveLDAP some other time.First and foremost, a good reference and place to start is the rdoc documentation for the extension library. With this in hand, it is easy to get started interacting with an LDAP server. In my examples, I’m assuming use of OpenLDAP. I’m also running on a Debian/sid system with the following package installed: libldap-ruby1.8. I have a fresh DIT with a base of dc=foo,dc=com and TLS configured like so in slapd.conf:
TLSCACertificateFile /etc/ldap/ssl/ca.crt
TLSCertificateFile /etc/ldap/ssl/localhost.foo.com.crt
TLSCertificateKeyFile /etc/ldap/ssl/localhost.foo.com.key
TLSVerifyClient never

Note: A very common gotcha with openldap/TLS configuration is that the certificate does not match exactly with the hostname. The cn of the server certificate MUST be the fqdn of the host and that fqdn MUST be used for the connection. This can also be the case for the client, but TLSVerifyClient is set to “never” by default as seen above. What does this mean? This means that if your OpenLDAP server is on baz.foo.com, the cn of the certificate must be “baz.foo.com” and you must use “baz.foo.com” to connect as the client. Anything else and you’ll probably see an error like “Error in the certificate.” Note that your ca certificate must also match accordingly.

My LDAP begins with the following:
rwoodrum@frums:~/tmp$ ldapsearch -x -W -D cn=admin,dc=foo,dc=com -b dc=foo,dc=com
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <dc=foo,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#</dc=foo,dc=com>
#
# foo.com
dn: dc=foo,dc=com
objectClass: top
objectClass: dcObject
objectClass: organization
o: foo
dc: foo
#
# group, foo.com
dn: ou=group,dc=foo,dc=com
ou: group
objectClass: top
objectClass: organizationalUnit
#
# bar_group, group, foo.com
dn: cn=bar_group,ou=group,dc=foo,dc=com
cn: bar_group
description: the bar group
gidNumber: 3000
objectClass: posixGroup
objectClass: top
memberUid: rwoodrum

And now, we’ll use a trivial module to modify this group, changing the memberUid attribute. Here’s the module code that will pull in from elsewhere:

require 'ldap'module FooLDAP
  def self.setup_ldap_connection
    conn = LDAP::Conn.new(host='localhost.foo.com')
    conn.set_option(LDAP::LDAP_OPT_PROTOCOL_VERSION, 3)
    begin
      conn.start_tls.nil?
    rescue
      conn.perror("debug")
      raise "error during start_tls"
    end

    conn.simple_bind(dn = 'cn=admin,dc=foo,dc=com', password = 'secret')

    begin
      !conn.bound?
    rescue
      conn.perror("debug")
      raise "error during simple_bind"
    end

    return conn
  end  def self.update_bar_group
    conn = setup_ldap_connection
    conn.modify('cn=bar_group,ou=group,dc=foo,dc=com', {"memberUid" => ["ryan_woodrum"]})
    conn.unbind
    return
  end
end

And now from outside, use the module:

require 'foo_ldap'

FooLDAP.update_bar_group()

After running, you can see the the memberUid attribute has changed inside of ldap:
rwoodrum@frums:~/tmp$ ldapsearch -x -W -D cn=admin,dc=foo,dc=com -b dc=foo,dc=com cn=bar_group
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <dc=foo,dc=com> with scope subtree
# filter: cn=bar_group
# requesting: ALL
#
#
# bar_group, group, foo.com
dn: cn=bar_group,ou=group,dc=foo,dc=com
cn: bar_group
description: the bar group
gidNumber: 3000
objectClass: posixGroup
objectClass: top
memberUid: ryan_woodrum
... snip ...
</dc=foo,dc=com>

Quid est demonstratum.

Form not submitting on enter key with multiple form fields

So this is one of those really stupid things that was virtually impossible to find on google. One permutation was the golden child and brought me to this wonderful blog post.

So I’ve got a stupid bug I’m trying to fix. Seems innocuous enough. Nope, one of those stupid, finicky, browser things. I have an AJAX form with multiple fields that has no submit button; instead onsubmit deals with what to do with the form. So my form looks simply like so:
<% form_field('password', @thing) do %>
<%= password_field(:thing, :password) %>
<%= password_field(:thing, :password_confirmation) %>
<% end %>
One may expect that if you entered information in these fields and pressed the enter key, it’d submit. It doesn’t. At least not in Firefox… which means IE has no hope. Safari, however is “smarter” and it actually works. But I digress…

If you had this form with a single field, it’d work like a charm:
<% form_field('password', @thing) do %>
<%= password_field(:thing, :password) %>
<% end %>
So the answer is simply that you can’t submit a form by pressing enter when having more than 1 input field. At least not with a submit button.

The solution?

Add an invisible input button:
<input style="display: none" type="submit" />

Stupid. Dirty. Hack.

***Update: This, of course, doesn’t work on shitty IE browsers. They do not like the style=”display: none” and probably do something “clever” like optimize it into non-existence. They will, however, accept it if you use a style=”width:0px;height:0px;border:none”.

Even more stupid and dirty and hackish. But it’s a Microsoft product, what do you expect?

ruby 1.8.5 not handling SIGTERM

I’ve deployed some applications using the nifty runit project because it offers very nice mechanisms for process supervision. I’ve been using it flawlessly in several environments for at least a year… until just the other day when I was sure I had found a problem with it. My investigation of sv term handling with a slow child process is available on the runit mailing list.

The problem I was experiencing was with a ruby application running under runit and not honoring SIGTERMs. Looking at the 1.8.0 runit source via `apt-get source runit` on debian sid, I learned more about how runit’s pieces work together to manage and supervise child processes. Riddling the code with debug info, I was able to trace the offending code to the sv.c source file in the control() function. A snippet of code was disallowing writing of another command to a pipe for later processing because the process was already in a particular state. In my case, runit thought the process was already down because I had previously sent it a TERM signal. Unfortunately, my process never handled the original TERM because of delayed registration of the handler and therefore never would receive another from runit.

This repro’d 100% of the time on ruby 1.8.5 with this snippet of code:

puts "doing initial sleep for 10..."
sleep(10)

puts "registering term handler..."
trap("TERM") do
  puts "got term"
  exit
end

while(true) do
  puts "in loop, sleeping for 2..."
  sleep 2
end

Here’s where it gets a little messy…

As I outline in my post to the mailing list, runit does indeed refrain from sending a TERM signal again to the process, but the real offender was ruby v1.8.5. The Google led me to a post on ruby-forum about a signal handling behavior change on an upgrade to ruby v1.8.5. Matsumoto chimes in and indicates that this seemingly related(?) issue is indeed a bug.

I tried a most base case running a ruby script consisting of ’sleep 30′. Running this under:
rwoodrum@fs1sea:~/tmp$ ruby -v
ruby 1.8.5 (2006-08-25) [i486-linux]

and subsequently sending the process a TERM does nothing. A la this annotated strace:
..... snip .....
..... we received a TERM .....
) = ? ERESTARTNOHAND (To be restarted)
--- SIGTERM (Terminated) @ 0 (0) ---
sigreturn() = ? (mask now [])
select(0, NULL, NULL, NULL, {12, 476000}

..... riding out the sleep .....

) = 0 (Timeout)
time(NULL) = 1200560526
sigprocmask(SIG_BLOCK, NULL, []) = 0
sigprocmask(SIG_BLOCK, NULL, []) = 0
rt_sigaction(SIGINT, {SIG_DFL}, {0xb7f180d0, [], 0}, 8) = 0
exit_group(0) = ?

I believe at this point that the SIGTERM handler is going to be SIG_DFL. I don’t understand why this doesn’t perk up and terminate the process however, so that is still somewhat unsolved. This doesn’t repro with a more recent version of ruby.

Lessons learned? The version of Ruby1.8 in debian etch (package ruby1.8 version 1.8.5-4etch1) does not appear to handle SIGTERM correctly, even in cases where a signal handler is registered.

submit a sortable_element in a form

Today I needed to use some of Scriptaculous’ nifty sortable_elements inside of a larger form for a rails app I’m working on. The way I did it works, but it is doubtlessly a terrible hack. Whatever.

Being a complete javascript newb, I took my cue from this post on the rails mailing list which unfortunately, as the guy requested, did not have the answer bundled up nice and neatly for me. On the other hand, it was pretty easy to do by updating a hidden field with the sortable_elements to get the data submitted with the form. Onward with an example…

In my particular case, I have two lists which I drag and drop between with sort order being important in the destination list. These sortable elements are part of a form which the user will post to the app. In my form rhtml template, I have a submit_tag like so:
<%= submit_tag("Add", :onclick => 'sortableToHidden()') %>

The layout for the template in question has a small snippet of javascript:

function sortableToHidden() {
  document.my_form_name.sortable_hidden.value = \\
    Sortable.sequence('my_sortable_list_id');
}

Where:

  • ‘my_form_name’ is the name of the form within which I want to submit the data
  • ’sortable_hidden’ is the name the hidden field in the form
  • ‘my_sortable_list_id’ is the id given to my sortable list in the sortable_element helper

That’s pretty much it. You can then easily access your sorted list via params[:sortable_hidden].

I’m sure there’s some much more nifty, magical way of doing this (which I would love to hear about) but this gets the job done.

sortable lists and an intransigent dropOnEmpty

I spent a good deal of today banging my head against some scriptaculous sortable lists thingies to no avail. I seemed to have hit this snag where if I have a sortable list that either starts or later becomes empty, I can no longer drag anything to it. This makes me sad. Naturally I cannot share my sadness with the scriptaculous sortable lists demo because theirs works just fine. Before I caught the bus I spent a few fleeting moments inside of firebug to no avail. Web shit sucks.

I guess I’m not the only one, however, who has found that sortable dropOnEmpty isn’t behaving itself. As soon as I find the answer, which is no doubt me being somehow ignorant, I shall certainly spread the word.

Alas, my ongoing battle with Ruby and Rails will continue on the morrow.

Update: I’m not exactly sure of anything I did in particular, but this problem somehow solved itself. Here’s the relevant scriptaculous code in my previously won’t-droponempty view.

<p id="available_members_list"></p>
<p id="current_members_list"></p>
<input type="hidden" id="sortable_hidden" name="sortable_hidden"/>
<%= sortable_element(:available_members, :update => 'available_members_list', :dropOnEmpty => true,
:containment => ['available_members','current_members'],
:complete => visual_effect(:highlight, 'available_members')) %>
<%= sortable_element(:current_members, :update => 'current_members_list', :dropOnEmpty => true,
:containment => ['current_members','available_members'],
:complete => visual_effect(:highlight, 'current_members'),
:url => {:action => 'order'}) %>