#156 ✓resolved
Simon Rozet

Defer builds by queuing

Reported by Simon Rozet | May 16th, 2009 @ 12:36 AM

Comments and changes to this ticket

  • Simon Rozet

    Simon Rozet May 16th, 2009 @ 12:37 AM

    • State changed from “new” to “review”
  • Simon Rozet

    Simon Rozet June 29th, 2009 @ 01:21 AM

    • Assigned user cleared.
    • Title changed from “[Review] Bob::BackgroundEngines::Queued” to “Defer builds by queuing”

    Gonna rework this (and actually fully implement a queue-based strategy) later

  • Tor Erik

    Tor Erik July 2nd, 2009 @ 11:47 AM

    • Tag changed from bob to bob, engine

    I tried using a ThreadPool with 1 thread to simulate a queue, and it kind of worked, but it would occasionally stop processing and just sit there even if there was more work to do.

    Now using the attached queue engine which does the job for us without any hangs.

    Also, could someone please clarify the purpose of the thread engine. As far as I can tell, it will check out a commit, start testing it, then check out the next commit, thus ruining the first checkout, before the test has completed. How is that supposed to work?

  • Tor Erik

    Tor Erik July 2nd, 2009 @ 11:50 AM

    Attachment doesn't work? Some inline code:

    require "thread"
    
    module Bob::Engine::Queue
    
      def self.call(job)
        (@queue ||= Queue.new) << job
        work
      end
    
      def self.work
        return true if @working
        @working = true
        Thread.new do
          while job = @queue.pop
            job.call
          end
          @working = false
        end  
      end 
    end
    
  • raggi

    raggi July 2nd, 2009 @ 03:14 PM

    Can you explain where the problem existed?

    Are you sure it was in the current implementation and not called code?

    The reason I ask, is that this implementation you suggest is not thread safe.

    Also @working = false is unreachable code, unless you push a nil to the queue. Did you mean to use @queue.pop(true)?

    I'm not resistant to changing the present implementation, but reasons and causes are important. Any chance of a bug report instead?

  • Tor Erik

    Tor Erik July 2nd, 2009 @ 07:33 PM

    No, I am not 100% sure that the problem didn't lie somewhere else. I put some logging statements in and could observe that occasionally the jobs would queue up with no processing taking place. I never managed to isolate it any further than that. However, when I tried using the code I wrote above, the problem went away. Note that we don't need parallels builds, we can only do one build at the time, we just want them detached from the web interface so that it can be browsed during builds.

    So the most specifics steps to reproduce I can give (not very specific at all) would be: pushes are coming in from GitHub to Integrity. Each project build takes about 5 minutes. It works fine for maybe 10 commits or for a day, and then eventually nothing is building and the number of jobs in the queue just keeps increasing.

    Thanks for pointing out that pop never returns if the queue is empty. I don't have much experience with using threads in Ruby (or in general) and never looked at the docs for Queue.

    Also, I am still curios about the purpose of parallel builds. Is the intention to use a different check out directory for each thread or something like that?

  • raggi

    raggi July 16th, 2009 @ 10:21 AM

    Yes, it must be done that way if it's to work correctly (re checkout directory).

    The reason for parallel builds is for places that run CI for more than one project, and have more users. It's not unusual for my commercial endeavours to include 10+ repositories for a particular project.

  • raggi

    raggi July 16th, 2009 @ 10:24 AM

    The best recommendation I can make for debugging is that you set:

    Thread.abort_on_exception = true

  • raggi

    raggi July 16th, 2009 @ 11:45 AM

    Ok, think it was the lack of an exception catching mechanism.

    Will start pushing exception data to Bob.logger

  • Simon Rozet

    Simon Rozet October 21st, 2009 @ 08:18 PM

    • State changed from “review” to “resolved”

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

Automated continuous integration server that doesn't suck.

Shared Ticket Bins

People watching this ticket

Attachments

Tags

Pages