When we made a commitment to Continuous Delivery, we struggled with some buzzword bingo.  The word “Continuous Delivery” actually applies to two very different strategies.  The “centralized” approach uses one code version for all developers and tests.  It is simpler and more efficient.  The “distributed” approach puts each change in a different version of the code.  is more complicated, more scalable, and it allows you to actually release every change. Read this article to learn how to use these strategies.

"Centralized" or "Single Stream Continuous Integration"

In "Centralized" or "Single Stream Continuous Integration" every developer commits changes to one version of the code – trunk, master, mainline.  A continuous integration system is constantly running automated tests on this shared version.  The developers are trained to structure their changes to avoid breaking the build.

Key concept: Early as possible integration

Important input: Automated tests

You can see the key concepts of continuous delivery at work.

* Automated tests.  You need enough automated tests to tell you immediately if something is broken.  You run them frequently (continuous integration).

* Developer responsibility.  It is the responsibility of the developers to make sure that their code doesn't break anything.  It should be releasable.  They can't just throw it over to QA and wait for bug reports.

* Feature switches.  Developers are constantly releasing code for the new features that they are working on, but they need to hide any features that aren't ready.  They put switches in the code to hide changes that aren't officially released or "unveiled".

Advantages

Centralization is efficient.  There is one test system, which is important if your test environment is complicated.  You don’t have to move code between multiple versions.   You know where to go for manual testing.  Because of its efficiency, this is a great pattern for a small team.

It relies on the principle of “as early as possible” integration and discovery of problems.  People are committing frequently, and tests are running frequently, and any conflicts between changes are discovered quickly.  This minimizes the amount of wasted programming effort and improves efficiency.

When developers are building new features or major refactoring, they use switches to hide them and prevent them from interfering with the working version.  Hiding and unveiling code is a tactic that is used in any continuous delivery process.

Single Stream Continuous Integration works well with Subversion, where it is called “active trunk”.  Subversion does a good job at synchronizing the developers onto the single stream.

Disadvantages

Centralized CD doesn’t scale very well if you have many contributors, new contributors, or distributed teams.  The more changes that you put into the shared version, the more often you will have a “broken build” where it doesn’t work, or possibly doesn’t even compile.

As you add contributors, the process becomes more stressful.  Every contributor has to be alert to a broken test or a broken build, and fix it immediately, regardless of time zone, so that the other contributors aren’t delayed.  This puts extra stress on distributed team members who have responsibilities after the end of their day.  That's why Humble doesn't recommend it for truly distributed teams.

The "early as possible" integration process limits the frequency of your releases.  Whenever you put two changes together, you expect to find problems.  It's not releasable until you have tested for the possibility of new problems.  Centralized CD well suited to a two-week release cycle that will include user acceptance testing, and batched releases.  It's really a continuous integration system, and not a continuous delivery system.  If you want to release every day, or every change, you may decide make a major leap from “as early as possible” integration to “as late as possible”.

The fix: Test and review

You an eliminate almost all of these disadvantages and go to continuous release, if est and review the incoming commits.  We will discuss this below.

Distributed or Continuous Release

Some modern online services have hundreds or thousands of developers, releasing once, twice, or 50 times per day.  They do it with a “Continuous Release” process.  In this process, each change gets tested and released independently. 

Key concept: Late as possible integration

Important input: Cloud computing with on-demand test environments

If you want to release every change, you can’t take the time to test it with every other change that a big group of developers is working on.  To solve this problem, Distributed CD makes the leap from “as early as possible” integration to “as late as possible” integration.  Each change is tested and released before you try it out with the other candidate changes. 

1) You put changes into a branch or fork for an individual or team committer.

3) You have multiple test environments.  QA acts in a consulting role, looking at a test environment if a developer needs help verifying that the change is good and desirable.

4) You release.  This is highly automated, and it may be incremental.  You are likely to be releasing some features that have switches to make them visible only to selected beta users.

Advantages

With the distributed process, you can release every change immediately, which provides a competitive advantage for online service providers.  The continuous release process reduces stress by getting fixes out more quickly, and by allowing big changes to take as much time as they need.  The distributed process can include a large number of contributors, and they can be distributed in multiple time zones.

Disadvantages

Test environments are more complicated.  There is more DevOps load to set up multiple test environments.  Developers are responsible for maintaining versions and shepherding changes through tests.  They need to know more about the test environments.

Critics often observe that developers using this approach can work a long time on branches, and then cause integration problems when they merge big batches of code.  Finding problems in these big batches of code is difficult.  If you use this approach, you should:

• Merge frequently "back" from the production version, to your development version.  So, you will do frequent integration and test in your development version, and you will not find problems when you merge "forward" into the production version.  Git has a special operation for this called "rebase", which is radical in the VCS world because it "rewrites history" to bundle your history of changes into one commit for forward delivery.
• Release changes as frequently as possible.  When you release smaller amounts of code, you have fewer places to look for problems and you will find them faster.

Gerrit-style change review – the middle ground

You can improve the scalability of the centralized process by adding a code review.  This will give you some of the simplicity of the centralized process, with some of the scalability of the distributed / continuous release process.   We like a review process where every contribution aimed at the master branch goes into a temporary branch, where it can be tested and reviewed by tests systems and other developers.  This has a lot of good effects.  First, it increases the quality and stability of the target branch.  Second, it gives you very convenient place to put “precommit” automated tests, so that your automated test system can reject any changes that don’t pass tests.   Third, it helps you bring in new contributors through reviews and comments.  Fourth, and perhaps most importantly, it improves your automated tests because you can ask for tests in the review process.

This change review workflow was originally built into Gerrit, the code contribution system used by the Google Android project and many related telecomm companies.  At Assembla, we think this type of change review is an important step for continuous delivery teams, so we implemented a streamlined version of this workflow that we call "protected branches."  The Assembla imlpementation gives you a very clean workflow, and you can add it to any git master with a few clicks.

Convergence:  How distributed starts to look like centralized

In a distributed CD process, you have an incentive to release frequent, small changes.  Why?  Because small changes are easier to debug.  If you discover a problem, you only have to look through a small amount of changed code to find it and fix it.  When problems show up in a large release, they are harder to fix and they create more stress.  So, over time, distributed, continuous release teams will move to frequent, small releases, with a trunk or master that is constantly updated, and is similar to a single centralized version.

In future articles, we’ll cover the hiding and “unveil” tactics that make it possible to smoothly sneak big changes into a continuously released system.

We printed up some "No Scrum" T-shirts for the Agile 2013 Conference.  The conference is mostly attended by Scrum coaches, so this should be a good conversation starter.  The back side promotes "Continuous Agile and Beyond Scrum Roadmap."  See you there.

Why are programmers kicking butt?  Because they are using powerful machines.  Vast data centers filled with computers do much of their work.  Ultimately, these machines will do experiments, and make data-driven decisions that replace a lot of work all the way up the chain to the CEO.

Decades ago, I heard the story that software developers (we called them programmers) would never improve their productivity, because programming is a labor intensive business, done by people, and people don’t change much over the generations.

That was a pretty good analysis from a time when software got delivered slowly.  It’s clearly wrong now.  We’re building software a lot faster now.  Small teams of 5 people are building systems that 20 years ago would have required 50 people, and 50 person teams are making talking advisory devices and self-driving cars with millions of lines of code.  Since software is eating the world, the world depends on our ability to build software faster and with bigger scale.

• The strategizing and fuzzy front end, where you decide what you want to do
• Task/issue management, team management, or project management, where you figure out who will work on what
• Code contribution, where you actually write code and evaluate changes to code
• Build, test, and deploy

In my observation, the amount of innovation and productivity improvement increases dramatically from left to right.

This picture helps us see how “agile” became a dead zone.  The typical Scrum-based agile process recommendations are all about how to organize teams and tasks.  It’s an art that may be very helpful for an individual organization, but overall, it’s advancing at a snail’s pace.  As a friend of mine said, “the last thing they came up with was the cardwall.”

However, there is a lot of innovation in the area of coding workflows.  We have new workflows for merge contribution, code review, and continuous integration, which all add up to much faster release cycles and more scalable teams.  And, when we get to build, test, and deploy methods, we find an avalanch of new capabilities from cloud computing providers.

Agile consultants and enthusiasts like to think of themselves as being important to the “C-suite” of chief managers.  They want to contribute to strategy, and make it more “agile”.  But, when they do that, they don’t have any sources of innovation.  They become the same as everyone else who is discussing management and strategy.  They want to improve productivity, but their desire for importance leads them in EXACTLY THE WRONG DIRECTION.  The world of strategy is even more static, with few innovations since the heyday of the industrial revolution 200 years ago.

The new agile techniques based on continuous delivery are tapping into a huge source of innovation by exploiting new coding and testing workflows.  This is important.  At Assembla, we believe that we can add a lot to agile just by reaching out and bringing a bunch of coding and testing innovation into the “agile” frame.  That’s why we make an integrated system that links tasks with leading edge code management, review, and test.

It seems to me that innovation and productivity improvement are closely related to the degree of mechanization.  On the left hand side, you have people talking.  On the right hand side, you have arrays of machines that run scripts to process changes, thousands of times.

So, it turns out that productivity improvement in software development works the same way as productivity improvement in car manufacturing, or coal mining.  You can improve productivity by adding machines.  And, the more machines you can add, the more you can improve.

John Henry, watch out!  This a megatrend that’s not going to stop for a long time.

The man to machine megatrend even gives us a route back to the C-suite, strategy, and planning.

On the long planning loop, you send all of your product initiatives past the C-suite to get aligned with strategy.  Your CEO and your VP of Marketing will argue about what to try next. They will have caveman-level productivity.  They’ll do the same thing that two cavemen used to do when they strategized around the fire, trying to figure out whether they’ll have better luck hunting mastodons up in the mountains, or down by the river.

On the short loop, you are using rapid development cycles to test options and gather data.  The shorter you can make this loop, the faster it will run.  It will also become more productive because you are moving to the right side of the process where you can use more machines.  

If the short loop gets good enough, you can decide more and more things with testing, and you will decide fewer and fewer things by having management discussions to decide what bets to make.  Eventually, you can get to the point where you can pass a lot of management work off to machines.

We launched Assembla to work with distributed teams, and this required a new kind of agile dev process. Scrum with its stuffy meetings clearly did not fit the teams we work with. Cloud-based teams need to work differently.  We want to:

• Release more frequently
• Respond faster to users and customers
• Scale to larger projects and teams
• Manage distributed teams and tap global sources of talent
• Compete better, especially with fast-cycle SaaS, Web, mobile, and big data projects
• Reduce or eliminate the stress surrounding fixes and releases

The winner in this voyage of discovery is continuous planning and continuous delivery. The pace of competition on the Internet is so fast that all of the winners will eventually need to go continuous. In recent months we have talked to many companies that want to release more frequently, or continuously.

Continuous Agile combines traditional Kanban and Lean task management with recent innovations in Continuous Delivery.  It is is compatible with existing Scrum teams, but also works for modern, cloud-based teams and leading edge online services.

Our Continuous Agile initiative gathers together ideas, tools, and services to help you (and us) release more frequently and innovate faster. 

Ideas: Unblock! A Guide to the New Continuous Agile

Please check out an early draft of the ebook. We use this material to answer real questions from teams that are trying to release more frequently. We update it as we find new questions and ideas. You will find a VARIETY of approaches that you can use. We started building a guide that would cover basic task management, code contribution, and testing, so you would have a practical guide to the previously mysterious art of releasing every change. As we got into it, we found that some of the most interesting discoveries were around product management, which is the true route to success.

Co-authors contributed valuable knowledge. Michael Chletsos, Assembla CTO, contributed the big idea of “distributed” versus “centralized” flavors of CD.  He also put together the detailed workflows that allow us to release 250 times per month. Luca Milanesio sends us his innovations for building mobile apps. Damon Poole, now Chief Agilist at Eliassen and formerly the founder of Accurev, introduced me to Kanban and continuous release ideas three years ago, and is now figuring out how to apply them in large enterprises.

We’re moving some Continuous Agile ideas to AndySingleton.com. We will expand our coverage of the big picture with more contributors and partners. Blog.assembla.com will be more focused on the product with “what’s new” and “how to get more out of it” for users. So, add your bookmarks.

Tools: The Assembla Renzoku Release

The Renzoku package helps you plan, code, and release changes with one-piece-flow. It includes planning, Kanban view, code contribution, code review, and reporting. All of these activities are linked together on a ticket with a complete discussion and activity stream. Here’s an expanding list of relevant blog articles that describe the features. Please write in if you have questions about how to set up a continuous process.

Services: Hands-on Help

We’re expanding our “Release More Frequently” workshops, where we introduce your team to Continuous Agile, and help them figure out how to release more frequently.

Learn more at ContinuousAgile.com, and check back soon on this blog for announcements of public workshops online and in Boston.

Using FriendCode to edit Assembla Git repositories online is really easy with their recent Assembla integration. You and your team members can simply work on a clone of your Assembla repository, test the code and then push back to Assembla. 

1. Sign up for FriendCo.de

You can easily sign up at www.friendco.de/subscribe and log in with your Assembla account. 

2. Import a Repository to Work On

Visit the repository settings page of your FriendCode account and select the 'Import' button for the repository you would like to import.

Note: If you already had a login to FriendCode and did not sign up via Assembla, you will have to connect your Assembla account on this page before you can import individual Assembla repositories. 

After clicking 'Import', a modal will guide you through the import process. You can change the name, the description, and choose to import as a private repository. Note: You must not have any spaces in the name to successfully Fork. 

3. Edit and Commit

FriendCode will now display the page of your newly imported repository, simply select the branch you want to work on (for exemple “master” or “dev”). You can also create new branches off other base branches.

FriendCode has now created a full workspace for you to edit that branch : you can open files, edit them, save changes, and commit your changes.

4. Push to Assembla

When you are ready to push to Assembla, open the left panel and select the deployment solution “Push upstream” which will push your new commits to the original repository on Assembla.

A logs tab will open to show you the result:

And you will see the commit in your Assembla repository:

Happy coding!

I will be leading a webinar on Tuesday, September 10 to show some of our material on Continuous Agile and help you release more frequently.

Webinar on Tuesday, September 10

• Noon-1:00 pm EDT
• 9:00-10:00 am PDT
• 16:00-17:00 UTC

Register here.

Continuous delivery is taking over the software business.  Our master plan to get continuous is simple: 1) Release more frequently;  2) Improve.  Everyone who watches this should get some hints to help release more frequently.

I will tell story of how Assembla moved from one release a month to ten or more per day, and touch on some other stories from Google, outsourced development, and mobile.  I will review our ebook and other references, so you can find the relevant material.  Then, I hope to take some questions and work through some specific issues.  I have been doing this type of workshop in person with software teams and learning a lot.

• Coding workflows
• Streamline testing and deployment
• "Unveil" new features to selected audiences
• Blend individual contributors and Scrum teams
• Change the roles of developers, product managers, QA staff and DevOps
• Reduce stress for the development team

As a CloudBees Technology Partner, connecting your Assembla Git & SVN repositories with Jenkins CI via the CloudBees Platform is pretty straightforward. The video below or verified CloudBees documentation will walk you through the setup.

By utilizing Jenkins CI in the CloudBees DEV@cloud, you eliminate the need for additional infrastructure and administration. Simply sign up online and configure. Once up and running, the build possibilities are endless. Happy coding!

In this recorded webinar, Andy Singleton, CEO of Assembla, describes key techniques for implementing Continuous Agile and releasing software every day.

Topics covered include:

• Gaining a competitive advantage by releasing continuously 
• Task management as it changes from Scrum to Scrumban to Kanban
• Centralized vs distributed continuous delivery
• Test layering, product switches and "unveiling" new features
• The changing roles of product managers, developers, QA and senior managers
• Questions and answers - Scroll to 25:00 to listen to Q&A

We have improved the Compare Report for Continuous Agile use cases. The Compare Report can be found via the "Compare" subtab of you repository. It can show you recent changes (for instance, changes that you just released), or it can show you changes between two branches or forks (for instance, changes that you want to submit for merge).

Continuous Use Cases

In continuous workflows it is very common to use the master branch as the "production" branch. Here are three common actions that we recently made very easy to accomplish.
 

See what was recently released

If you are using tags, it’s very easy to see what your team has recently released - just click on recent tags and you will see what your team has merged to master recently. If you want to see a more fine grained view, go to the commits tab and scroll through the commits to find the one you want.

Protip: Select the most recent tag to see if your team is really following your process and keeping master in production (if they are, there should be no changes).
 

See released tickets:

You will also see a subtab that links to the affected tickets list. This can be used as automated release notes that show you what actually got out. Our CEO Andy loves this feature.

Create merge request after seeing changes:

Simply select your branch and a new merge request button will appear. If you want to create a merge request against master, just click “Create Merge Request”, fill in description and send the request!

We hope you like our freshly improved compare report. Let us know in the comments!

To get a free Assembla Renzoku plan with repository hosting and the Compare Report, sign up here.

Every Project Manager wants to reduce waste on their projects. The problem usually lies in detecting which parts of the process are ineffective. We are introducing the Cycle Time Report to help you deal with this exact issue.

End To End Lead Time

First of all, the End To End Lead Time is there to show you the overall pace of your team. It is the median time in which the ticket goes from being created to being closed. This includes all the steps in your process - analysis, planning, delivery, etc. This metric is great for strategic planning and the overall overview on the team's efficiency.
 

Delivery Lead Time

Delivery Lead Time helps you drill down to only the delivery step. You are doing analysis and planning while the ticket is in the Assembla Planners Backlog Milestone and moving it to Current Milestone when it is ready to be worked on. Delivery Lead Time will show you exactly how much time a ticket spends in your delivery pipeline. This metric (when compared to End To End Lead Time) will show you how effective you are at planning and analysis.
 

Cycle Time

Analyze the delivery step further with Cycle Time. This will eliminate all the waiting. In a pull system, tickets in status "New" are waiting until a developer picks it up and works on it. Cycle Time will show you how much time a ticket spends in every other status, waiting in "New" eliminated. This metric (when compared to Delivery Lead Time) will show you how much waiting you have in your Current pipeline and which steps on the process are the slowest.

For each of these metrics, we show the median value and the standard deviation. If your standard deviation is high, you can not trust the median. First step in optimizing your process will usually be making sure that you can get standard deviation as low as possible. Most common reason for this is tickets being too different in development effort needed. Try to break down bigger tasks into smaller ones.
 

Quality

What is the relative cost of a software bug fix? Some sources state up to 150-fold increases in costs of fixing bugs in latest steps of the pipeline. Time Cycle Report will show you the quality percentage in your tickets. 85% means, that 85 out of 100 of your tickets go through your pipeline without any hiccups - developers do not throw the issue back to analysis, Quality Assurance does not find bugs, etc. Any status regression in the ticket will decrease the quality. This is the best place to start with the Time Cycle Report.
 

Use Cases for the Time Cycle Report

This is far from everything an insightful manager will do with this report, but it might give you some ideas:

• Reduce disruption that impacts efficiency, quality, and value - look at what parts of the process are slowest and ask yourself why that is

• Create Service Level Agreements - agree with your team that tickets should be delivered on your prefered time scale and easily monitor your team's success

• Continuously improve - track the overall Quality rating of your tickets. This is one of the easiest ways to ensure that you are improving

You can find the Time Cycle Report in the Metrics subtab of your Tickets Tool. Share the insights that you have got from the Time Cycle Report in the comments!

This report is available with the Renzoku feature pack. Subscribe for your free project here!

Today, Assembla rolled out a new tool for quickly creating and sharing code snippets with team members. The new Snippets tool supports a variety of languages, provides syntax highlighting, and allows team members to comment and interact with the snippets.
 

Airbrake.io recently released an integration that will automatically creates Assembla tickets for errors detected via Airbrake’s monitoring and reporting. With this integration, your team can quickly begin correcting issues while documenting the progress and status in an Assembla ticket.

To learn how to connect your Airbrake account with an Assembla project’s ticket tool, check out the documentation here.

If you are not familiar with Airbrake, they are one of the leading exception reporting services available for tracking application errors. Their reports tell you what errors are happening, what bit of code are responsible, and allow you to recreate the error for rapid debugging. At Assembla, we use and love Airbrake!

Back in May, Assembla and Airbrake put on a joint webinar on “Production Monitoring: The Key Steps Towards Continuous Delivery.” To learn more on how and why Production Monitoring is the most important process in any application, but particularly online applications when practicing Continuous Delivery, check out the recorded webinar.

Last month we released a new Snippets tool that allows you to easily create syntax highlighted code snippets to share with team members. While the tool has been widely accepted, many users asked for the ability to insert these code snippets inline with wiki pages, tickets, messages, etc. We listened and just released this updated functionality. Here is how it works:

Create the snippet and copy the markup reference (see below).

Paste the markup reference into any wiki page, ticket description, ticket comment, merge request comment, message, etc.

Save and your snippet will now show inline. Any changes to the main snippet will reflect wher ever it is embedded.

If you do not have the Snippets tool installed, check out the release announcement for instructions on how to install it. 

We hope you enjoy this new feature. Happy coding!

Disclaimer: TimeLoggr is a third party Chrome extension that is not affiliated with or maintained by Assembla in any way.

TimeLogger was recently brought to my attention and since a lot of Assembla users track their time in tickets, I thought it would be helpful to provide a walkthrough for this useful tool.

TimeLogger is a free Chrome extension available in the Chrome Web Store - click here to install. Once installed, you will see a clock icon in your Chrome extensions bar. The extension will recognize your Assembla session, so you must be logged in to Assembla for the extension to work.

Clicking on the extension will display an expandable list of your projects and tasks you are assigned within each project.

A. Clicking on your profile picture or name will open Assembla’s view of recent time entries across all projects. This view also allows you to manually input time entries.

B. The orange tasks box will display how many tickets you are assigned. Clicking this area will open your active tickets view within Assembla, displaying all tickets that are assigned to you across all projects.

C. You can click on any tasks to open the Assembla ticket in a new browser tab.

D. From the extension, you can click on the ‘Start’ button for any task and to begin tracking time associated with that tasks. Once you have started tracking time for a task, the view will change:

You can comment on the ticket from this view but it is important to note that entering a comment and clicking the ‘Stop’ button does not add the comment. To add a comment, you must type the comment and hit Enter on your keyboard.

At anytime, you can click the “Stop” button and your time entry will be created and associated with the ticket.

If you track time in Assembla, the TimeLoggr Chrome extension is a simple, handy tool that can save you some time. Thank you optini for developing this extension and making it available for everyone to use. 

On Thursday, November 7, Andy Singleton, CEO and Founder of Assembla, will be presenting a live webcast for the Product Development and Management Association (PDMA.org): “How Lean Agile Changes the Product Manager’s Job.”

Watch the video introduction below. For more information and registration, go to the PDMA web site. There is a registration fee.

Happy New Year from the Assembla team! As we start another year, we wanted to reflect on some improvements and announcements from 2013 that can help you and your team work smarter and faster in 2014.

Happy New Year and thank you for being a part of Assembla in 2013. We are very excited about 2014 and continuing to improve the Assembla experience. If you have any specific requests, please let us know on our feedback site. 

Thanks,
- The Assembla Team

TimeCamp provides fully automated time tracking and invoicing software as a service for individuals and companies. When they recently reached out to me about their new Assembla integration, I figured I would give it a whirl. Below shows how to setup and use this integration based on my experience.

It is important to note that this is a one-way integration meaning once you enable the Assembla integration, you can begin tracking time expenditures for Assembla tasks in TimeCamp, but these time expenditures do not post back to your tasks inside of Assembla.

To get started, you will need to create a TimeCamp account, login, and visit ‘Settings’ > ‘Add-ons & Integrations.’

Scroll down and click on the Assembla option. Select ‘Enable the Integration’ and you will be directed to Assembla to ‘Allow’ TimeCamp to access your projects and pull your tasks. You will then be redirected back to TimeCamp where you can select the projects you would like to import and optionally invite Assembla team members to your TimeCamp account.

Once the integration is setup, you can begin tracking time to Assembla tasks. From the main dashboard, you will select ‘Track time for [day]’ and be able to see and select tasks from your Assembla projects.

When you select a task, you can either manually enter time or click the play button to start the timer.

TimeCamp also has a desktop application that when installed, allows you track time to tasks from your desktop. By default, the desktop application is set with Tracking Mode on Auto which is supposed to automatically search for keywords in window titles, urls and application names and track time accordingly. For complete accuracy and control, I personally prefered switching the Tracking Mode to Manual so I could manually select the desired task.

After you start tracking time with TimpCamp, you can explore additional features such as computer activity tracking, reporting, and invoicing. To learn more about TimeCamp and sign up for a free trial, visit their website at www.TimeCamp.com.

After a massive weight loss, our database feels much better and our operations team is happier.

Until recently, Assembla supported 2.3 million wiki pages in our main database. This required about 30GB of data, prevented the database from fitting in the memory of the database server, and frequently affected the performance of all database transactions.

The arrangement was also extremely wasteful, because every page was included in full, even if it had only minor differences from a previous version. If a user changed a one character on a 1,000-byte page, the revised version would also require 1,000 bytes of storage, for a total of 2,000 bytes.

We realized that migrating the wiki pages to Git repositories would free up the 30GB in the database and speed up performance.

First try

Since we already have an established process for communicating with remote Git repositories from our web app using the BERT-RPC (http://bert-rpc.org), we decided to see if that would be a good way to carry out the migration.

However, initial results were not promising. The process was very slow, because every Ruby worker is single threaded. Also, every commit to a bare Git repository, and every blob retrieved from a repository, required about 5 calls to a Git binary, which slowed performance even more.

Git-ting it right

Our revised plan was to write our own RPC server using the Go programming language (http://golang.org), Git to go (https://github.com/libgit2/git2go) and libgit2 (http://libgit2.github.com). We also used MessagePack (http://msgpack.org) serialization format and its RPC capabilities.

With this RPC server on our development machine, the Go daemon was able to handle thousands of requests per second using a single OS thread (the default value for GOMAXPROCS). Increasing the value of GOMAXPROCS increased throughput even more.

The migration process

We started by migrating the biggest wikis, some of which had hundreds of thousands of versions of the same page, and which totaled gigabytes of information. After running “git gc” compression, the biggest wiki was down to 98 MB.

However, InnoDB doesn’t release space after content is deleted. Therefore we did a complete database reload by dumping the database, removing the InnoDB data files, then restoring everything back. Our operations team was able to do this without any downtime.

Healthier and happier

After this weight loss program our database is healthier and the Assembla operations team is happier. The new RPC Go daemon is a success, the database is no longer a resource hog, CPU and RAM consumption are much better, the team receives much less alerts than before, and our customers have fast access to their wiki pages.

There are now three ways to attach files in Assembla - from your computer, via Google Docs, and now via Dropbox. The new integration conveniently allows you to select and share Dropbox files with fellow team members, and since Dropbox takes care of versioning, the most recent version will always be available to your team.

In this release, the Dropbox functionality is only available in the Files tool and when attaching files to created tickets. We are actively working on expanding the functionality system-wide. Until then, here is how you can start using the current dropbox integration:

1. Select ‘Add Dropbox File’ from the list of file controls to expose the Dropbox ‘Choose File’ button.

2. When you click on ‘Choose File,’ you will get a Dropbox pop-up. If you are logged into Dropbox, you will automatically see your dropbox folders and files. If you are not logged into Dropbox, you will be prompted to login. Once your folders and files are in front of you, select the desired file from the pop-up and click ‘Choose.’

3. The selected file will be displayed. You can optionally add a short description and/or labels. Click ‘Upload File’ and team members will now be able to access the shared Dropbox file.

If you have any questions, comments, or suggestions, please leave them in the comments below. Stay tuned for more improvements coming your way soon.

On Friday and Saturday, many of our users experienced 504 error and dropped connections.  We apologize for these annoying events.  We have a plan to make sure that it does not happen again.  Thank you for your patience.

What happened?

 We were overloaded.  We got too many requests for data from Subversion, and our firewall and proxy servers were overloaded.  Load spikes from the repository servers disrupted both repository traffic, and Web traffic.  The load came from game updates.  Some of our users are hosting game content, and on Friday night, when a lot of gamers start up, thousands of them were pulling updates from Assembla Subversion servers.

What are we doing?

• Rate limiting our repository servers, so that the servers in our private, high-availability datacenter cannot become overloaded in this way.
• Organizing our proxy servers so that repository problems cannot affect the Web app.
• Adding a high-throughput server in a public cloud datacenter, so that we can host games and other high-throughput applications.