Shrink docker’s VirtualBox machine for mac

Over time, the boot2docker virtual machine image disk.vmdk will consume more and more disk space as each disk sector is written to for the first time (up until its maximum virtual disk size or the host system runs out of resources).

There are a couple of approaches to reduce the size of the virtual box image. First approach is to delete the virtual machine and then reinstall it. Alternatively you can shrink the virtual machine disk.

This guide demonstrates how to shrink the boot2docker virtual machine image. It is intended for reference only and you should always make sure you have appropriate backups first before attempting to do this.

Before you begin

Docker can leave being dangling images when using the docker commands. These images build up taking consume additional space, so before we begin, lets remove all the containers that are in an exited state as well as any dangling images.

Shrinking the VirtualBox image

This approach will take the .vmdk VirtualBox image, convert it into a .vdi image, compact it and then convert it back to .vmdk. Sadly there are no tools included within VirtualBox to compact a .vmdk image directly

The drawback with the approach is that you need to have enough free disk space on the host machine to clone the image. If you do not have enough free disk space then you will need to use external storage for storing the temporary .vdi image.

This guide assumes your docker virtual machine is called “default”.

  1. Log into the the docker machine and execute a command to fill all the free space on the disk with 0’s.


    This creates a file zero.fill in /mnt/sd1/tmp and the current working directory. The purpose of this file is to force all the data bits in available free space to be reset to 0. By doing this, VirtualBox will be able to compact that space thus reducing the boot2docker virtual machine size.

    Note: You will receive the error message cat: write error: No space left on device that the device has run out of space, this is expected.

    Exit the docker machine host.

  2. Back on the host system, stop the running boot2docker virtual machine so the image can be compacted.

  3. Navigate to the directory where your docker machine is kept.

  4. Convert the .vmdk image to .vdi using the clone command


    – If you do not have enough disk space on your machine, consider using external storage and change any reference of disk.vdi to /Volumes/<external-drive-name/disk.vdi.
    – This will take several minutes. Time to make a cup of tea.

  5. Compact the .vdi image

  6. Get the UUID for the original disk.vmdk disk image, and write it down.


    ! Make a note of the UUID
    This will be needed later when converting the .vdi image back into a .vmdk image.

  7. Remove to original disk.vmdk

  8. Convert the .vdi image back into a .vmdk image

  9. Reset the UUID of the new disk.vmdk image to be the same as the original disk.vmdk UUID

  10. Restart the docker virtual machine

  11. Lastly, remove the disk.vdi file as it is no longer needed.

    All done.

    Checking the size of the virtual disk, it’s now only 3.4GB. That’s almost 85% smaller! Over time it will increase again and you’ll need to go through this same routine.

One liner to parse a JavaScript string into a boolean

This ES6 one liner is a handy little JavaScript method for converting a string value into a boolean:

 

This maps values such as ‘true’, ‘TRUE’ ‘on’, ‘1’, ‘-1’ to true, whilst values such as undefined, null, ‘false’, ‘FALSE’, ‘0’, ‘off’ are treated as false.

View Demo

Here’s a TypeScript version of the same thing:-

 

and just for completeness, below is the ES5 version

 

How to find out which node process is listening on port 3000?

Every now and then it seems that some process will be listening in on the port that another process is trying to use.

This terminal command quickly allows you to check which process is listening to the given port on either Mac OSX or Linux such as Ubuntu.

In this example, we can see that the node process 5557 is listening on port 3000.

To terminate the process, find the PID e.g. 5557 and use the kill command:-

 

 

 

Countdown timer using RxJS

Creating a countdown timer with RxJS is pretty simple. The block of code below creates an Observable that when it receives a truthy on the input stream, the countdown will commence emitting the next value in the sequence on every interval until 0 is reached.

Any further truthy values on the input stream will restart the countdown whilst falsey values will cancel the countdown.

Times are based in milliseconds.

Breaking it down

The magic in this stream occurs within the switchMap. When the input stream sends through a truthy value, an Rx timer starts counting incrementing based on the defined interval. This timer continues until the duration has been reached, plus one (to allow the countdown to reach 0 later) e.g. [0, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000].

If a falsey value comes through the input stream then switchMap will swap out the timer for an Obversable that will never emit, thus cancelling the countdown.

Finally, the map function inverts the direction of the counting timer so that we get a countdown e.g. [10000, 9000, 8000, 7000, 6000, 5000, 4000, 3000, 2000, 1000, 0].

Example

See the example on jsbin.com demonstrating the use of the stream to launch a rocket.
https://jsbin.com/sibacav/edit?js,output

 

The code is written in TypeScript but is easily convertible to JavaScript.

Git: Node Pre-commit Hooks

Background

Git hooks allow scripts to be fired off when important events such as committing a change to the repository occur. This guide details how to set up a pre-commit hook for a node application that will execute a testing task that must pass before any new commits are added into the repository.

Why?

Pre-commit hooks allow house-keeping tasks such as code linting and running tests to execute before the commit is added to the repository. If any of these tasks fail then the commit is rejected and not added into the repository. Doing this helps enforce code integrity and prevents simple easy to catch mistakes from creeping in. Of course, any other other kind of tasks could be run like emailing your boss that you’re about to commit, and there are other hooks that can be used for post-commits.

Set up

For this guide, I’ve set up a very simple node command line calculator application that can be cloned and used to follow along with the rest of the article.

The app takes exactly two whole number parameters and sums them together to output the result in the console.

This app has a small test suite that can be run by executing the following command.

Currently, all the tests are passing.

New feature request

Scenario: Your boss comes over to you and says “that calculator app is great… but I urgently need to be able to subtract numbers as well as add. Make it accept an operator as a parameter”.

So the new feature is loosely defined, we could go rushing ahead make changes to the code and commit it immediately. But what about the tests? Well they would be broken because in our haste we forgot to run the tests before committing. Now other team members are upset with us and the build server is sending you emails that you have broken everything. Not good.

If only the tests had been run automatically before the commit was made. And that’s where pre-commit hooks come in.

Adding pre-commit hooks

First set up the git pre-commit hook that will enforce running our tests and making sure they pass before committing to the repository.

The node package “pre-commit” allows the defining of tasks in package.json that will be run before making the commit. If any of these tasks return an error code then the commit will be aborted.

To install:-

Installing pre-commit will add a new task called “pre-commit” into the hidden .git/hooks folder located under the root of the project. It is this script that git will execute before committing to the repository. If the script returns a non-zero value, then the commit will be aborted.

To configure a test task to execute on pre-commit, add the following block into package.json:-

silent reduces the amount of noise that pre-commit logs out to the terminal.

run takes an array of npm tasks to execute. These tasks are defined within the “scripts” block in the package.json file.

By default, npm test will be called if no run tasks are defined, but chances are that more tasks such as linting will be added the project so I prefer explicitly defining this upfront. Read the pre-commit documentation for more information.

That’s it. Now whenever anybody commits changes to the repository, the tests will execute. If the tests return a non-zero result, the commit will be abandoned with an error message.

Adding the feature

To add the subtract feature to the code, I’ve created a gist that you can copy the changes from. Replacing the contents of calc.js with this gist will add the subtract function into the program.

Now the user needs to enter 3 arguments rather than two to operate this simple app.

Giff diff report showing the changes made to calc.js to add the subtract feature
Giff diff report showing the changes made to calc.js to add the subtract feature

In the terminal, the user can now execute the following command to add or subtract two numbers:-

Terminal output showing program running addition and subtraction operations.
Terminal output shows that the calc app now can add and subtract numbers together

Great! The changes work, feature complete. Time to commit.

Git prehook commit failure message
Terminal output showing the broken test caused by adding the new feature in.

The commit was rejected because the tests are no longer passing. Excellent, the tests were automatically run and have stopped the feature from accidentally being committed to the code base.

Now all that remains to be done is to fix the existing tests and add a new test for the new subtract feature, then commit. Check out the subtract-feature branch on the repo for the solution.

Git pre hook success message
Terminal output showing that are tests are now passing before making the commit

Further reading

You can find more information about git hooks on https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks

Recipe: How to create a self-signed SSL certificate Ubuntu + Nginx

RECIPE: Create a self-signed SSL certificate for Ubuntu and Nginx website.

  1. Create a directory to place the ssl certificates.
  2. Generate the ssl certificate replacing [CERTIFICATE_NAME] with a valid file name.

    Example:-
  3. Reference the certificate inside the server block in the nginx site config file.
  4. Go to your site in a browser e.g. https://example.com/ and you should get a browser warning message saying the site is unsafe because the certificate authority i.e. you, is not trusted by your computer. Trust the certificate to view your site over SSL.

    Untrusted connection warning message
    An example of the kind of warning messages browsers give when using self-signed certificates

Pretty git log in the terminal

It’s easy to get a graphical git log tree view in the terminal like this:-

Pretty git log screenshot display the este repository commit tree
Pretty git log screenshot display the este repository commit tree

Running the git log command below inside your git project produces similar output to that shown above:-

Using a git config alias, the command can be shortened to make it easier type in the terminal by defining an alias e.g. “tree” (or whatever you want to call it) in the global config.

Now to get the pretty git log tree view, simply type:-

Debugging Performance Bottlenecks in a Node.js Application

Background

A recent update to one of our high volume production apps had the undesired consequence of increasing the time-to-first-byte by a staggering 500%. Although we had done rigorous testing, this bug did not clearly manifest itself until it was in production. To make matters worse, the application is isomorphic and we had just updated from React v0.13 to v0.14 refactoring components along the way so there was not one single change we could easily pinpoint as being the most probable cause. Tasked with identifying the problematic code, I began CPU profiling the JavaScript node application using the readily available tool – node-inspector.

This guide aims to demonstrate how to diagnose bottlenecks in your node application using node-inspector and Google Chrome.

Sample project

I’ve put together a very basic express app that calls off to a fake service which adds a delay of ~1 second (depending on your machine) to the response from the server. The code can be cloned from github:-

Quick start

To start the server, execute the following from the application root:-

The server should then output the following response:-

Node performance debug site server listening console output
Server ready and listening

Identifying the problem

Next, open up Chrome and with debugging tools enabled and focused on the network tab, navigate to http://localhost:3000/

Network trace for the node performance debug site showing a 1 second delay for time-to-first-byte
Network trace of our sample node application showing a delay of over a 1 second for time-to-first-byte.

We can clearly see that something is not right; over 1 second to get a response is too slow. So how can we diagnose what’s causing the delay?

CPU profilling Node.js

Node inspector Can help us to identify bottlenecks in the execution of the node application by profiling the code execution.

First install node-inspector globally (you may need to run as sudo),

then execute the node application using node-debug instead of node.

This will launch your application in node-inspector and open a Chrome debugger window.

Application console.log running under node-inspector
Application console.log running under node-inspector

If your browser does not automatically load, then open up a new browser instance and navigate to the debug url displayed in the console e.g. http://127.0.0.1:8080/?ws=127.0.0.1:8080&port=5858

node-inspector Chrome debug window
node-inspector Chrome debugger attached to the process and paused at the first line of code

By default, node inspector will pause execution of the application at the very first line of code.

Allow code execution to continue by pressing F8 or clicking the resume execution button.

Recording the CPU Profile

To record the JavaScript CPU profile, we need to instruct the node inspector debugging tools to record the CPU profiles, then in another browser window visit our site to generate a request that will appear in the CPU profile.

Step 1. Profiles tab

Start CPU profiling in node inspector
Start CPU profiling in node inspector

Step 2. Make sure the “Collect JavaScript CPU Profile” option has been selected.

Step 3. Click start.

Step 4. In a separate browser, navigate to http://localhost:3000/ and wait for the request to complete.

Step 5. Back in node inspector, click “Stop”.

Stop recording of the CPU profile in node inspector
Stop CPU profile recording in node inspector

6. The profile will automatically load.

Analysing the CPU profile graph

The CPU profile chart helps to easily identify where our application is spending its time whilst processing the request.

1. Make sure the chart view is selected.

Annotated node inspector CPU profile trace
The node inspector CPU profile trace – Note that the duration of the call time has increased from 1 second to almost 1 minute. Running with a debugger attached slows down the application significantly.

2. The graph shows the activity over time. We’re interested in this big block.

3. The call stack starting from top to bottom.

Looking at the call stack, we can quickly identify where our application is spending most of its time processing the request.

Using the mouse to hover over each method call, we can see that it looks like the culprit lies in slowService.js, around line 14. Armed with the knowledge let’s go to the code and see what’s going on.

So the method that’s taking a lot of time to execute is appropriately named cpuIntensiveProcess.

Clicking on the block with the name “service” will directly take you through to the source code whilst inside node inspector.

Node inspector source view of slowService.js
Viewing the problematic file in node inspector

In our sample application we can now refactor the cpuIntensiveProcess method out to be more performant, or in this example just delete it entirely because it’s not doing anything useful.

Conclusion

It is relatively pain free to troubleshoot performance problems in our application. Node-inspector is just one of a suite of tools available to help us monitor, report and diagnose performance problems.

In our production environment, we use newrelic to alert if there is ever a significant degradation in time-to-first-byte performance, as well as capturing server and client side errors giving better insights into how our applications behave in the real world.

Fixing node package vulnerabilities

It’s always a good idea to keep on top of your security updates because if you’re not looking at it, someone else probably will.

Acting on a recent denial of service vulnerability security alert from Node.js, I was tasked with upgrading our production servers to the latest Node.js version to fix the security alert. This was pretty straight forward upgrade only involving changing a version number in a config file then rerunning the build / production pipelines in teamcity to pull in the latest Node.js.

Following on from this I wanted to check if our applications themselves contained any security vulnerabilities, the only problem is that in just one project we have at least 30 top level package dependencies defined in our package.json and each referenced package references other packages and so on. Reviewing these packages by hand would take hours and is highly error prone to missing known vulnerabilities.

Thankfully there’s a tool to do all the hard work for you by scanning your node package.json and npm-shrinkwrap.json files for vulnerabilities.

nsp from nodesecurity.io audits your node application dependencies against its database of known security vulnerabilities.

To install globally:-

Then inside your project directory, where package.json is located, run this:-

Here’s a sample of the vulnerabilities found in just one of our production apps (security vulnerabilities now fixed!).

Node security vulnerability report showing 13 found vulnerabilities.
Node security vulnerability report showing 13 found vulnerabilities.

After upgrading all the affected packages and rerunning nsp check, I now get no security alerts reported:-

Node security vulnerability report showing no known vulnerabilities.
Node security vulnerability report showing no known vulnerabilities. All fixed.

Good times.

Take this further by incorporating nsp into your build processes using gulp to help stay on top of security.

Check out Node Security Project for more information.