Our products

We are passionate about building niche multi-platform products that provide real utility to our customers.

Anfield
Keep up to date with Liverpool FC news from across the web.
Domain Bites
Domain name industry news and resources.
Pub Reviews
Find the best pubs near you. Share your thoughts and photos.
Mmmm
Capture your taste and share it with the world.

Blog

We like to blog about software. Our stack, engineering problems, open source developments.. you will find it all discussed here.

Ethereum - Security and wallets

Don't tell gavofyork or tomusdrw, but two days ago I robbed myself. Let me explain.

Since mid 2015 I have been aware of Ethereum. I read up, felt that it was the future, and decided to invest some of my personal savings in the project (by proxy of their Ether token).

Fast forward a year or so and rather than simply keeping 'in the know' as regards developments in the project I decided to dive deep and try and get to grips with some of the complexities of the project. Why? Simple. Double Negative (the company which I own and operate) has less on its plate, and we are considering building out web based applications that integrate with the Ethereum blockchain.

Two days ago I stormed into the Parity Gitter shouting and screaming. I was being robbed !

Having read this thread on reddit the previous day, security was at the top of my list of 'things to investigate'. Talk about synchronicity.. All of a sudden I was being asked to sign a request for a transaction which I hadn't initiated through the Parity Wallet UI.

Lets pause for a moment, and explain/outline in depth the totality of my post-resolution research. Then I will explain what had happened. Here we go..

What I learned..

NOTE I tried to make this as beginner accessible as possible, but the engineer in me got a bit out of hand. Apologies if it gets too technical.

What is an Ethereum wallet

In the physical world you interface with a physical wallet. You put fiat cash into you wallet, and take it out when you want to spend some.

In cryptocurrency land a wallet tends to refer to the platform through which you interface with your Ether.

In that vein, the following are Ether wallets:

  • MyEtherWallet - a web based wallet written in Javascript.
  • Ethereum Wallet - the wallet software created by the Ethereum foundation.
  • Parity Wallet - a web based wallet interface coupled with the Parity Ethereum client.
  • MetaMask.io - a wallet served as a chrom extension. Watch this video outlining how it works.

Behind the scenes

Behind the scenes each of these wallets is working with a key file.

  • In Parity (on macOS) these are stored in ~/.parity/keys.
  • Ethereum Wallet interfaces with your running Geth node which (on macOS) stores keys in /Library/Ethereum/keystore.

What is in a key file?

The below refers specifically to Ethereum Wallet and Parity Wallet.

As outlined in this Stack Exchange question this key file contains your private key encrypted using a (user defined) password.

From your private key is derived a public key and an address. How this is done is outlined in this fantastic answer.

Your address is the address that others use to send you Ether. The address is also contained within the key file.

Are addresses unique?

I was intrigued as to how one guarantees the uniqueness of Ethereum addresses. There is an answer for that too.

A wallet generates a random private key. There is a "2^160 or about 1 in 1,461,501,637,330,902,918,203,684,832,716,283,019,655,932,542,976" of two private keys colliding.

This comment on reddit explains this extremely well.

So basically whilst private keys can theoretically collide, in practice they don't because of PROBABILITY !! :)

Offline wallets

MyEtherWallet is wallet software written in Javascript. You can download the source code and run it on an 'air locked' device to generate a wallet offline. This refers to a device that is not connected to the Internet. This is outlined by the author here.

But how can you create a wallet offline?!

Another confusing point for me was how you can create an account without being on the Internet.

My initial naive thinking was: "If this new account is not transferred to the blockchain over the Internet, how will it be 'registered'".

Based on the above explanation of public/private key combinations (and their relation to account addresses) hopefully you can see the answer. You have created a random private key from which the account is derived. That same private key will never (probabilistically) be generated again.

As outlined in the Ethereum docs, "The Ethereum blockchain tracks the state of every account, and all state transitions on the Ethereum blockchain are transfers of value and information between accounts.".

That is to say, accounts don't need to be 'created' or 'registered' on the blockchain - all possible accounts inherently exist.

This comment provides clarity as regards this matter.

Any other tools?

There is another tool called Icebox which will allow you to generate account(s) on cold storage.

This tool creates unique Hierarchical Deterministic accounts. The best explanation of this can be found here.

Hierarchical Deterministic accounts are a number of accounts that are cryptographically related and will be deterministically created given a certain mnemonic input.

The "12 word mnemonic" is basically a passphrase. It is generated using complex cryptography outlined in BIP 39. With Icebox some additional randomness is supplied in the form of user supplied entropy.

Tell me more about BIP 39

I found some interesting stuff about BIP 39. Here you go.

Keystore formats

In principle wallet software can produce keystore files in any format that it likes, as long as it retains knowledge of how to decode that contained within into the private key.

The defined format (that is utilised by Geth and Parity) is detailed here. 'Secret Storage Definition' - how mysterious.

If one were implementing their own wallet software it would be perverse not to follow this format as it allows simple cross compatibility.

Transferring Ether

In the world of Ethereum, 'Gas' is an important concept. Here is an explanation of gas.

Transferring Ether is an example of a transaction. To execute such a transaction you need to 'sign' it. This essentially amounts to authenticating that you want the transaction to proceed.

Etherscan has a web form which allows you to post the hex code for an already signed transaction directly to the network.

This answer outlines in laymans terms that:

"Given the transaction, its signature and the signer's public key, any party can check for the transaction's authenticity. Authenticity means that only the signer could have signed the respective transaction."

Note that your private key is not required.
Essentially complex cryptography allows validation of a transactions source without knowledge of the private key that initially signed it.

This is outlined in this comment:

"This is one of the most magical things about bitcoin/crypto currency IMHO. Private/Public keypairs are "mathematical trap doors", that is, the private key, run through the crypto algorithm, always solves to the same public key. The blockchain will never see the private key, EVER! When you sign a transaction you do not show the private key, but the blockchain can cryptographically ensure that the private key had to be used to sign the transaction for that public address. This means that public/private keypairs can be created without interacting with the blockchain.".

This question demystifies the process of transaction signing.

But how does one get a signed transaction hex?

Um.. you can use Icebox on an airlocked device.

You can also use MyEtherWallet's offline transaction functionality, but I do not feel that its usage is as intuative.

One important consideration with transactions is the nonce. Nonces need to be incremented with each transaction. On an air gapped device you can't access the blockchain to discern the account nonce so must identify it manually.

Here is a post on nonces to prevent double spends.

This answer outlines what happens if you use the wrong nonce.

Demonstration

I wanted to be sure that my understanding of public/private keys, account numbers, and key stores was correct. Furthermore if you generate an offline wallet you want to be sure that the passcode or mnemonic that you have used will always generate the correct account hex. As such I did some testing. You can do the same to reassure yourself..

  1. Download Icebox.

Note: This is my own fork. Only the hdPathString has been changed as seen in this singular commit. This change was made so it matches these proposed standards that are used by Jaxx, MyEtherWallet etc.

  1. Generate some accounts (follow the README), and note your mnemonic.

  2. Note your account hex(s) somewhere.

  3. Visit MyEtherWallet.com or download their source code from their github and run it on your local machine.

  4. Navigate to the 'View Wallet Info' tab, select the 'Mnemonic phrase' option, and input your phrase.

At this point a popup should appear which displays the same account hexes as generated by Icebox.

Note There may be differences in capitalisation as Icebox does not implement Checksums.

MyEtherWallet does not generate a key store file for usage with Mist or Parity (I am not sure why..) and as such I wanted to take this further such that were one to want to they could import a wallet from Icebox to Geth/Parity.

So.. next..

  1. Setup a Parity node and run it (parity --warp).

  2. As outlined in 'Getting Started', install web3 with npm.

  3. Whilst you are at it, install keythereum with node too. Their github is here.

  4. Start a node console.

  5. Type/have a look through the following commented code snippet.

//Include and initialise web3
var Web3 = require("web3")  
var web3 = new Web3(new Web3.providers.HttpProvider("http://localhost:8545"));

//Output the current block number (to check things are working)
web3.eth.blockNumber

//Output a list of accounts. On a new Parity node this will be empty []
web3.eth.accounts

//Include the keytherum library
var keythereum = require("keythereum");

//Define some options/config 
var password = "wheethereum";  
var kdf = "pbkdf2";  
var options = {  
    kdf: "pbkdf2",
    cipher: "aes-128-ctr",
    kdfparams: {
        c: 262144,
        dklen: 32,
        prf: "hmac-sha256"
    }
};

//The unencrypted private key that MyEtherWallet displayed to us
var privateKey="0065c0e3b3da40d985e8467e5a9e2cfaa1627b55ca7d1ec2cd13a3c0a25d784f";

//Require complex cryptography library
var ecdsa = new (require("elliptic").ec)("secp256k1");

//Convert the private key to its public key counterpart
var publicKey = new Buffer(ecdsa.keyFromPrivate(privateKey).getPublic("arr"));

//Require the utility library for converting a public key to an address hex
var pubToAddress = require("ethereumjs-util").pubToAddress;

//Use it
var address = pubToAddress(publicKey).toString("hex");

//At this point you can print out the address, and verify that it matches what you started with in Icebox.

//Next I want to produce a keystore file for Mist/Parity

//I wasn't sure how to make a salt and a Initialisation Vector.
//My understanding is that they are random and play a part in the crypto security of this all.
//I use keythereums create method and steal the salt/IV from there.
//Can someone more knowledgable confirm that this is OK? :P
var dk = keythereum.create();

//I build the keystore file contents
var key = keythereum.dump(password, privateKey, dk.salt, dk.iv);

//And convert it to JSON
var jsonKey = JSON.stringify(key);  

At this point I can copy the contents of jsonKey and place it in a file in the ~/.parity/keys directory.

If I now restart my Parity node (so it reloads the keys) and execute web3.eth.accounts from my node console one should see the account hex that they generated many moons ago using Icebox.

You can also import the keystore file into MyEtherWallet, enter your password, and verify that it works there too.

So what happened?

I wasn't actually being robbed. I had previously created a script which attempts to register a contract on the blockchain.

I hadn't appreciated that I had opened this page in my browser. I had also forgotten that it used web3 to communicate with the blockchain in such a manner.

The transaction I was being asked to sign was not for all my funds (as I had initially thought in my panicked state). I didn't realise this until many hours later.

When I saw the signing request I panicked, took a screenshot, and instantly rushed to the Parity Gitter.

Gav, Tomasz, and user @ppratscher helped reassure me that I was probably secure and ran through with me what could have possibly gone wrong. I learned a lot (so it wasn't all for nothing), and am endlessly appreciative that they took the time to help out.

After many hours of security research I was closing some browser tabs and noticed some weird logs in my console. This was my Eureka moment. I investigated what had caused these messages to be outputted. I had written code (that I had forgotten about) such that failure to submit the contract resulted in the console logs. Only at this point did I think to look at the screenshot that I had taken. Looking at it with a clearer mind made it patently apparent that the transaction was for 0 ETH.

The screenshot in question

Oops.

How do I actually stay secure?

Hopefully the content of this post makes it apparent that the valuable piece of information in this puzzle is your private key.

If someone has your private key in its raw form then they can sign transactions from your account.

As such to keep yourself secure you need to make sure that no-one can access your unencrypted private key.

In principle you can hand out your encrypted private key to whoever you want and they can do absolutely nothing. If however they manage to discern your passcode or mnemonic (with which the key is encrypted), then you are at risk.

The reason that people advocate offline wallets is because no-one, and no malware can access a locked down private key. Furthermore, if you sign a transaction on an offline machine you can submit it to the network without having to type your private key into a live machine. This means that even if someone had managed to install a keylogger on your machine they'd never be able to get your private key.

Respect the sanctity of your private key.

TLDR

Some idiocy, and some panic made me think I was being robbed.

I learned a lot more about Ethereum which will hopefully help Double Negative Solutions to produce some awesome products (based on the platform) in the future.

Hopefully you can learn something from my foolishness too.

Questions/comments

If you have any questions or anything needs clarifying, please let me know.

If anything is wrong, or you feel you could improve my knowledge/understanding in any way I would really appreciate your insight.

Background tasks on Ubuntu

Across the Double Negative product line there are a number of vitally important 'behind the scenes' processes which allow our applications to function smoothly.

For example:

  • This blog is powered by the Ghost blogging platform. Ghost is a node application and as such behind the scenes it is run using npm: npm start.

  • Across our Android and iOS application lineup we utilise constantly running scripts to connect and process the sending of notifications to your Android and iOS devices.

  • There are a number of background scripts which validate and maintain data integrity across our network sites.

  • Revision.net is powered by a number of scripts which constantly poll and submit your scheduled posts to Twitter, and Facebook.

So what..?

It is very rare that I have a need to reboot our servers. Likewise the scripts running behind the scenes are fairly well tested and tend not to stop without reason. This knowledge has led to a false sense of security such that when I did need to reboot our servers it completely passed me by that I would also be taking this blog down, and stopping all of our behind the scenes processes.

Having made the mistake once I did not want to do it again, and as such I investigated how one could automatically restart all of our background processes on server reboot.

This is what I found.

Ubuntu

All of our servers are Ubuntu servers. I recently upgraded them to 16.04.1 LTS (Xenial Xerus).

One tool that I utilise heavily is screen. It allows me to have multiple terminal tabs in which I can execute different commands. For example, some tabs tail error logs, whilst others simply execute shell scripts.

This allows me to quickly switch to what I need when I need it, and allows me a good general overview of the vital server processes that are executing.

A good 'quick reference' of screen commands can be found here.

The above mentioned 'behind the scenes' processes run in tabs of a detached screen session, and as such when my server reboots what I really want to do is restart the screen session and all its contained tabs.

screen with a .sh script

The first thing that I did was I wrote a shell script that launched screen, created my tabs, and executed the appropriate commands in each.

That looked something like this:

#!/bin/sh

echo "uid is ${UID}"  
echo "user is ${USER}"  
echo "username is ${USERNAME}"

killall screen

screen -AdmS myshell  
screen -S myshell -X screen -t 'iOS Notification Sender' sh /site/scripts/ios-notifications-sender.sh  
screen -S myshell -X screen -t 'Android Notification Sender' sh /site/scripts/android-notifications-sender.sh  

Executing that script from the command line did exactly as expected.

Credit where due.. I gained a lot of insight into working with screen from this answer on Stack Overflow. Give it a read.

It is also important to note that you need to make sure that any scripts that you are executing/logs that you are tailing are owned by the appropriate user/group, and have the correct permissions.

Now I just needed to execute the script on startup..

Upstart? No.. systemd

I made the mistake of rushing in without due consideration. I googled 'Ubuntu startup scripts' or equivalent and stumbled upon Upstart. I dived in without noting that in the latest versions of Ubuntu systemd is the default init daemon.

I don't want to confuse any potential readers by posting links to things that did not work. OK.. I will anyway :)

I did try a number of things that yielded less than ideal results. For example I tried this, and whilst it did execute my shell script, for some reason it was completely inconsistent in the screen tabs that it actually opened. That is to say that each reboot it opened a different (but incomplete) combination of tabs. It also seemed a little more complex (read low level) than it surely needed to be? Not ideal.

What did work

Eventually I discovered systemd and investigated its usage.

I found this answer which gave a general insight into configuration of systemd startup items.

I then worked backwards and read the documentation :)

I managed to get systemd to execute my shell script, but again I was getting inconsistent results - not every tab was being opened every time. I can only assume that this has something to do with permissions or timing (if anyone could share any insight, that would be appreciated), but the fact I got things working (see below) makes this somewhat bemusing.

I tried getting rid of the shell script entirely and executing my commands directly within the ExecStart configuration. This however.. did not work, and felt somewhat dirty.

The cleanest and best solution came to me in the end:

.screenrc

.screenrc is a configuration file for the initiation of a screen instance. If I hadn't rushed in and had taken the time to think about my approach in advance I would have surely used this methodology from the get go.

I wrote a .screenrc file and in it placed the tabs and code that I wanted executed in each. I then tested that it worked by instantiating a screen instance using the configuration file: screen -c ~/.screenrc.

This worked as expected.

Now it was simply a case of setting the ExecStart configuration command in my systemd service configuration to execute the above command. This 'native' approach executed without issue and creates all of the appropriate tabs each time.

Don't forget ghost

I rebooted my server a few times to test my setup. Everything worked as expected.

I had however completely forgotten about ghost. I had once again taken all our blogs down, and I had not put them back up. This is in itself an example use case of this type of startup script - to prevent downtime caused by oversight. Server configuration is complex, and it is certainly the case that you will forget something at somepoint. Make it so that it doesn't matter when you do.

Whilst researching some ghost specific queries I stumbled upon the crontab @reboot property. This might be of interest to you - it seems like another way of executing commands on reboot. I did not investigate this too much because:

  • It doesn't seem like a semantically logical place to restart background tasks.

  • It seems to have numerous issues, bugs, etc.

In the end I went down the same line as above - systemd.

This post by Chris Ebert outlines the specifics of setting up systemd for a ghost blog. It is well worth a read.

Paying back my Javascript/React knowledge debt

Recently I was suffering from a little bit of work induced burnout and as such I took the opportunity to step back from 12 hours of daily coding to attempt to reduce my knowledge debt and refactor our various products.

At the most basic level this would involve updating and implementing newer/updated versions of the various frameworks that we utilise on both the front and backend of our various products.

I however wanted to delve a little deeper and make sure that our codebases (across various platforms) were written in a clean, commented manner, and that they followed appropriate design patterns and paradigms.

In the process I was made aware of my lack of knowledge of some relatively simple (yet powerful) concepts. I filled in these gaps.

I explored ES6 and current best practices for utilising React. Here are some of the things that I discovered.

Refactoring some aspects of the front end of our web applications was somewhat arduous, and demonstrated that common situation - that whereby after hours of work to the viewing public absolutely nothing has changed.

  • displayName

I saw the displayName property being utilised in an open source project that I was working with. Whilst 'Non-standard' I see no harm in setting this property. It is utilised by browser consoles to display a more informative name for your functions. It can make debugging a little easier.

  • Spread syntax (...)

This syntax is really useful. It allows much simpler usage and manipulation of arrays, including:

  • passing arrays as function parameters
  • more succinct copying and pushing

To read more, take a look here.

  • map

Whilst I was aware of the map function on the Array prototype, I never used it. I found/find that traditional for loops are clear and concise (enough).

In certain situations however map is not appropriate. For example it doesn't allow you to break out of the loop.

Whilst working with React I have had situations whereby I have wanted to inline an array loop, and using map is the logical choice for code readability.

You can specify the this context as the second parameter. It is all really quite simple.

  • reduce

I was not aware of the reduce function on the Array prototype. I discovered it as a result of wanting to flatten an array of arrays. It is simple and powerful.

To read more, take a look here.

  • Hoisting

This is one of those buzzwords utilised in every beginner Javascript book/tutorial. That said I have never attributed a concept which I fully understand and appreciate to its 'term'. It is weird how one just gets things.. understands things, and uses them. Concepts like 'hoisting' are second nature, and are subconsciously understood.

Hoisting refers to the situation whereby a variable declaration is hoisted to the top of its enclosing scope.

The declaration is 'hoisted' but the initialisation is not. See here for more.

  • const, let, and var

const

As outlined on the Mozilla website, "The const declaration creates a read-only reference to a value. It does not mean the value it holds is immutable, just that the variable identifier cannot be reassigned."

You must assign it a value when you initialise it as the value can not be changed.

You can however modify object properties, and array values when an object/array respectively is defined as a const.

const protects from reassignment but object properties can still be changed. Use Object.freeze to stop that.

You can have block scoped const definitions.

let and var

Again as outlined by Mozilla, in my opinion the most important difference between let and var is with scoping.

let is scoped to its defining block, whilst var is defined to its enclosing function.

This Stack Overflow answer demonstrates this well.

  • Loop labels

You can name for loops so you can break them by name.

A label allows you to identify a particular loop such that when nested you can break or continue from a specific loop.

This makes your code considerably easier to follow, and comes in handy when you are not using map ;)

React

  • ES6 Syntax

I was aware of ES6 but had not investigated it (until now). With a significant amount on my plate I was of the view that the time investment was not worth it. In hindsight, having now gained a thorough understanding of the new functionality this was foolish. Going forward writing Javascript code (generally) will be significantly easier with the new and powerful functionality provided by ES6.

This post outlines a number of ES6 changes that directly relate to React.

  • Arrow functions

One aspect of the new ES6 syntax which is particularly powerful is arrow functions.

One downside? of moving to ES6 class syntax is that you lose some of the behind the scenes benefits of the createClass syntax such as autobinding of component context.

Implicitly having to set context is a little strange (when you have gotten used to it being done for you), but it is relatively simple to do, and makes your code better - I like to minimise the amount of 'magic' in my code.

  • Stateless functions

As outlined in the docs, stateless functions allow a simpler syntax for defining components which do not need to maintain. For example 'presentational components' (mentioned below).

  • Presentational components

Dan Abramov has written a detailed piece on separation of concerns within React.

I had gotten somewhat complacent in my approach to writing Reactive components. I could write code quickly, but everything knew about everything and as such following the chain up the virtual DOM to discern where/why a particular value was set as it was was becoming somewhat arduous. This would have been much worse had the components in question been more complex.

Refactoring such that a singular container component 'controls' and interacts with the backend has essentially 'contained' my complexity which in fact makes things significantly simpler.

As a result of investigating and implementing these changes I also took the time to investigate..

  • Redux

Redux is a state management system developed by Dan Abramov at Facebook. He has created a 30 part (2 hours) series of screencasts which explain how to get started.

Whilst I have not implemented Redux across our products (for the most part our state requirements are simple enough that it is not currently worth the time investment), I certainly will utilise it in new products.

Its simplicity is incredible.

  • Scaling

During my research I stumbled upon this post about scaling React applications. It mentions generators, and more complex implementations of data fetching alongside Redux. An interesting read, and something I will be looking into further..

It mentions redux-thunk which defines a 'thunk' well - "A thunk is a function that wraps an expression to delay its evaluation.". I mention this because a thunk is another one of those things that I understand and utilise daily yet have never attributed to its associated buzzword. Fun stuff.

Thats it..

Thats it for now. Nothing particularly complex or exciting.. simply an overview of some interesting and useful 'things'. Enjoy !