update go.mod

Add nats broker as a default

.travis.yml

@@ -1,7 +1,9 @@
 language: go
 go:
-- 1.10.x
 - 1.11.x
+- 1.12.x
+env:
+  - GO111MODULE=on
 notifications:
   slack:
     secure: aEvhLbhujaGaKSrOokiG3//PaVHTIrc3fBpoRbCRqfZpyq6WREoapJJhF+tIpWWOwaC9GmChbD6aHo/jMUgwKXVyPSaNjiEL87YzUUpL8B2zslNp1rgfTg/LrzthOx3Q1TYwpaAl3to0fuHUVFX4yMeC2vuThq7WSXgMMxFCtbc=

README.md

@@ -1,6 +1,6 @@
 # Go Micro [![License](https://img.shields.io/:license-apache-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![GoDoc](https://godoc.org/github.com/micro/go-micro?status.svg)](https://godoc.org/github.com/micro/go-micro) [![Travis CI](https://api.travis-ci.org/micro/go-micro.svg?branch=master)](https://travis-ci.org/micro/go-micro) [![Go Report Card](https://goreportcard.com/badge/micro/go-micro)](https://goreportcard.com/report/github.com/micro/go-micro)
 
-Go Micro is a pluggable framework for micro service development.
+Go Micro is a framework for microservice development.
 
 ## Overview
 
@@ -8,7 +8,7 @@ Go Micro provides the core requirements for distributed systems development incl
 The **micro** philosophy is sane defaults with a pluggable architecture. We provide defaults to get you started quickly 
 but everything can be easily swapped out. 
 
-<img src="https://micro.mu/docs/images/go-micro.png" />
+<img src="https://micro.mu/docs/images/go-micro.svg" />
 
 Plugins are available at [github.com/micro/go-plugins](https://github.com/micro/go-plugins).
 
@@ -19,8 +19,9 @@ Follow us on [Twitter](https://twitter.com/microhq) or join the [Slack](http://s
 Go Micro abstracts away the details of distributed systems. Here are the main features.
 
 - **Service Discovery** - Automatic service registration and name resolution. Service discovery is at the core of micro service 
-development. When service A needs to speak to service B it needs the location of that service. Consul is the default discovery 
-system with multicast DNS (mdns) as a local option or the SWIM protocol (gossip) for zero dependency p2p networks.
+development. When service A needs to speak to service B it needs the location of that service. The default discovery mechanism is 
+multicast DNS (mdns), a zeroconf system. You can optionally set gossip using the SWIM protocol for p2p networks or consul for a
+resilient cloud-native setup.
 
 - **Load Balancing** - Client side load balancing built on service discovery. Once we have the addresses of any number of instances 
 of a service we now need a way to decide which node to route to. We use random hashed load balancing to provide even distribution 
@@ -30,7 +31,7 @@ across the services and retry a different node if there's a problem.
 to seamlessly encode and decode Go types for you. Any variety of messages could be encoded and sent from different clients. The client 
 and server handle this by default. This includes protobuf and json by default.
 
-- **Sync Streaming** - RPC based request/response with support for bidirectional streaming. We provide an abstraction for synchronous 
+- **Request/Response** - RPC based request/response with support for bidirectional streaming. We provide an abstraction for synchronous 
 communication. A request made to a service will be automatically resolved, load balanced, dialled and streamed. The default 
 transport is http/1.1 or http2 when tls is enabled.
 
@@ -46,10 +47,3 @@ are pluggable and allows Go Micro to be runtime agnostic. You can plugin any und
 
 See the [docs](https://micro.mu/docs/go-micro.html) for detailed information on the architecture, installation and use of go-micro.
 
-## Sponsors
-
-Sixt is an Enterprise Sponsor of Micro
-
-<a href="https://micro.mu/blog/2016/04/25/announcing-sixt-sponsorship.html"><img src="https://micro.mu/sixt_logo.png" width=150px height="auto" /></a>
-
-Become a sponsor by backing micro on [Patreon](https://www.patreon.com/microhq)

README.zh-cn.md

@@ -0,0 +1,36 @@
+# Go Micro [![License](https://img.shields.io/:license-apache-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![GoDoc](https://godoc.org/github.com/micro/go-micro?status.svg)](https://godoc.org/github.com/micro/go-micro) [![Travis CI](https://api.travis-ci.org/micro/go-micro.svg?branch=master)](https://travis-ci.org/micro/go-micro) [![Go Report Card](https://goreportcard.com/badge/micro/go-micro)](https://goreportcard.com/report/github.com/micro/go-micro)
+
+Go Micro是基于Golang的微服务开发框架。
+
+## 概览
+
+Go Micro提供分布式系统开发的核心库，包含RPC与事件驱动的通信机制。
+
+**micro**的设计哲学是可插拔的架构理念，她提供可快速构建系统的组件，并且可以根据自身的需求剥离默认实现并自行定制。
+
+<img src="https://micro.mu/docs/images/go-micro.svg" />
+
+所有插件可在仓库[github.com/micro/go-plugins](https://github.com/micro/go-plugins)中找到。
+
+可以订阅我们的[Twitter](https://twitter.com/microhq)或者加入[Slack](http://slack.micro.mu/)论坛。
+
+## 特性
+
+Go Micro把分布式系统的各种细节抽象出来。下面是它的主要特性。
+
+- **服务发现（Service Discovery）** - 自动服务注册与名称解析。服务发现是微服务开发中的核心。当服务A要与服务B协作时，它得知道B在哪里。默认的服务发现系统是Consul，而multicast DNS (mdns，组播)机制作为本地解决方案，或者零依赖的P2P网络中的SWIM协议（gossip）。 
+
+- **负载均衡（Load Balancing）** - 在服务发现之上构建了负载均衡机制。当我们得到一个服务的任意多个的实例节点时，我们要一个机制去决定要路由到哪一个节点。我们使用随机处理过的哈希负载均衡机制来保证对服务请求颁发的均匀分布，并且在发生问题时进行重试。
+
+- **消息编码（Message Encoding）** - 支持基于内容类型（content-type）动态编码消息。客户端和服务端会一起使用content-type的格式来对Go进行无缝编/解码。各种各样的消息被编码会发送到不同的客户端，客户端服服务端默认会处理这些消息。content-type默认包含proto-rpc和json-rpc。
+
+- **Request/Response** - RPC通信基于支持双向流的请求/响应方式，我们提供有抽象的同步通信机制。请求发送到服务时，会自动解析、负载均衡、拨号、转成字节流。默认的传输协议是http/1.1，而tls下使用http2协议。
+
+- **异步消息（Async Messaging）** - 发布订阅（PubSub）头等功能内置在异步通信与事件驱动架构中。事件通知在微服务开发中处于核心位置。默认的消息传送使用点到点http/1.1，激活tls时则使用http2。
+
+- **可插拔接口（Pluggable Interfaces）** - Go Micro为每个分布式系统抽象出接口。因此，Go Micro的接口都是可插拔的，允许其在运行时不可知的情况下仍可支持。所以只要实现接口，可以在内部使用任何的技术。更多插件请参考：[github.com/micro/go-plugins](https://github.com/micro/go-plugins)。
+
+## 快速上手
+
+更多关于架构、安装的资料可以查看[文档](https://micro.mu/docs/go-micro_cn.html)。
+

agent/README.md

@@ -0,0 +1,197 @@
+# Agent 
+
+Agent is a library used to create commands, inputs and robot services
+
+## Getting Started
+
+- [Commands](#commands) - Commands are functions executed by the bot based on text based pattern matching.
+- [Inputs](#inputs) - Inputs are plugins for communication e.g Slack, Telegram, IRC, etc.
+- [Services](#services) - Write bots as micro services
+
+## Commands
+
+Commands are functions executed by the bot based on text based pattern matching.
+
+### Write a Command
+
+```go
+import "github.com/micro/go-micro/agent/command"
+
+func Ping() command.Command {
+	usage := "ping"
+	description := "Returns pong"
+
+	return command.NewCommand("ping", usage, desc, func(args ...string) ([]byte, error) {
+		return []byte("pong"), nil
+	})
+}
+```
+
+### Register the command
+
+Add the command to the Commands map with a pattern key that can be matched by golang/regexp.Match
+
+```go
+import "github.com/micro/go-micro/agent/command"
+
+func init() {
+	command.Commands["^ping$"] = Ping()
+}
+```
+
+### Rebuild Micro
+
+Build binary
+```shell
+cd github.com/micro/micro
+
+// For local use
+go build -i -o micro ./main.go
+
+// For docker image
+CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o micro ./main.go
+```
+
+## Inputs
+
+Inputs are plugins for communication e.g Slack, HipChat, XMPP, IRC, SMTP, etc, etc. 
+
+New inputs can be added in the following way.
+
+### Write an Input
+
+Write an input that satisfies the Input interface.
+
+```go
+type Input interface {
+	// Provide cli flags
+	Flags() []cli.Flag
+	// Initialise input using cli context
+	Init(*cli.Context) error
+	// Stream events from the input
+	Stream() (Conn, error)
+	// Start the input
+	Start() error
+	// Stop the input
+	Stop() error
+	// name of the input
+	String() string
+}
+```
+
+### Register the input
+
+Add the input to the Inputs map.
+
+```go
+import "github.com/micro/micro/bot/input"
+
+func init() {
+	input.Inputs["name"] = MyInput
+}
+```
+
+### Rebuild Micro
+
+Build binary
+```shell
+cd github.com/micro/micro
+
+// For local use
+go build -i -o micro ./main.go
+
+// For docker image
+CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o micro ./main.go
+```
+
+## Services
+
+The micro bot supports the ability to create commands as micro services. 
+
+### How does it work?
+
+The bot watches the service registry for services with it's namespace. The default namespace is `go.micro.bot`. 
+Any service within this namespace will automatically be added to the list of available commands. When a command 
+is executed, the bot will call the service with method `Command.Exec`. It also expects the method `Command.Help` 
+to exist for usage info.
+
+
+The service interface is as follows and can be found at [go-micro/agent/proto](https://github.com/micro/go-micro/agent/blob/master/proto/bot.proto)
+
+```
+syntax = "proto3";
+
+package go.micro.bot;
+
+service Command {
+	rpc Help(HelpRequest) returns (HelpResponse) {};
+	rpc Exec(ExecRequest) returns (ExecResponse) {};
+}
+
+message HelpRequest {
+}
+
+message HelpResponse {
+	string usage = 1;
+	string description = 2;
+}
+
+message ExecRequest {
+	repeated string args = 1;
+}
+
+message ExecResponse {
+	bytes result = 1;
+	string error = 2;
+}
+```
+
+### Example
+
+Here's an example echo command as a microservice
+
+```go
+package main
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/micro/go-micro"
+	"golang.org/x/net/context"
+
+	proto "github.com/micro/go-micro/agent/proto"
+)
+
+type Command struct{}
+
+// Help returns the command usage
+func (c *Command) Help(ctx context.Context, req *proto.HelpRequest, rsp *proto.HelpResponse) error {
+	// Usage should include the name of the command
+	rsp.Usage = "echo"
+	rsp.Description = "This is an example bot command as a micro service which echos the message"
+	return nil
+}
+
+// Exec executes the command
+func (c *Command) Exec(ctx context.Context, req *proto.ExecRequest, rsp *proto.ExecResponse) error {
+	rsp.Result = []byte(strings.Join(req.Args, " "))
+	// rsp.Error could be set to return an error instead
+	// the function error would only be used for service level issues
+	return nil
+}
+
+func main() {
+	service := micro.NewService(
+		micro.Name("go.micro.bot.echo"),
+	)
+
+	service.Init()
+
+	proto.RegisterCommandHandler(service.Server(), new(Command))
+
+	if err := service.Run(); err != nil {
+		fmt.Println(err)
+	}
+}
+```

agent/agent.go

@@ -0,0 +1,2 @@
+// Package agent provides an interface for building robots
+package agent

agent/command/command.go

@@ -0,0 +1,54 @@
+// Package command is an interface for defining bot commands
+package command
+
+var (
+	// Commmands keyed by golang/regexp patterns
+	// regexp.Match(key, input) is used to match
+	Commands = map[string]Command{}
+)
+
+// Command is the interface for specific named
+// commands executed via plugins or the bot.
+type Command interface {
+	// Executes the command with args passed in
+	Exec(args ...string) ([]byte, error)
+	// Usage of the command
+	Usage() string
+	// Description of the command
+	Description() string
+	// Name of the command
+	String() string
+}
+
+type cmd struct {
+	name        string
+	usage       string
+	description string
+	exec        func(args ...string) ([]byte, error)
+}
+
+func (c *cmd) Description() string {
+	return c.description
+}
+
+func (c *cmd) Exec(args ...string) ([]byte, error) {
+	return c.exec(args...)
+}
+
+func (c *cmd) Usage() string {
+	return c.usage
+}
+
+func (c *cmd) String() string {
+	return c.name
+}
+
+// NewCommand helps quickly create a new command
+func NewCommand(name, usage, description string, exec func(args ...string) ([]byte, error)) Command {
+	return &cmd{
+		name:        name,
+		usage:       usage,
+		description: description,
+		exec:        exec,
+	}
+}

agent/command/command_test.go

@@ -0,0 +1,65 @@
+package command
+
+import (
+	"testing"
+)
+
+func TestCommand(t *testing.T) {
+	c := &cmd{
+		name:        "test",
+		usage:       "test usage",
+		description: "test description",
+		exec: func(args ...string) ([]byte, error) {
+			return []byte("test"), nil
+		},
+	}
+
+	if c.String() != c.name {
+		t.Fatalf("expected name %s got %s", c.name, c.String())
+	}
+
+	if c.Usage() != c.usage {
+		t.Fatalf("expected usage %s got %s", c.usage, c.Usage())
+	}
+
+	if c.Description() != c.description {
+		t.Fatalf("expected description %s got %s", c.description, c.Description())
+	}
+
+	if r, err := c.Exec(); err != nil {
+		t.Fatal(err)
+	} else if string(r) != "test" {
+		t.Fatalf("expected exec result test got %s", string(r))
+	}
+}
+
+func TestNewCommand(t *testing.T) {
+	c := &cmd{
+		name:        "test",
+		usage:       "test usage",
+		description: "test description",
+		exec: func(args ...string) ([]byte, error) {
+			return []byte("test"), nil
+		},
+	}
+
+	nc := NewCommand(c.name, c.usage, c.description, c.exec)
+
+	if nc.String() != c.name {
+		t.Fatalf("expected name %s got %s", c.name, nc.String())
+	}
+
+	if nc.Usage() != c.usage {
+		t.Fatalf("expected usage %s got %s", c.usage, nc.Usage())
+	}
+
+	if nc.Description() != c.description {
+		t.Fatalf("expected description %s got %s", c.description, nc.Description())
+	}
+
+	if r, err := nc.Exec(); err != nil {
+		t.Fatal(err)
+	} else if string(r) != "test" {
+		t.Fatalf("expected exec result test got %s", string(r))
+	}
+}

agent/input/discord/README.md

@@ -0,0 +1,22 @@
+# Discord input for micro-bot
+[Discord](https://discordapp.com) support for micro bot based on [discordgo](github.com/bwmarrin/discordgo).
+
+This was originally written by Aleksandr Tihomirov (@zet4) and can be found at https://github.com/zet4/micro-misc/.
+
+## Options
+### discord_token
+
+You have to supply an application token via `--discord_token`.
+
+Head over to Discord's [developer introduction](https://discordapp.com/developers/docs/intro)
+to learn how to create applications and how the API works.
+
+### discord_prefix
+
+Set a command prefix with `--discord_prefix`. The default prefix is `Micro `.
+You can mention the bot or use the prefix to run a command.
+
+### discord_whitelist
+
+Pass a list of comma-separated user IDs with `--discord_whitelist`. Only allow
+these users to use the bot.

agent/input/discord/conn.go

@@ -0,0 +1,94 @@
+package discord
+
+import (
+	"errors"
+	"strings"
+	"sync"
+
+	"github.com/bwmarrin/discordgo"
+	"github.com/micro/go-micro/agent/input"
+	"github.com/micro/go-micro/util/log"
+)
+
+type discordConn struct {
+	master *discordInput
+	exit   chan struct{}
+	recv   chan *discordgo.Message
+
+	sync.Mutex
+}
+
+func newConn(master *discordInput) *discordConn {
+	conn := &discordConn{
+		master: master,
+		exit:   make(chan struct{}),
+		recv:   make(chan *discordgo.Message),
+	}
+
+	conn.master.session.AddHandler(func(s *discordgo.Session, m *discordgo.MessageCreate) {
+		if m.Author.ID == master.botID {
+			return
+		}
+
+		whitelisted := false
+		for _, ID := range conn.master.whitelist {
+			if m.Author.ID == ID {
+				whitelisted = true
+				break
+			}
+		}
+
+		if len(master.whitelist) > 0 && !whitelisted {
+			return
+		}
+
+		var valid bool
+		m.Message.Content, valid = conn.master.prefixfn(m.Message.Content)
+		if !valid {
+			return
+		}
+
+		conn.recv <- m.Message
+	})
+
+	return conn
+}
+
+func (dc *discordConn) Recv(event *input.Event) error {
+	for {
+		select {
+		case <-dc.exit:
+			return errors.New("connection closed")
+		case msg := <-dc.recv:
+
+			event.From = msg.ChannelID + ":" + msg.Author.ID
+			event.To = dc.master.botID
+			event.Type = input.TextEvent
+			event.Data = []byte(msg.Content)
+			return nil
+		}
+	}
+}
+
+func (dc *discordConn) Send(e *input.Event) error {
+	fields := strings.Split(e.To, ":")
+	_, err := dc.master.session.ChannelMessageSend(fields[0], string(e.Data))
+	if err != nil {
+		log.Log("[bot][loop][send]", err)
+	}
+	return nil
+}
+
+func (dc *discordConn) Close() error {
+	if err := dc.master.session.Close(); err != nil {
+		return err
+	}
+
+	select {
+	case <-dc.exit:
+		return nil
+	default:
+		close(dc.exit)
+	}
+	return nil
+}

agent/input/discord/discord.go

@@ -0,0 +1,153 @@
+package discord
+
+import (
+	"fmt"
+	"sync"
+
+	"errors"
+	"strings"
+
+	"github.com/bwmarrin/discordgo"
+	"github.com/micro/cli"
+	"github.com/micro/go-micro/agent/input"
+)
+
+func init() {
+	input.Inputs["discord"] = newInput()
+}
+
+func newInput() *discordInput {
+	return &discordInput{}
+}
+
+type discordInput struct {
+	token     string
+	whitelist []string
+	prefix    string
+	prefixfn  func(string) (string, bool)
+	botID     string
+
+	session *discordgo.Session
+
+	sync.Mutex
+	running bool
+	exit    chan struct{}
+}
+
+func (d *discordInput) Flags() []cli.Flag {
+	return []cli.Flag{
+		cli.StringFlag{
+			Name:   "discord_token",
+			EnvVar: "MICRO_DISCORD_TOKEN",
+			Usage:  "Discord token (prefix with Bot if it's for bot account)",
+		},
+		cli.StringFlag{
+			Name:   "discord_whitelist",
+			EnvVar: "MICRO_DISCORD_WHITELIST",
+			Usage:  "Discord Whitelist (seperated by ,)",
+		},
+		cli.StringFlag{
+			Name:   "discord_prefix",
+			Usage:  "Discord Prefix",
+			EnvVar: "MICRO_DISCORD_PREFIX",
+			Value:  "Micro ",
+		},
+	}
+}
+
+func (d *discordInput) Init(ctx *cli.Context) error {
+	token := ctx.String("discord_token")
+	whitelist := ctx.String("discord_whitelist")
+	prefix := ctx.String("discord_prefix")
+
+	if len(token) == 0 {
+		return errors.New("require token")
+	}
+
+	d.token = token
+	d.prefix = prefix
+
+	if len(whitelist) > 0 {
+		d.whitelist = strings.Split(whitelist, ",")
+	}
+
+	return nil
+}
+
+func (d *discordInput) Start() error {
+	if len(d.token) == 0 {
+		return errors.New("missing discord configuration")
+	}
+
+	d.Lock()
+	defer d.Unlock()
+
+	if d.running {
+		return nil
+	}
+
+	var err error
+	d.session, err = discordgo.New(d.token)
+	if err != nil {
+		return err
+	}
+
+	u, err := d.session.User("@me")
+	if err != nil {
+		return err
+	}
+
+	d.botID = u.ID
+	d.prefixfn = CheckPrefixFactory(fmt.Sprintf("<@%s> ", d.botID), fmt.Sprintf("<@!%s> ", d.botID), d.prefix)
+
+	d.exit = make(chan struct{})
+	d.running = true
+
+	return nil
+}
+
+func (d *discordInput) Stream() (input.Conn, error) {
+	d.Lock()
+	defer d.Unlock()
+	if !d.running {
+		return nil, errors.New("not running")
+	}
+
+	//Fire-n-forget close just in case...
+	d.session.Close()
+
+	conn := newConn(d)
+	if err := d.session.Open(); err != nil {
+		return nil, err
+	}
+	return conn, nil
+}
+
+func (d *discordInput) Stop() error {
+	d.Lock()
+	defer d.Unlock()
+
+	if !d.running {
+		return nil
+	}
+
+	close(d.exit)
+	d.running = false
+	return nil
+}
+
+func (d *discordInput) String() string {
+	return "discord"
+}
+
+// CheckPrefixFactory Creates a prefix checking function and stuff.
+func CheckPrefixFactory(prefixes ...string) func(string) (string, bool) {
+	return func(content string) (string, bool) {
+		for _, prefix := range prefixes {
+			if strings.HasPrefix(content, prefix) {
+				return strings.TrimPrefix(content, prefix), true
+			}
+		}
+		return "", false
+	}
+}

agent/input/input.go

@@ -0,0 +1,55 @@
+// Package input is an interface for bot inputs
+package input
+
+import (
+	"github.com/micro/cli"
+)
+
+type EventType string
+
+const (
+	TextEvent EventType = "text"
+)
+
+var (
+	// Inputs keyed by name
+	// Example slack or hipchat
+	Inputs = map[string]Input{}
+)
+
+// Event is the unit sent and received
+type Event struct {
+	Type EventType
+	From string
+	To   string
+	Data []byte
+	Meta map[string]interface{}
+}
+
+// Input is an interface for sources which
+// provide a way to communicate with the bot.
+// Slack, HipChat, XMPP, etc.
+type Input interface {
+	// Provide cli flags
+	Flags() []cli.Flag
+	// Initialise input using cli context
+	Init(*cli.Context) error
+	// Stream events from the input
+	Stream() (Conn, error)
+	// Start the input
+	Start() error
+	// Stop the input
+	Stop() error
+	// name of the input
+	String() string
+}
+
+// Conn interface provides a way to
+// send and receive events. Send and
+// Recv both block until succeeding
+// or failing.
+type Conn interface {
+	Close() error
+	Recv(*Event) error
+	Send(*Event) error
+}

agent/input/slack/conn.go

@@ -0,0 +1,160 @@
+package slack
+
+import (
+	"errors"
+	"fmt"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/micro/go-micro/agent/input"
+	"github.com/nlopes/slack"
+)
+
+// Satisfies the input.Conn interface
+type slackConn struct {
+	auth *slack.AuthTestResponse
+	rtm  *slack.RTM
+	exit chan bool
+
+	sync.Mutex
+	names map[string]string
+}
+
+func (s *slackConn) run() {
+	// func retrieves user names and maps to IDs
+	setNames := func() {
+		names := make(map[string]string)
+		users, err := s.rtm.Client.GetUsers()
+		if err != nil {
+			return
+		}
+
+		for _, user := range users {
+			names[user.ID] = user.Name
+		}
+
+		s.Lock()
+		s.names = names
+		s.Unlock()
+	}
+
+	setNames()
+
+	t := time.NewTicker(time.Minute)
+	defer t.Stop()
+
+	for {
+		select {
+		case <-s.exit:
+			return
+		case <-t.C:
+			setNames()
+		}
+	}
+}
+
+func (s *slackConn) getName(id string) string {
+	s.Lock()
+	name := s.names[id]
+	s.Unlock()
+	return name
+}
+
+func (s *slackConn) Close() error {
+	select {
+	case <-s.exit:
+		return nil
+	default:
+		close(s.exit)
+	}
+	return nil
+}
+
+func (s *slackConn) Recv(event *input.Event) error {
+	if event == nil {
+		return errors.New("event cannot be nil")
+	}
+
+	for {
+		select {
+		case <-s.exit:
+			return errors.New("connection closed")
+		case e := <-s.rtm.IncomingEvents:
+			switch ev := e.Data.(type) {
+			case *slack.MessageEvent:
+				// only accept type message
+				if ev.Type != "message" {
+					continue
+				}
+
+				// only accept DMs or messages to me
+				switch {
+				case strings.HasPrefix(ev.Channel, "D"):
+				case strings.HasPrefix(ev.Text, s.auth.User):
+				case strings.HasPrefix(ev.Text, fmt.Sprintf("<@%s>", s.auth.UserID)):
+				default:
+					continue
+				}
+
+				// Strip username from text
+				switch {
+				case strings.HasPrefix(ev.Text, s.auth.User):
+					args := strings.Split(ev.Text, " ")[1:]
+					ev.Text = strings.Join(args, " ")
+					event.To = s.auth.User
+				case strings.HasPrefix(ev.Text, fmt.Sprintf("<@%s>", s.auth.UserID)):
+					args := strings.Split(ev.Text, " ")[1:]
+					ev.Text = strings.Join(args, " ")
+					event.To = s.auth.UserID
+				}
+
+				if event.Meta == nil {
+					event.Meta = make(map[string]interface{})
+				}
+
+				// fill in the blanks
+				event.From = ev.Channel + ":" + ev.User
+				event.Type = input.TextEvent
+				event.Data = []byte(ev.Text)
+				event.Meta["reply"] = ev
+				return nil
+			case *slack.InvalidAuthEvent:
+				return errors.New("invalid credentials")
+			}
+		}
+	}
+}
+
+func (s *slackConn) Send(event *input.Event) error {
+	var channel, message, name string
+
+	if len(event.To) == 0 {
+		return errors.New("require Event.To")
+	}
+
+	parts := strings.Split(event.To, ":")
+
+	if len(parts) == 2 {
+		channel = parts[0]
+		name = s.getName(parts[1])
+		// try using reply meta
+	} else if ev, ok := event.Meta["reply"]; ok {
+		channel = ev.(*slack.MessageEvent).Channel
+		name = s.getName(ev.(*slack.MessageEvent).User)
+	}
+
+	// don't know where to send the message
+	if len(channel) == 0 {
+		return errors.New("could not determine who message is to")
+	}
+
+	if len(name) == 0 || strings.HasPrefix(channel, "D") {
+		message = string(event.Data)
+	} else {
+		message = fmt.Sprintf("@%s: %s", name, string(event.Data))
+	}
+
+	s.rtm.SendMessage(s.rtm.NewOutgoingMessage(message, channel))
+	return nil
+}

agent/input/slack/slack.go

@@ -0,0 +1,147 @@
+package slack
+
+import (
+	"errors"
+	"sync"
+
+	"github.com/micro/cli"
+	"github.com/micro/go-micro/agent/input"
+	"github.com/nlopes/slack"
+)
+
+type slackInput struct {
+	debug bool
+	token string
+
+	sync.Mutex
+	running bool
+	exit    chan bool
+
+	api *slack.Client
+}
+
+func init() {
+	input.Inputs["slack"] = NewInput()
+}
+
+func (p *slackInput) Flags() []cli.Flag {
+	return []cli.Flag{
+		cli.BoolFlag{
+			Name:   "slack_debug",
+			Usage:  "Slack debug output",
+			EnvVar: "MICRO_SLACK_DEBUG",
+		},
+		cli.StringFlag{
+			Name:   "slack_token",
+			Usage:  "Slack token",
+			EnvVar: "MICRO_SLACK_TOKEN",
+		},
+	}
+}
+
+func (p *slackInput) Init(ctx *cli.Context) error {
+	debug := ctx.Bool("slack_debug")
+	token := ctx.String("slack_token")
+
+	if len(token) == 0 {
+		return errors.New("missing slack token")
+	}
+
+	p.debug = debug
+	p.token = token
+
+	return nil
+}
+
+func (p *slackInput) Stream() (input.Conn, error) {
+	p.Lock()
+	defer p.Unlock()
+
+	if !p.running {
+		return nil, errors.New("not running")
+	}
+
+	// test auth
+	auth, err := p.api.AuthTest()
+	if err != nil {
+		return nil, err
+	}
+
+	rtm := p.api.NewRTM()
+	exit := make(chan bool)
+
+	go rtm.ManageConnection()
+
+	go func() {
+		select {
+		case <-p.exit:
+			select {
+			case <-exit:
+				return
+			default:
+				close(exit)
+			}
+		case <-exit:
+		}
+
+		rtm.Disconnect()
+	}()
+
+	conn := &slackConn{
+		auth:  auth,
+		rtm:   rtm,
+		exit:  exit,
+		names: make(map[string]string),
+	}
+
+	go conn.run()
+
+	return conn, nil
+}
+
+func (p *slackInput) Start() error {
+	if len(p.token) == 0 {
+		return errors.New("missing slack token")
+	}
+
+	p.Lock()
+	defer p.Unlock()
+
+	if p.running {
+		return nil
+	}
+
+	api := slack.New(p.token, slack.OptionDebug(p.debug))
+
+	// test auth
+	_, err := api.AuthTest()
+	if err != nil {
+		return err
+	}
+
+	p.api = api
+	p.exit = make(chan bool)
+	p.running = true
+	return nil
+}
+
+func (p *slackInput) Stop() error {
+	p.Lock()
+	defer p.Unlock()
+
+	if !p.running {
+		return nil
+	}
+
+	close(p.exit)
+	p.running = false
+	return nil
+}
+
+func (p *slackInput) String() string {
+	return "slack"
+}
+
+func NewInput() input.Input {
+	return &slackInput{}
+}

agent/input/telegram/README.md

@@ -0,0 +1,18 @@
+# Telegram Messenger input for micro bot
+[Telegram](https://telegram.org) support for micro bot based on [telegram-bot-api](https://github.com/go-telegram-bot-api/telegram-bot-api).
+
+## Options
+### --telegram_token (required)
+
+Sets bot's token for interacting with API.
+
+Head over to Telegram's [API documentation](https://core.telegram.org/bots/api)
+to learn how to create bots and how the API works.
+
+### --telegram_debug
+
+Sets the debug flag to make the bot's output verbose.
+
+### --telegram_whitelist
+
+Sets a list of comma-separated nicknames (without @ symbol in the beginning) for interacting with bot. Only these users can use the bot.

agent/input/telegram/conn.go

@@ -0,0 +1,115 @@
+package telegram
+
+import (
+	"errors"
+	"strings"
+	"sync"
+
+	"github.com/forestgiant/sliceutil"
+	"github.com/micro/go-micro/agent/input"
+	"github.com/micro/go-micro/util/log"
+	"gopkg.in/telegram-bot-api.v4"
+)
+
+type telegramConn struct {
+	input *telegramInput
+
+	recv <-chan tgbotapi.Update
+	exit chan bool
+
+	syncCond *sync.Cond
+	mutex    sync.Mutex
+}
+
+func newConn(input *telegramInput) (*telegramConn, error) {
+	conn := &telegramConn{
+		input: input,
+	}
+
+	conn.syncCond = sync.NewCond(&conn.mutex)
+
+	go conn.run()
+
+	return conn, nil
+}
+
+func (tc *telegramConn) run() {
+	u := tgbotapi.NewUpdate(0)
+	u.Timeout = 60
+	updates, err := tc.input.api.GetUpdatesChan(u)
+	if err != nil {
+		return
+	}
+
+	tc.recv = updates
+	tc.syncCond.Signal()
+
+	for {
+		select {
+		case <-tc.exit:
+			return
+		}
+	}
+}
+
+func (tc *telegramConn) Close() error {
+	return nil
+}
+
+func (tc *telegramConn) Recv(event *input.Event) error {
+	if event == nil {
+		return errors.New("event cannot be nil")
+	}
+
+	for {
+		if tc.recv == nil {
+			tc.mutex.Lock()
+			tc.syncCond.Wait()
+		}
+
+		update := <-tc.recv
+
+		if update.Message == nil || (len(tc.input.whitelist) > 0 && !sliceutil.Contains(tc.input.whitelist, update.Message.From.UserName)) {
+			continue
+		}
+
+		if event.Meta == nil {
+			event.Meta = make(map[string]interface{})
+		}
+
+		event.Type = input.TextEvent
+		event.From = update.Message.From.UserName
+		event.To = tc.input.api.Self.UserName
+		event.Data = []byte(update.Message.Text)
+		event.Meta["chatId"] = update.Message.Chat.ID
+		event.Meta["chatType"] = update.Message.Chat.Type
+		event.Meta["messageId"] = update.Message.MessageID
+
+		return nil
+	}
+}
+
+func (tc *telegramConn) Send(event *input.Event) error {
+	messageText := strings.TrimSpace(string(event.Data))
+
+	chatId := event.Meta["chatId"].(int64)
+	chatType := ChatType(event.Meta["chatType"].(string))
+
+	msgConfig := tgbotapi.NewMessage(chatId, messageText)
+	msgConfig.ParseMode = tgbotapi.ModeHTML
+
+	if sliceutil.Contains([]ChatType{Group, Supergroup}, chatType) {
+		msgConfig.ReplyToMessageID = event.Meta["messageId"].(int)
+	}
+
+	_, err := tc.input.api.Send(msgConfig)
+
+	if err != nil {
+		// probably it could be because of nested HTML tags -- telegram doesn't allow nested tags
+		log.Log("[telegram][Send] error:", err)
+		msgConfig.Text = "This bot couldn't send the response (Internal error)"
+		tc.input.api.Send(msgConfig)
+	}
+
+	return nil
+}

agent/input/telegram/telegram.go

@@ -0,0 +1,101 @@
+package telegram
+
+import (
+	"errors"
+	"strings"
+	"sync"
+
+	"github.com/micro/cli"
+	"github.com/micro/go-micro/agent/input"
+	"gopkg.in/telegram-bot-api.v4"
+)
+
+type telegramInput struct {
+	sync.Mutex
+
+	debug     bool
+	token     string
+	whitelist []string
+
+	api *tgbotapi.BotAPI
+}
+
+type ChatType string
+
+const (
+	Private    ChatType = "private"
+	Group      ChatType = "group"
+	Supergroup ChatType = "supergroup"
+)
+
+func init() {
+	input.Inputs["telegram"] = &telegramInput{}
+}
+
+func (ti *telegramInput) Flags() []cli.Flag {
+	return []cli.Flag{
+		cli.BoolFlag{
+			Name:   "telegram_debug",
+			EnvVar: "MICRO_TELEGRAM_DEBUG",
+			Usage:  "Telegram debug output",
+		},
+		cli.StringFlag{
+			Name:   "telegram_token",
+			EnvVar: "MICRO_TELEGRAM_TOKEN",
+			Usage:  "Telegram token",
+		},
+		cli.StringFlag{
+			Name:   "telegram_whitelist",
+			EnvVar: "MICRO_TELEGRAM_WHITELIST",
+			Usage:  "Telegram bot's users (comma-separated values)",
+		},
+	}
+}
+
+func (ti *telegramInput) Init(ctx *cli.Context) error {
+	ti.debug = ctx.Bool("telegram_debug")
+	ti.token = ctx.String("telegram_token")
+
+	whitelist := ctx.String("telegram_whitelist")
+
+	if whitelist != "" {
+		ti.whitelist = strings.Split(whitelist, ",")
+	}
+
+	if len(ti.token) == 0 {
+		return errors.New("missing telegram token")
+	}
+
+	return nil
+}
+
+func (ti *telegramInput) Stream() (input.Conn, error) {
+	ti.Lock()
+	defer ti.Unlock()
+
+	return newConn(ti)
+}
+
+func (ti *telegramInput) Start() error {
+	ti.Lock()
+	defer ti.Unlock()
+
+	api, err := tgbotapi.NewBotAPI(ti.token)
+	if err != nil {
+		return err
+	}
+
+	ti.api = api
+
+	api.Debug = ti.debug
+
+	return nil
+}
+
+func (ti *telegramInput) Stop() error {
+	return nil
+}
+
+func (p *telegramInput) String() string {
+	return "telegram"
+}

agent/proto/bot.micro.go

@@ -0,0 +1,118 @@
+// Code generated by protoc-gen-micro. DO NOT EDIT.
+// source: github.com/micro/go-bot/proto/bot.proto
+
+/*
+Package go_micro_bot is a generated protocol buffer package.
+
+It is generated from these files:
+	github.com/micro/go-bot/proto/bot.proto
+
+It has these top-level messages:
+	HelpRequest
+	HelpResponse
+	ExecRequest
+	ExecResponse
+*/
+package go_micro_bot
+
+import proto "github.com/golang/protobuf/proto"
+import fmt "fmt"
+import math "math"
+
+import (
+	context "context"
+	client "github.com/micro/go-micro/client"
+	server "github.com/micro/go-micro/server"
+)
+
+// Reference imports to suppress errors if they are not otherwise used.
+var _ = proto.Marshal
+var _ = fmt.Errorf
+var _ = math.Inf
+
+// This is a compile-time assertion to ensure that this generated file
+// is compatible with the proto package it is being compiled against.
+// A compilation error at this line likely means your copy of the
+// proto package needs to be updated.
+const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package
+
+// Reference imports to suppress errors if they are not otherwise used.
+var _ context.Context
+var _ client.Option
+var _ server.Option
+
+// Client API for Command service
+
+type CommandService interface {
+	Help(ctx context.Context, in *HelpRequest, opts ...client.CallOption) (*HelpResponse, error)
+	Exec(ctx context.Context, in *ExecRequest, opts ...client.CallOption) (*ExecResponse, error)
+}
+
+type commandService struct {
+	c    client.Client
+	name string
+}
+
+func NewCommandService(name string, c client.Client) CommandService {
+	if c == nil {
+		c = client.NewClient()
+	}
+	if len(name) == 0 {
+		name = "go.micro.bot"
+	}
+	return &commandService{
+		c:    c,
+		name: name,
+	}
+}
+
+func (c *commandService) Help(ctx context.Context, in *HelpRequest, opts ...client.CallOption) (*HelpResponse, error) {
+	req := c.c.NewRequest(c.name, "Command.Help", in)
+	out := new(HelpResponse)
+	err := c.c.Call(ctx, req, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+func (c *commandService) Exec(ctx context.Context, in *ExecRequest, opts ...client.CallOption) (*ExecResponse, error) {
+	req := c.c.NewRequest(c.name, "Command.Exec", in)
+	out := new(ExecResponse)
+	err := c.c.Call(ctx, req, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// Server API for Command service
+
+type CommandHandler interface {
+	Help(context.Context, *HelpRequest, *HelpResponse) error
+	Exec(context.Context, *ExecRequest, *ExecResponse) error
+}
+
+func RegisterCommandHandler(s server.Server, hdlr CommandHandler, opts ...server.HandlerOption) error {
+	type _command interface {
+		Help(ctx context.Context, in *HelpRequest, out *HelpResponse) error
+		Exec(ctx context.Context, in *ExecRequest, out *ExecResponse) error
+	}
+	type Command struct {
+		_command
+	}
+	h := &commandHandler{hdlr}
+	return s.Handle(s.NewHandler(&Command{h}, opts...))
+}
+
+type commandHandler struct {
+	CommandHandler
+}
+
+func (h *commandHandler) Help(ctx context.Context, in *HelpRequest, out *HelpResponse) error {
+	return h.CommandHandler.Help(ctx, in, out)
+}
+
+func (h *commandHandler) Exec(ctx context.Context, in *ExecRequest, out *ExecResponse) error {
+	return h.CommandHandler.Exec(ctx, in, out)
+}

agent/proto/bot.pb.go

@@ -0,0 +1,210 @@
+// Code generated by protoc-gen-go. DO NOT EDIT.
+// source: github.com/micro/go-bot/proto/bot.proto
+
+package go_micro_bot
+
+import proto "github.com/golang/protobuf/proto"
+import fmt "fmt"
+import math "math"
+
+// Reference imports to suppress errors if they are not otherwise used.
+var _ = proto.Marshal
+var _ = fmt.Errorf
+var _ = math.Inf
+
+// This is a compile-time assertion to ensure that this generated file
+// is compatible with the proto package it is being compiled against.
+// A compilation error at this line likely means your copy of the
+// proto package needs to be updated.
+const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package
+
+type HelpRequest struct {
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
+}
+
+func (m *HelpRequest) Reset()         { *m = HelpRequest{} }
+func (m *HelpRequest) String() string { return proto.CompactTextString(m) }
+func (*HelpRequest) ProtoMessage()    {}
+func (*HelpRequest) Descriptor() ([]byte, []int) {
+	return fileDescriptor_bot_654832eab83ed4b5, []int{0}
+}
+func (m *HelpRequest) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_HelpRequest.Unmarshal(m, b)
+}
+func (m *HelpRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_HelpRequest.Marshal(b, m, deterministic)
+}
+func (dst *HelpRequest) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_HelpRequest.Merge(dst, src)
+}
+func (m *HelpRequest) XXX_Size() int {
+	return xxx_messageInfo_HelpRequest.Size(m)
+}
+func (m *HelpRequest) XXX_DiscardUnknown() {
+	xxx_messageInfo_HelpRequest.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_HelpRequest proto.InternalMessageInfo
+
+type HelpResponse struct {
+	Usage                string   `protobuf:"bytes,1,opt,name=usage,proto3" json:"usage,omitempty"`
+	Description          string   `protobuf:"bytes,2,opt,name=description,proto3" json:"description,omitempty"`
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
+}
+
+func (m *HelpResponse) Reset()         { *m = HelpResponse{} }
+func (m *HelpResponse) String() string { return proto.CompactTextString(m) }
+func (*HelpResponse) ProtoMessage()    {}
+func (*HelpResponse) Descriptor() ([]byte, []int) {
+	return fileDescriptor_bot_654832eab83ed4b5, []int{1}
+}
+func (m *HelpResponse) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_HelpResponse.Unmarshal(m, b)
+}
+func (m *HelpResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_HelpResponse.Marshal(b, m, deterministic)
+}
+func (dst *HelpResponse) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_HelpResponse.Merge(dst, src)
+}
+func (m *HelpResponse) XXX_Size() int {
+	return xxx_messageInfo_HelpResponse.Size(m)
+}
+func (m *HelpResponse) XXX_DiscardUnknown() {
+	xxx_messageInfo_HelpResponse.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_HelpResponse proto.InternalMessageInfo
+
+func (m *HelpResponse) GetUsage() string {
+	if m != nil {
+		return m.Usage
+	}
+	return ""
+}
+
+func (m *HelpResponse) GetDescription() string {
+	if m != nil {
+		return m.Description
+	}
+	return ""
+}
+
+type ExecRequest struct {
+	Args                 []string `protobuf:"bytes,1,rep,name=args,proto3" json:"args,omitempty"`
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
+}
+
+func (m *ExecRequest) Reset()         { *m = ExecRequest{} }
+func (m *ExecRequest) String() string { return proto.CompactTextString(m) }
+func (*ExecRequest) ProtoMessage()    {}
+func (*ExecRequest) Descriptor() ([]byte, []int) {
+	return fileDescriptor_bot_654832eab83ed4b5, []int{2}
+}
+func (m *ExecRequest) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_ExecRequest.Unmarshal(m, b)
+}
+func (m *ExecRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_ExecRequest.Marshal(b, m, deterministic)
+}
+func (dst *ExecRequest) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_ExecRequest.Merge(dst, src)
+}
+func (m *ExecRequest) XXX_Size() int {
+	return xxx_messageInfo_ExecRequest.Size(m)
+}
+func (m *ExecRequest) XXX_DiscardUnknown() {
+	xxx_messageInfo_ExecRequest.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_ExecRequest proto.InternalMessageInfo
+
+func (m *ExecRequest) GetArgs() []string {
+	if m != nil {
+		return m.Args
+	}
+	return nil
+}
+
+type ExecResponse struct {
+	Result               []byte   `protobuf:"bytes,1,opt,name=result,proto3" json:"result,omitempty"`
+	Error                string   `protobuf:"bytes,2,opt,name=error,proto3" json:"error,omitempty"`
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
+}
+
+func (m *ExecResponse) Reset()         { *m = ExecResponse{} }
+func (m *ExecResponse) String() string { return proto.CompactTextString(m) }
+func (*ExecResponse) ProtoMessage()    {}
+func (*ExecResponse) Descriptor() ([]byte, []int) {
+	return fileDescriptor_bot_654832eab83ed4b5, []int{3}
+}
+func (m *ExecResponse) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_ExecResponse.Unmarshal(m, b)
+}
+func (m *ExecResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_ExecResponse.Marshal(b, m, deterministic)
+}
+func (dst *ExecResponse) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_ExecResponse.Merge(dst, src)
+}
+func (m *ExecResponse) XXX_Size() int {
+	return xxx_messageInfo_ExecResponse.Size(m)
+}
+func (m *ExecResponse) XXX_DiscardUnknown() {
+	xxx_messageInfo_ExecResponse.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_ExecResponse proto.InternalMessageInfo
+
+func (m *ExecResponse) GetResult() []byte {
+	if m != nil {
+		return m.Result
+	}
+	return nil
+}
+
+func (m *ExecResponse) GetError() string {
+	if m != nil {
+		return m.Error
+	}
+	return ""
+}
+
+func init() {
+	proto.RegisterType((*HelpRequest)(nil), "go.micro.bot.HelpRequest")
+	proto.RegisterType((*HelpResponse)(nil), "go.micro.bot.HelpResponse")
+	proto.RegisterType((*ExecRequest)(nil), "go.micro.bot.ExecRequest")
+	proto.RegisterType((*ExecResponse)(nil), "go.micro.bot.ExecResponse")
+}
+
+func init() {
+	proto.RegisterFile("github.com/micro/go-bot/proto/bot.proto", fileDescriptor_bot_654832eab83ed4b5)
+}
+
+var fileDescriptor_bot_654832eab83ed4b5 = []byte{
+	// 246 bytes of a gzipped FileDescriptorProto
+	0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x6c, 0x90, 0x41, 0x4b, 0xc4, 0x30,
+	0x10, 0x85, 0xb7, 0xba, 0xae, 0xec, 0xb4, 0x5e, 0x82, 0x48, 0xdd, 0x53, 0xcd, 0xc5, 0xbd, 0x98,
+	0x82, 0x5e, 0x05, 0x0f, 0xa2, 0x78, 0xee, 0x3f, 0x68, 0xba, 0x43, 0x2c, 0x6c, 0x3b, 0x35, 0x99,
+	0x82, 0xff, 0xc1, 0x3f, 0x2d, 0x4d, 0x72, 0x08, 0xcb, 0xde, 0xe6, 0x65, 0x86, 0xf7, 0xbe, 0x17,
+	0x78, 0x34, 0x3d, 0x7f, 0xcf, 0x5a, 0x75, 0x34, 0xd4, 0x43, 0xdf, 0x59, 0xaa, 0x0d, 0x3d, 0x69,
+	0xe2, 0x7a, 0xb2, 0xc4, 0x54, 0x6b, 0x62, 0xe5, 0x27, 0x51, 0x18, 0x52, 0xfe, 0x40, 0x69, 0x62,
+	0x79, 0x03, 0xf9, 0x17, 0x1e, 0xa7, 0x06, 0x7f, 0x66, 0x74, 0x2c, 0x3f, 0xa1, 0x08, 0xd2, 0x4d,
+	0x34, 0x3a, 0x14, 0xb7, 0x70, 0x35, 0xbb, 0xd6, 0x60, 0x99, 0x55, 0xd9, 0x7e, 0xdb, 0x04, 0x21,
+	0x2a, 0xc8, 0x0f, 0xe8, 0x3a, 0xdb, 0x4f, 0xdc, 0xd3, 0x58, 0x5e, 0xf8, 0x5d, 0xfa, 0x24, 0x1f,
+	0x20, 0xff, 0xf8, 0xc5, 0x2e, 0xda, 0x0a, 0x01, 0xeb, 0xd6, 0x1a, 0x57, 0x66, 0xd5, 0xe5, 0x7e,
+	0xdb, 0xf8, 0x59, 0xbe, 0x42, 0x11, 0x4e, 0x62, 0xd4, 0x1d, 0x6c, 0x2c, 0xba, 0xf9, 0xc8, 0x3e,
+	0xab, 0x68, 0xa2, 0x5a, 0x10, 0xd0, 0x5a, 0xb2, 0x31, 0x26, 0x88, 0xe7, 0xbf, 0x0c, 0xae, 0xdf,
+	0x69, 0x18, 0xda, 0xf1, 0x20, 0xde, 0x60, 0xbd, 0x40, 0x8b, 0x7b, 0x95, 0x56, 0x53, 0x49, 0xaf,
+	0xdd, 0xee, 0xdc, 0x2a, 0x04, 0xcb, 0xd5, 0x62, 0xb0, 0xa0, 0x9c, 0x1a, 0x24, 0x0d, 0x4e, 0x0d,
+	0x52, 0x72, 0xb9, 0xd2, 0x1b, 0xff, 0xb5, 0x2f, 0xff, 0x01, 0x00, 0x00, 0xff, 0xff, 0xcb, 0x77,
+	0xdf, 0x28, 0x85, 0x01, 0x00, 0x00,
+}

agent/proto/bot.proto

@@ -0,0 +1,25 @@
+syntax = "proto3";
+
+package go.micro.bot;
+
+service Command {
+	rpc Help(HelpRequest) returns (HelpResponse) {};
+	rpc Exec(ExecRequest) returns (ExecResponse) {};
+}
+
+message HelpRequest {
+}
+
+message HelpResponse {
+	string usage = 1;
+	string description = 2;
+}
+
+message ExecRequest {
+	repeated string args = 1;
+}
+
+message ExecResponse {
+	bytes result = 1;
+	string error = 2;
+}

broker/codec/codec.go

@@ -1,10 +0,0 @@
-package codec
-
-// Codec is used for encoding where the broker doesn't natively support
-// headers in the message type. In this case the entire message is
-// encoded as the payload
-type Codec interface {
-	Marshal(interface{}) ([]byte, error)
-	Unmarshal([]byte, interface{}) error
-	String() string
-}

broker/codec/json/json.go

@@ -1,25 +0,0 @@
-package json
-
-import (
-	"encoding/json"
-
-	"github.com/micro/go-micro/broker/codec"
-)
-
-type jsonCodec struct{}
-
-func (j jsonCodec) Marshal(v interface{}) ([]byte, error) {
-	return json.Marshal(v)
-}
-
-func (j jsonCodec) Unmarshal(d []byte, v interface{}) error {
-	return json.Unmarshal(d, v)
-}
-
-func (j jsonCodec) String() string {
-	return "json"
-}
-
-func NewCodec() codec.Codec {
-	return jsonCodec{}
-}

broker/codec/noop/noop.go

@@ -1,35 +0,0 @@
-package noop
-
-import (
-	"errors"
-
-	"github.com/micro/go-micro/broker"
-	"github.com/micro/go-micro/broker/codec"
-)
-
-type noopCodec struct{}
-
-func (n noopCodec) Marshal(v interface{}) ([]byte, error) {
-	msg, ok := v.(*broker.Message)
-	if !ok {
-		return nil, errors.New("invalid message")
-	}
-	return msg.Body, nil
-}
-
-func (n noopCodec) Unmarshal(d []byte, v interface{}) error {
-	msg, ok := v.(*broker.Message)
-	if !ok {
-		return errors.New("invalid message")
-	}
-	msg.Body = d
-	return nil
-}
-
-func (n noopCodec) String() string {
-	return "noop"
-}
-
-func NewCodec() codec.Codec {
-	return noopCodec{}
-}

broker/http_broker.go

@@ -19,14 +19,13 @@ import (
 	"time"
 
 	"github.com/google/uuid"
-	"github.com/micro/go-log"
-	"github.com/micro/go-micro/broker/codec/json"
+	"github.com/micro/go-micro/codec/json"
 	merr "github.com/micro/go-micro/errors"
 	"github.com/micro/go-micro/registry"
-	"github.com/micro/go-rcache"
-	maddr "github.com/micro/util/go/lib/addr"
-	mnet "github.com/micro/util/go/lib/net"
-	mls "github.com/micro/util/go/lib/tls"
+	maddr "github.com/micro/go-micro/util/addr"
+	mnet "github.com/micro/go-micro/util/net"
+	mls "github.com/micro/go-micro/util/tls"
+	"github.com/micro/go-micro/registry/cache"
 	"golang.org/x/net/http2"
 )
 
@@ -45,6 +44,10 @@ type httpBroker struct {
 	subscribers map[string][]*httpSubscriber
 	running     bool
 	exit        chan chan error
+
+	// offline message inbox
+	mtx   sync.RWMutex
+	inbox map[string][][]byte
 }
 
 type httpSubscriber struct {
@@ -104,7 +107,7 @@ func newTransport(config *tls.Config) *http.Transport {
 
 func newHttpBroker(opts ...Option) Broker {
 	options := Options{
-		Codec:   json.NewCodec(),
+		Codec:   json.Marshaler{},
 		Context: context.TODO(),
 	}
 
@@ -133,6 +136,7 @@ func newHttpBroker(opts ...Option) Broker {
 		subscribers: make(map[string][]*httpSubscriber),
 		exit:        make(chan chan error),
 		mux:         http.NewServeMux(),
+		inbox:       make(map[string][][]byte),
 	}
 
 	// specify the message handler
@@ -175,6 +179,49 @@ func (h *httpSubscriber) Unsubscribe() error {
 	return h.hb.unsubscribe(h)
 }
 
+func (h *httpBroker) saveMessage(topic string, msg []byte) {
+	h.mtx.Lock()
+	defer h.mtx.Unlock()
+
+	// get messages
+	c := h.inbox[topic]
+
+	// save message
+	c = append(c, msg)
+
+	// max length 64
+	if len(c) > 64 {
+		c = c[:64]
+	}
+
+	// save inbox
+	h.inbox[topic] = c
+}
+
+func (h *httpBroker) getMessage(topic string, num int) [][]byte {
+	h.mtx.Lock()
+	defer h.mtx.Unlock()
+
+	// get messages
+	c, ok := h.inbox[topic]
+	if !ok {
+		return nil
+	}
+
+	// more message than requests
+	if len(c) >= num {
+		msg := c[:num]
+		h.inbox[topic] = c[num:]
+		return msg
+	}
+
+	// reset inbox
+	h.inbox[topic] = nil
+
+	// return all messages
+	return c
+}
+
 func (h *httpBroker) subscribe(s *httpSubscriber) error {
 	h.Lock()
 	defer h.Unlock()
@@ -349,7 +396,6 @@ func (h *httpBroker) Connect() error {
 		return err
 	}
 
-	log.Logf("Broker Listening on %s", l.Addr().String())
 	addr := h.address
 	h.address = l.Addr().String()
 
@@ -366,8 +412,8 @@ func (h *httpBroker) Connect() error {
 	if !ok {
 		reg = registry.DefaultRegistry
 	}
-	// set rcache
-	h.r = rcache.New(reg)
+	// set cache
+	h.r = cache.New(reg)
 
 	// set running
 	h.running = true
@@ -386,8 +432,8 @@ func (h *httpBroker) Disconnect() error {
 	h.Lock()
 	defer h.Unlock()
 
-	// stop rcache
-	rc, ok := h.r.(rcache.Cache)
+	// stop cache
+	rc, ok := h.r.(cache.Cache)
 	if ok {
 		rc.Stop()
 	}
@@ -431,13 +477,13 @@ func (h *httpBroker) Init(opts ...Option) error {
 		reg = registry.DefaultRegistry
 	}
 
-	// get rcache
-	if rc, ok := h.r.(rcache.Cache); ok {
+	// get cache
+	if rc, ok := h.r.(cache.Cache); ok {
 		rc.Stop()
 	}
 
 	// set registry
-	h.r = rcache.New(reg)
+	h.r = cache.New(reg)
 
 	// reconfigure tls config
 	if c := h.opts.TLSConfig; c != nil {
@@ -454,14 +500,7 @@ func (h *httpBroker) Options() Options {
 }
 
 func (h *httpBroker) Publish(topic string, msg *Message, opts ...PublishOption) error {
-	h.RLock()
-	s, err := h.r.GetService("topic:" + topic)
-	if err != nil {
-		h.RUnlock()
-		return err
-	}
-	h.RUnlock()
-
+	// create the message first
 	m := &Message{
 		Header: make(map[string]string),
 		Body:   msg.Body,
@@ -473,12 +512,26 @@ func (h *httpBroker) Publish(topic string, msg *Message, opts ...PublishOption)
 
 	m.Header[":topic"] = topic
 
+	// encode the message
 	b, err := h.opts.Codec.Marshal(m)
 	if err != nil {
 		return err
 	}
 
-	pub := func(node *registry.Node, b []byte) {
+	// save the message
+	h.saveMessage(topic, b)
+
+	// now attempt to get the service
+	h.RLock()
+	s, err := h.r.GetService("topic:" + topic)
+	if err != nil {
+		h.RUnlock()
+		// ignore error
+		return nil
+	}
+	h.RUnlock()
+
+	pub := func(node *registry.Node, t string, b []byte) error {
 		scheme := "http"
 
 		// check if secure is added in metadata
@@ -491,34 +544,71 @@ func (h *httpBroker) Publish(topic string, msg *Message, opts ...PublishOption)
 
 		uri := fmt.Sprintf("%s://%s:%d%s?%s", scheme, node.Address, node.Port, DefaultSubPath, vals.Encode())
 		r, err := h.c.Post(uri, "application/json", bytes.NewReader(b))
-		if err == nil {
-			io.Copy(ioutil.Discard, r.Body)
-			r.Body.Close()
+		if err != nil {
+			return err
 		}
+
+		// discard response body
+		io.Copy(ioutil.Discard, r.Body)
+		r.Body.Close()
+		return nil
 	}
 
-	for _, service := range s {
-		// only process if we have nodes
-		if len(service.Nodes) == 0 {
-			continue
-		}
-
-		switch service.Version {
-		// broadcast version means broadcast to all nodes
-		case broadcastVersion:
-			for _, node := range service.Nodes {
-				// publish async
-				go pub(node, b)
+	srv := func(s []*registry.Service, b []byte) {
+		for _, service := range s {
+			// only process if we have nodes
+			if len(service.Nodes) == 0 {
+				continue
 			}
-		default:
-			// select node to publish to
-			node := service.Nodes[rand.Int()%len(service.Nodes)]
 
-			// publish async
-			go pub(node, b)
+			switch service.Version {
+			// broadcast version means broadcast to all nodes
+			case broadcastVersion:
+				var success bool
+
+				// publish to all nodes
+				for _, node := range service.Nodes {
+					// publish async
+					if err := pub(node, topic, b); err == nil {
+						success = true
+					}
+				}
+
+				// save if it failed to publish at least once
+				if !success {
+					h.saveMessage(topic, b)
+				}
+			default:
+				// select node to publish to
+				node := service.Nodes[rand.Int()%len(service.Nodes)]
+
+				// publish async to one node
+				if err := pub(node, topic, b); err != nil {
+					// if failed save it
+					h.saveMessage(topic, b)
+				}
+			}
 		}
 	}
 
+	// do the rest async
+	go func() {
+		// get a third of the backlog
+		messages := h.getMessage(topic, 8)
+		delay := (len(messages) > 1)
+
+		// publish all the messages
+		for _, msg := range messages {
+			// serialize here
+			srv(s, msg)
+
+			// sending a backlog of messages
+			if delay {
+				time.Sleep(time.Millisecond * 100)
+			}
+		}
+	}()
+
 	return nil
 }
 

broker/http_broker_test.go

@@ -5,13 +5,26 @@ import (
 	"testing"
 	"time"
 
+	glog "github.com/go-log/log"
 	"github.com/google/uuid"
-	"github.com/micro/go-micro/registry/mock"
+	"github.com/micro/go-micro/registry/memory"
+	"github.com/micro/go-micro/util/log"
 )
 
+func newTestRegistry() *memory.Registry {
+	r := memory.NewRegistry()
+	m := r.(*memory.Registry)
+	m.Setup()
+	return m
+}
+
 func sub(be *testing.B, c int) {
+	// set no op logger
+	log.SetLogger(glog.DefaultLogger)
+
 	be.StopTimer()
-	m := mock.NewRegistry()
+	m := newTestRegistry()
+
 	b := NewBroker(Registry(m))
 	topic := uuid.New().String()
 
@@ -69,8 +82,11 @@ func sub(be *testing.B, c int) {
 }
 
 func pub(be *testing.B, c int) {
+	// set no op logger
+	log.SetLogger(glog.DefaultLogger)
+
 	be.StopTimer()
-	m := mock.NewRegistry()
+	m := newTestRegistry()
 	b := NewBroker(Registry(m))
 	topic := uuid.New().String()
 
@@ -139,7 +155,7 @@ func pub(be *testing.B, c int) {
 }
 
 func TestBroker(t *testing.T) {
-	m := mock.NewRegistry()
+	m := newTestRegistry()
 	b := NewBroker(Registry(m))
 
 	if err := b.Init(); err != nil {
@@ -186,7 +202,7 @@ func TestBroker(t *testing.T) {
 }
 
 func TestConcurrentSubBroker(t *testing.T) {
-	m := mock.NewRegistry()
+	m := newTestRegistry()
 	b := NewBroker(Registry(m))
 
 	if err := b.Init(); err != nil {
@@ -243,7 +259,7 @@ func TestConcurrentSubBroker(t *testing.T) {
 }
 
 func TestConcurrentPubBroker(t *testing.T) {
-	m := mock.NewRegistry()
+	m := newTestRegistry()
 	b := NewBroker(Registry(m))
 
 	if err := b.Init(); err != nil {

broker/memory/memory.go

@@ -1,5 +1,5 @@
-// Package mock provides a mock broker for testing
-package mock
+// Package memory provides a memory broker
+package memory
 
 import (
 	"errors"
@@ -9,20 +9,20 @@ import (
 	"github.com/micro/go-micro/broker"
 )
 
-type mockBroker struct {
+type memoryBroker struct {
 	opts broker.Options
 
 	sync.RWMutex
 	connected   bool
-	Subscribers map[string][]*mockSubscriber
+	Subscribers map[string][]*memorySubscriber
 }
 
-type mockPublication struct {
+type memoryPublication struct {
 	topic   string
 	message *broker.Message
 }
 
-type mockSubscriber struct {
+type memorySubscriber struct {
 	id      string
 	topic   string
 	exit    chan bool
@@ -30,15 +30,15 @@ type mockSubscriber struct {
 	opts    broker.SubscribeOptions
 }
 
-func (m *mockBroker) Options() broker.Options {
+func (m *memoryBroker) Options() broker.Options {
 	return m.opts
 }
 
-func (m *mockBroker) Address() string {
+func (m *memoryBroker) Address() string {
 	return ""
 }
 
-func (m *mockBroker) Connect() error {
+func (m *memoryBroker) Connect() error {
 	m.Lock()
 	defer m.Unlock()
 
@@ -51,7 +51,7 @@ func (m *mockBroker) Connect() error {
 	return nil
 }
 
-func (m *mockBroker) Disconnect() error {
+func (m *memoryBroker) Disconnect() error {
 	m.Lock()
 	defer m.Unlock()
 
@@ -64,14 +64,14 @@ func (m *mockBroker) Disconnect() error {
 	return nil
 }
 
-func (m *mockBroker) Init(opts ...broker.Option) error {
+func (m *memoryBroker) Init(opts ...broker.Option) error {
 	for _, o := range opts {
 		o(&m.opts)
 	}
 	return nil
 }
 
-func (m *mockBroker) Publish(topic string, message *broker.Message, opts ...broker.PublishOption) error {
+func (m *memoryBroker) Publish(topic string, message *broker.Message, opts ...broker.PublishOption) error {
 	m.Lock()
 	defer m.Unlock()
 
@@ -84,7 +84,7 @@ func (m *mockBroker) Publish(topic string, message *broker.Message, opts ...brok
 		return nil
 	}
 
-	p := &mockPublication{
+	p := &memoryPublication{
 		topic:   topic,
 		message: message,
 	}
@@ -98,7 +98,7 @@ func (m *mockBroker) Publish(topic string, message *broker.Message, opts ...brok
 	return nil
 }
 
-func (m *mockBroker) Subscribe(topic string, handler broker.Handler, opts ...broker.SubscribeOption) (broker.Subscriber, error) {
+func (m *memoryBroker) Subscribe(topic string, handler broker.Handler, opts ...broker.SubscribeOption) (broker.Subscriber, error) {
 	m.Lock()
 	defer m.Unlock()
 
@@ -111,7 +111,7 @@ func (m *mockBroker) Subscribe(topic string, handler broker.Handler, opts ...bro
 		o(&options)
 	}
 
-	sub := &mockSubscriber{
+	sub := &memorySubscriber{
 		exit:    make(chan bool, 1),
 		id:      uuid.New().String(),
 		topic:   topic,
@@ -124,7 +124,7 @@ func (m *mockBroker) Subscribe(topic string, handler broker.Handler, opts ...bro
 	go func() {
 		<-sub.exit
 		m.Lock()
-		var newSubscribers []*mockSubscriber
+		var newSubscribers []*memorySubscriber
 		for _, sb := range m.Subscribers[topic] {
 			if sb.id == sub.id {
 				continue
@@ -138,31 +138,31 @@ func (m *mockBroker) Subscribe(topic string, handler broker.Handler, opts ...bro
 	return sub, nil
 }
 
-func (m *mockBroker) String() string {
-	return "mock"
+func (m *memoryBroker) String() string {
+	return "memory"
 }
 
-func (m *mockPublication) Topic() string {
+func (m *memoryPublication) Topic() string {
 	return m.topic
 }
 
-func (m *mockPublication) Message() *broker.Message {
+func (m *memoryPublication) Message() *broker.Message {
 	return m.message
 }
 
-func (m *mockPublication) Ack() error {
+func (m *memoryPublication) Ack() error {
 	return nil
 }
 
-func (m *mockSubscriber) Options() broker.SubscribeOptions {
+func (m *memorySubscriber) Options() broker.SubscribeOptions {
 	return m.opts
 }
 
-func (m *mockSubscriber) Topic() string {
+func (m *memorySubscriber) Topic() string {
 	return m.topic
 }
 
-func (m *mockSubscriber) Unsubscribe() error {
+func (m *memorySubscriber) Unsubscribe() error {
 	m.exit <- true
 	return nil
 }
@@ -173,8 +173,8 @@ func NewBroker(opts ...broker.Option) broker.Broker {
 		o(&options)
 	}
 
-	return &mockBroker{
+	return &memoryBroker{
 		opts:        options,
-		Subscribers: make(map[string][]*mockSubscriber),
+		Subscribers: make(map[string][]*memorySubscriber),
 	}
 }

broker/memory/memory_test.go

@@ -1,4 +1,4 @@
-package mock
+package memory
 
 import (
 	"fmt"
@@ -7,7 +7,7 @@ import (
 	"github.com/micro/go-micro/broker"
 )
 
-func TestBroker(t *testing.T) {
+func TestMemoryBroker(t *testing.T) {
 	b := NewBroker()
 
 	if err := b.Connect(); err != nil {

broker/nats/context.go

@@ -0,0 +1,37 @@
+package nats
+
+import (
+	"context"
+
+	"github.com/micro/go-micro/broker"
+)
+
+// setSubscribeOption returns a function to setup a context with given value
+func setSubscribeOption(k, v interface{}) broker.SubscribeOption {
+	return func(o *broker.SubscribeOptions) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, k, v)
+	}
+}
+
+// setBrokerOption returns a function to setup a context with given value
+func setBrokerOption(k, v interface{}) broker.Option {
+	return func(o *broker.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, k, v)
+	}
+}
+
+// setPublishOption returns a function to setup a context with given value
+func setPublishOption(k, v interface{}) broker.PublishOption {
+	return func(o *broker.PublishOptions) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, k, v)
+	}
+}

broker/nats/nats.go

@@ -0,0 +1,245 @@
+// Package nats provides a NATS broker
+package nats
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"sync"
+
+	"github.com/micro/go-micro/broker"
+	"github.com/micro/go-micro/codec/json"
+	nats "github.com/nats-io/nats.go"
+)
+
+type natsBroker struct {
+	sync.RWMutex
+	addrs []string
+	conn  *nats.Conn
+	opts  broker.Options
+	nopts nats.Options
+	drain bool
+}
+
+type subscriber struct {
+	s     *nats.Subscription
+	opts  broker.SubscribeOptions
+	drain bool
+}
+
+type publication struct {
+	t string
+	m *broker.Message
+}
+
+func (p *publication) Topic() string {
+	return p.t
+}
+
+func (p *publication) Message() *broker.Message {
+	return p.m
+}
+
+func (p *publication) Ack() error {
+	// nats does not support acking
+	return nil
+}
+
+func (s *subscriber) Options() broker.SubscribeOptions {
+	return s.opts
+}
+
+func (s *subscriber) Topic() string {
+	return s.s.Subject
+}
+
+func (s *subscriber) Unsubscribe() error {
+	if s.drain {
+		return s.s.Drain()
+	}
+	return s.s.Unsubscribe()
+}
+
+func (n *natsBroker) Address() string {
+	if n.conn != nil && n.conn.IsConnected() {
+		return n.conn.ConnectedUrl()
+	}
+	if len(n.addrs) > 0 {
+		return n.addrs[0]
+	}
+
+	return ""
+}
+
+func setAddrs(addrs []string) []string {
+	var cAddrs []string
+	for _, addr := range addrs {
+		if len(addr) == 0 {
+			continue
+		}
+		if !strings.HasPrefix(addr, "nats://") {
+			addr = "nats://" + addr
+		}
+		cAddrs = append(cAddrs, addr)
+	}
+	if len(cAddrs) == 0 {
+		cAddrs = []string{nats.DefaultURL}
+	}
+	return cAddrs
+}
+
+func (n *natsBroker) Connect() error {
+	n.Lock()
+	defer n.Unlock()
+
+	status := nats.CLOSED
+	if n.conn != nil {
+		status = n.conn.Status()
+	}
+
+	switch status {
+	case nats.CONNECTED, nats.RECONNECTING, nats.CONNECTING:
+		return nil
+	default: // DISCONNECTED or CLOSED or DRAINING
+		opts := n.nopts
+		opts.Servers = n.addrs
+		opts.Secure = n.opts.Secure
+		opts.TLSConfig = n.opts.TLSConfig
+
+		// secure might not be set
+		if n.opts.TLSConfig != nil {
+			opts.Secure = true
+		}
+
+		c, err := opts.Connect()
+		if err != nil {
+			return err
+		}
+		n.conn = c
+		return nil
+	}
+}
+
+func (n *natsBroker) Disconnect() error {
+	n.RLock()
+	if n.drain {
+		n.conn.Drain()
+	} else {
+		n.conn.Close()
+	}
+	n.RUnlock()
+	return nil
+}
+
+func (n *natsBroker) Init(opts ...broker.Option) error {
+	for _, o := range opts {
+		o(&n.opts)
+	}
+	n.addrs = setAddrs(n.opts.Addrs)
+	return nil
+}
+
+func (n *natsBroker) Options() broker.Options {
+	return n.opts
+}
+
+func (n *natsBroker) Publish(topic string, msg *broker.Message, opts ...broker.PublishOption) error {
+	b, err := n.opts.Codec.Marshal(msg)
+	if err != nil {
+		return err
+	}
+	n.RLock()
+	defer n.RUnlock()
+	return n.conn.Publish(topic, b)
+}
+
+func (n *natsBroker) Subscribe(topic string, handler broker.Handler, opts ...broker.SubscribeOption) (broker.Subscriber, error) {
+	if n.conn == nil {
+		return nil, errors.New("not connected")
+	}
+
+	opt := broker.SubscribeOptions{
+		AutoAck: true,
+		Context: context.Background(),
+	}
+
+	for _, o := range opts {
+		o(&opt)
+	}
+
+	var drain bool
+	if _, ok := opt.Context.Value(drainSubscriptionKey{}).(bool); ok {
+		drain = true
+	}
+
+	fn := func(msg *nats.Msg) {
+		var m broker.Message
+		if err := n.opts.Codec.Unmarshal(msg.Data, &m); err != nil {
+			return
+		}
+		handler(&publication{m: &m, t: msg.Subject})
+	}
+
+	var sub *nats.Subscription
+	var err error
+
+	n.RLock()
+	if len(opt.Queue) > 0 {
+		sub, err = n.conn.QueueSubscribe(topic, opt.Queue, fn)
+	} else {
+		sub, err = n.conn.Subscribe(topic, fn)
+	}
+	n.RUnlock()
+	if err != nil {
+		return nil, err
+	}
+	return &subscriber{s: sub, opts: opt, drain: drain}, nil
+}
+
+func (n *natsBroker) String() string {
+	return "nats"
+}
+
+func NewBroker(opts ...broker.Option) broker.Broker {
+	options := broker.Options{
+		// Default codec
+		Codec:   json.Marshaler{},
+		Context: context.Background(),
+	}
+
+	for _, o := range opts {
+		o(&options)
+	}
+
+	natsOpts := nats.GetDefaultOptions()
+	if n, ok := options.Context.Value(optionsKey{}).(nats.Options); ok {
+		natsOpts = n
+	}
+
+	var drain bool
+	if _, ok := options.Context.Value(drainSubscriptionKey{}).(bool); ok {
+		drain = true
+	}
+
+	// broker.Options have higher priority than nats.Options
+	// only if Addrs, Secure or TLSConfig were not set through a broker.Option
+	// we read them from nats.Option
+	if len(options.Addrs) == 0 {
+		options.Addrs = natsOpts.Servers
+	}
+
+	if !options.Secure {
+		options.Secure = natsOpts.Secure
+	}
+
+	if options.TLSConfig == nil {
+		options.TLSConfig = natsOpts.TLSConfig
+	}
+
+	return &natsBroker{
+		opts:  options,
+		nopts: natsOpts,
+		addrs: setAddrs(options.Addrs),
+		drain: drain,
+	}
+}

broker/nats/nats_test.go

@@ -0,0 +1,99 @@
+package nats
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/micro/go-micro/broker"
+	nats "github.com/nats-io/nats.go"
+)
+
+var addrTestCases = []struct {
+	name        string
+	description string
+	addrs       map[string]string // expected address : set address
+}{
+	{
+		"brokerOpts",
+		"set broker addresses through a broker.Option in constructor",
+		map[string]string{
+			"nats://192.168.10.1:5222": "192.168.10.1:5222",
+			"nats://10.20.10.0:4222":   "10.20.10.0:4222"},
+	},
+	{
+		"brokerInit",
+		"set broker addresses through a broker.Option in broker.Init()",
+		map[string]string{
+			"nats://192.168.10.1:5222": "192.168.10.1:5222",
+			"nats://10.20.10.0:4222":   "10.20.10.0:4222"},
+	},
+	{
+		"natsOpts",
+		"set broker addresses through the nats.Option in constructor",
+		map[string]string{
+			"nats://192.168.10.1:5222": "192.168.10.1:5222",
+			"nats://10.20.10.0:4222":   "10.20.10.0:4222"},
+	},
+	{
+		"default",
+		"check if default Address is set correctly",
+		map[string]string{
+			"nats://localhost:4222": "",
+		},
+	},
+}
+
+// TestInitAddrs tests issue #100. Ensures that if the addrs is set by an option in init it will be used.
+func TestInitAddrs(t *testing.T) {
+
+	for _, tc := range addrTestCases {
+		t.Run(fmt.Sprintf("%s: %s", tc.name, tc.description), func(t *testing.T) {
+
+			var br broker.Broker
+			var addrs []string
+
+			for _, addr := range tc.addrs {
+				addrs = append(addrs, addr)
+			}
+
+			switch tc.name {
+			case "brokerOpts":
+				// we know that there are just two addrs in the dict
+				br = NewBroker(broker.Addrs(addrs[0], addrs[1]))
+				br.Init()
+			case "brokerInit":
+				br = NewBroker()
+				// we know that there are just two addrs in the dict
+				br.Init(broker.Addrs(addrs[0], addrs[1]))
+			case "natsOpts":
+				nopts := nats.GetDefaultOptions()
+				nopts.Servers = addrs
+				br = NewBroker(Options(nopts))
+				br.Init()
+			case "default":
+				br = NewBroker()
+				br.Init()
+			}
+
+			natsBroker, ok := br.(*natsBroker)
+			if !ok {
+				t.Fatal("Expected broker to be of types *natsBroker")
+			}
+			// check if the same amount of addrs we set has actually been set, default
+			// have only 1 address nats://127.0.0.1:4222 (current nats code) or
+			// nats://localhost:4222 (older code version)
+			if len(natsBroker.addrs) != len(tc.addrs) && tc.name != "default" {
+				t.Errorf("Expected Addr count = %d, Actual Addr count = %d",
+					len(natsBroker.addrs), len(tc.addrs))
+			}
+
+			for _, addr := range natsBroker.addrs {
+				_, ok := tc.addrs[addr]
+				if !ok {
+					t.Errorf("Expected '%s' has not been set", addr)
+				}
+			}
+		})
+
+	}
+}

broker/nats/options.go

@@ -0,0 +1,25 @@
+package nats
+
+import (
+	"github.com/micro/go-micro/broker"
+	nats "github.com/nats-io/nats.go"
+)
+
+type optionsKey struct{}
+type drainConnectionKey struct{}
+type drainSubscriptionKey struct{}
+
+// Options accepts nats.Options
+func Options(opts nats.Options) broker.Option {
+	return setBrokerOption(optionsKey{}, opts)
+}
+
+// DrainConnection will drain subscription on close
+func DrainConnection() broker.Option {
+	return setBrokerOption(drainConnectionKey{}, true)
+}
+
+// DrainSubscription will drain pending messages when unsubscribe
+func DrainSubscription() broker.SubscribeOption {
+	return setSubscribeOption(drainSubscriptionKey{}, true)
+}

broker/options.go

@@ -4,14 +4,14 @@ import (
 	"context"
 	"crypto/tls"
 
-	"github.com/micro/go-micro/broker/codec"
+	"github.com/micro/go-micro/codec"
 	"github.com/micro/go-micro/registry"
 )
 
 type Options struct {
 	Addrs     []string
 	Secure    bool
-	Codec     codec.Codec
+	Codec     codec.Marshaler
 	TLSConfig *tls.Config
 	// Other options for implementations of the interface
 	// can be stored in a context
@@ -44,10 +44,8 @@ type PublishOption func(*PublishOptions)
 
 type SubscribeOption func(*SubscribeOptions)
 
-type contextKeyT string
-
 var (
-	registryKey = contextKeyT("github.com/micro/go-micro/registry")
+	registryKey = "github.com/micro/go-micro/registry"
 )
 
 func NewSubscribeOptions(opts ...SubscribeOption) SubscribeOptions {
@@ -71,7 +69,7 @@ func Addrs(addrs ...string) Option {
 
 // Codec sets the codec used for encoding/decoding used where
 // a broker does not support headers
-func Codec(c codec.Codec) Option {
+func Codec(c codec.Marshaler) Option {
 	return func(o *Options) {
 		o.Codec = c
 	}
@@ -111,3 +109,10 @@ func TLSConfig(t *tls.Config) Option {
 		o.TLSConfig = t
 	}
 }
+
+// SubscribeContext set context
+func SubscribeContext(ctx context.Context) SubscribeOption {
+	return func(o *SubscribeOptions) {
+		o.Context = ctx
+	}
+}

client/client.go

@@ -4,6 +4,8 @@ package client
 import (
 	"context"
 	"time"
+
+	"github.com/micro/go-micro/codec"
 )
 
 // Client is the interface used to make requests to services.
@@ -13,13 +15,18 @@ type Client interface {
 	Init(...Option) error
 	Options() Options
 	NewMessage(topic string, msg interface{}, opts ...MessageOption) Message
-	NewRequest(service, method string, req interface{}, reqOpts ...RequestOption) Request
+	NewRequest(service, endpoint string, req interface{}, reqOpts ...RequestOption) Request
 	Call(ctx context.Context, req Request, rsp interface{}, opts ...CallOption) error
 	Stream(ctx context.Context, req Request, opts ...CallOption) (Stream, error)
 	Publish(ctx context.Context, msg Message, opts ...PublishOption) error
 	String() string
 }
 
+// Router manages request routing
+type Router interface {
+	SendRequest(context.Context, Request) (Response, error)
+}
+
 // Message is the interface for publishing asynchronously
 type Message interface {
 	Topic() string
@@ -29,21 +36,47 @@ type Message interface {
 
 // Request is the interface for a synchronous request used by Call or Stream
 type Request interface {
+	// The service to call
 	Service() string
+	// The action to take
 	Method() string
+	// The endpoint to invoke
+	Endpoint() string
+	// The content type
 	ContentType() string
-	Request() interface{}
+	// The unencoded request body
+	Body() interface{}
+	// Write to the encoded request writer. This is nil before a call is made
+	Codec() codec.Writer
 	// indicates whether the request will be a streaming one rather than unary
 	Stream() bool
 }
 
+// Response is the response received from a service
+type Response interface {
+	// Read the response
+	Codec() codec.Reader
+	// read the header
+	Header() map[string]string
+	// Read the undecoded response
+	Read() ([]byte, error)
+}
+
 // Stream is the inteface for a bidirectional synchronous stream
 type Stream interface {
+	// Context for the stream
 	Context() context.Context
+	// The request made
 	Request() Request
+	// The response read
+	Response() Response
+	// Send will encode and send a request
 	Send(interface{}) error
+	// Recv will decode and read a response
 	Recv(interface{}) error
+	// Error returns the stream error
 	Error() error
+	// Close closes the stream
 	Close() error
 }
 
@@ -74,7 +107,7 @@ var (
 	// DefaultRequestTimeout is the default request timeout
 	DefaultRequestTimeout = time.Second * 5
 	// DefaultPoolSize sets the connection pool size
-	DefaultPoolSize = 1
+	DefaultPoolSize = 100
 	// DefaultPoolTTL sets the connection pool ttl
 	DefaultPoolTTL = time.Minute
 )
@@ -102,8 +135,8 @@ func NewClient(opt ...Option) Client {
 
 // Creates a new request using the default client. Content Type will
 // be set to the default within options and use the appropriate codec
-func NewRequest(service, method string, request interface{}, reqOpts ...RequestOption) Request {
-	return DefaultClient.NewRequest(service, method, request, reqOpts...)
+func NewRequest(service, endpoint string, request interface{}, reqOpts ...RequestOption) Request {
+	return DefaultClient.NewRequest(service, endpoint, request, reqOpts...)
 }
 
 // Creates a streaming connection with a service and returns responses on the

client/mock/mock.go

@@ -16,7 +16,7 @@ var (
 )
 
 type MockResponse struct {
-	Method   string
+	Endpoint string
 	Response interface{}
 	Error    error
 }
@@ -54,8 +54,8 @@ func (m *MockClient) NewMessage(topic string, msg interface{}, opts ...client.Me
 	return m.Client.NewMessage(topic, msg, opts...)
 }
 
-func (m *MockClient) NewRequest(service, method string, req interface{}, reqOpts ...client.RequestOption) client.Request {
-	return m.Client.NewRequest(service, method, req, reqOpts...)
+func (m *MockClient) NewRequest(service, endpoint string, req interface{}, reqOpts ...client.RequestOption) client.Request {
+	return m.Client.NewRequest(service, endpoint, req, reqOpts...)
 }
 
 func (m *MockClient) Call(ctx context.Context, req client.Request, rsp interface{}, opts ...client.CallOption) error {
@@ -68,7 +68,7 @@ func (m *MockClient) Call(ctx context.Context, req client.Request, rsp interface
 	}
 
 	for _, r := range response {
-		if r.Method != req.Method() {
+		if r.Endpoint != req.Endpoint() {
 			continue
 		}
 
@@ -83,7 +83,11 @@ func (m *MockClient) Call(ctx context.Context, req client.Request, rsp interface
 		}
 		response := r.Response
 		if t := reflect.TypeOf(r.Response); t.Kind() == reflect.Func {
-			response = reflect.ValueOf(r.Response).Call([]reflect.Value{})[0].Interface()
+			var request []reflect.Value
+			if t.NumIn() == 1 {
+				request = append(request, reflect.ValueOf(req.Body()))
+			}
+			response = reflect.ValueOf(r.Response).Call(request)[0].Interface()
 		}
 
 		v.Set(reflect.ValueOf(response))
@@ -91,7 +95,7 @@ func (m *MockClient) Call(ctx context.Context, req client.Request, rsp interface
 		return nil
 	}
 
-	return fmt.Errorf("rpc: can't find service %s", req.Method())
+	return fmt.Errorf("rpc: can't find service %s", req.Endpoint())
 }
 
 func (m *MockClient) Stream(ctx context.Context, req client.Request, opts ...client.CallOption) (client.Stream, error) {

client/mock/mock_test.go

@@ -13,17 +13,23 @@ func TestClient(t *testing.T) {
 	}
 
 	response := []MockResponse{
-		{Method: "Foo.Bar", Response: map[string]interface{}{"foo": "bar"}},
-		{Method: "Foo.Struct", Response: &TestResponse{Param: "aparam"}},
-		{Method: "Foo.Fail", Error: errors.InternalServerError("go.mock", "failed")},
-		{Method: "Foo.Func", Response: func() string { return "string" }},
-		{Method: "Foo.FuncStruct", Response: func() *TestResponse { return &TestResponse{Param: "aparam"} }},
+		{Endpoint: "Foo.Bar", Response: map[string]interface{}{"foo": "bar"}},
+		{Endpoint: "Foo.Struct", Response: &TestResponse{Param: "aparam"}},
+		{Endpoint: "Foo.Fail", Error: errors.InternalServerError("go.mock", "failed")},
+		{Endpoint: "Foo.Func", Response: func() string { return "string" }},
+		{Endpoint: "Foo.FuncStruct", Response: func() *TestResponse { return &TestResponse{Param: "aparam"} }},
+		{Endpoint: "Foo.FuncWithReqBody", Response: func(req interface{}) string {
+			if req.(map[string]string)["foo"] == "bar" {
+				return "string"
+			}
+			return "wrong"
+		}},
 	}
 
 	c := NewClient(Response("go.mock", response))
 
 	for _, r := range response {
-		req := c.NewRequest("go.mock", r.Method, map[string]interface{}{"foo": "bar"})
+		req := c.NewRequest("go.mock", r.Endpoint, map[string]string{"foo": "bar"})
 		var rsp interface{}
 
 		err := c.Call(context.TODO(), req, &rsp)
@@ -33,6 +39,20 @@ func TestClient(t *testing.T) {
 		}
 
 		t.Log(rsp)
+		if r.Endpoint == "Foo.FuncWithReqBody" {
+			req := c.NewRequest("go.mock", r.Endpoint, map[string]string{"foo": "wrong"})
+			var rsp interface{}
+
+			err := c.Call(context.TODO(), req, &rsp)
+
+			if err != r.Error {
+				t.Fatalf("Expecter error %v got %v", r.Error, err)
+			}
+			if rsp.(string) != "wrong" {
+				t.Fatalf("Expecter response 'wrong' got %v", rsp)
+			}
+			t.Log(rsp)
+		}
 	}
 
 }

client/options.go

@@ -22,6 +22,9 @@ type Options struct {
 	Selector  selector.Selector
 	Transport transport.Transport
 
+	// Router sets the router
+	Router Router
+
 	// Connection Pool
 	PoolSize int
 	PoolTTL  time.Duration
@@ -62,6 +65,8 @@ type CallOptions struct {
 }
 
 type PublishOptions struct {
+	// Exchange is the routing exchange for the message
+	Exchange string
 	// Other options for implementations of the interface
 	// can be stored in a context
 	Context context.Context
@@ -99,7 +104,7 @@ func newOptions(options ...Option) Options {
 	}
 
 	if len(opts.ContentType) == 0 {
-		opts.ContentType = defaultContentType
+		opts.ContentType = DefaultContentType
 	}
 
 	if opts.Broker == nil {
@@ -233,6 +238,13 @@ func DialTimeout(d time.Duration) Option {
 
 // Call Options
 
+// WithExchange sets the exchange to route a message through
+func WithExchange(e string) PublishOption {
+	return func(o *PublishOptions) {
+		o.Exchange = e
+	}
+}
+
 // WithAddress sets the remote address to use rather than using service discovery
 func WithAddress(a string) CallOption {
 	return func(o *CallOptions) {
@@ -306,3 +318,10 @@ func StreamingRequest() RequestOption {
 		o.Stream = true
 	}
 }
+
+// WithRouter sets the client router
+func WithRouter(r Router) Option {
+	return func(o *Options) {
+		o.Router = r
+	}
+}

client/rpc_client.go

@@ -4,11 +4,14 @@ import (
 	"bytes"
 	"context"
 	"fmt"
+	"net"
+	"os"
+	"strconv"
 	"sync"
+	"sync/atomic"
 	"time"
 
-	"sync/atomic"
-
+	"github.com/google/uuid"
 	"github.com/micro/go-micro/broker"
 	"github.com/micro/go-micro/codec"
 	"github.com/micro/go-micro/errors"
@@ -49,13 +52,18 @@ func (r *rpcClient) newCodec(contentType string) (codec.NewCodec, error) {
 	if c, ok := r.opts.Codecs[contentType]; ok {
 		return c, nil
 	}
-	if cf, ok := defaultCodecs[contentType]; ok {
+	if cf, ok := DefaultCodecs[contentType]; ok {
 		return cf, nil
 	}
 	return nil, fmt.Errorf("Unsupported Content-Type: %s", contentType)
 }
 
-func (r *rpcClient) call(ctx context.Context, address string, req Request, resp interface{}, opts CallOptions) error {
+func (r *rpcClient) call(ctx context.Context, node *registry.Node, req Request, resp interface{}, opts CallOptions) error {
+	address := node.Address
+	if node.Port > 0 {
+		address = fmt.Sprintf("%s:%d", address, node.Port)
+	}
+
 	msg := &transport.Message{
 		Header: make(map[string]string),
 	}
@@ -74,9 +82,16 @@ func (r *rpcClient) call(ctx context.Context, address string, req Request, resp
 	// set the accept header
 	msg.Header["Accept"] = req.ContentType()
 
-	cf, err := r.newCodec(req.ContentType())
-	if err != nil {
-		return errors.InternalServerError("go.micro.client", err.Error())
+	// setup old protocol
+	cf := setupProtocol(msg, node)
+
+	// no codec specified
+	if cf == nil {
+		var err error
+		cf, err = r.newCodec(req.ContentType())
+		if err != nil {
+			return errors.InternalServerError("go.micro.client", err.Error())
+		}
 	}
 
 	var grr error
@@ -91,13 +106,20 @@ func (r *rpcClient) call(ctx context.Context, address string, req Request, resp
 
 	seq := atomic.LoadUint64(&r.seq)
 	atomic.AddUint64(&r.seq, 1)
+	codec := newRpcCodec(msg, c, cf)
+
+	rsp := &rpcResponse{
+		socket: c,
+		codec:  codec,
+	}
 
 	stream := &rpcStream{
-		context: ctx,
-		request: req,
-		closed:  make(chan bool),
-		codec:   newRpcPlusCodec(msg, c, cf),
-		seq:     seq,
+		context:  ctx,
+		request:  req,
+		response: rsp,
+		codec:    codec,
+		closed:   make(chan bool),
+		id:       fmt.Sprintf("%v", seq),
 	}
 	defer stream.Close()
 
@@ -111,7 +133,7 @@ func (r *rpcClient) call(ctx context.Context, address string, req Request, resp
 		}()
 
 		// send request
-		if err := stream.Send(req.Request()); err != nil {
+		if err := stream.Send(req.Body()); err != nil {
 			ch <- err
 			return
 		}
@@ -136,7 +158,12 @@ func (r *rpcClient) call(ctx context.Context, address string, req Request, resp
 	}
 }
 
-func (r *rpcClient) stream(ctx context.Context, address string, req Request, opts CallOptions) (Stream, error) {
+func (r *rpcClient) stream(ctx context.Context, node *registry.Node, req Request, opts CallOptions) (Stream, error) {
+	address := node.Address
+	if node.Port > 0 {
+		address = fmt.Sprintf("%s:%d", address, node.Port)
+	}
+
 	msg := &transport.Message{
 		Header: make(map[string]string),
 	}
@@ -155,9 +182,16 @@ func (r *rpcClient) stream(ctx context.Context, address string, req Request, opt
 	// set the accept header
 	msg.Header["Accept"] = req.ContentType()
 
-	cf, err := r.newCodec(req.ContentType())
-	if err != nil {
-		return nil, errors.InternalServerError("go.micro.client", err.Error())
+	// set old codecs
+	cf := setupProtocol(msg, node)
+
+	// no codec specified
+	if cf == nil {
+		var err error
+		cf, err = r.newCodec(req.ContentType())
+		if err != nil {
+			return nil, errors.InternalServerError("go.micro.client", err.Error())
+		}
 	}
 
 	dOpts := []transport.DialOption{
@@ -173,17 +207,30 @@ func (r *rpcClient) stream(ctx context.Context, address string, req Request, opt
 		return nil, errors.InternalServerError("go.micro.client", "connection error: %v", err)
 	}
 
+	codec := newRpcCodec(msg, c, cf)
+
+	rsp := &rpcResponse{
+		socket: c,
+		codec:  codec,
+	}
+
+	// set request codec
+	if r, ok := req.(*rpcRequest); ok {
+		r.codec = codec
+	}
+
 	stream := &rpcStream{
-		context: ctx,
-		request: req,
-		closed:  make(chan bool),
-		codec:   newRpcPlusCodec(msg, c, cf),
+		context:  ctx,
+		request:  req,
+		response: rsp,
+		closed:   make(chan bool),
+		codec:    codec,
 	}
 
 	ch := make(chan error, 1)
 
 	go func() {
-		ch <- stream.Send(req.Request())
+		ch <- stream.Send(req.Body())
 	}()
 
 	var grr error
@@ -227,21 +274,43 @@ func (r *rpcClient) Options() Options {
 }
 
 func (r *rpcClient) next(request Request, opts CallOptions) (selector.Next, error) {
+	service := request.Service()
+
+	// get proxy
+	if prx := os.Getenv("MICRO_PROXY"); len(prx) > 0 {
+		service = prx
+	}
+
+	// get proxy address
+	if prx := os.Getenv("MICRO_PROXY_ADDRESS"); len(prx) > 0 {
+		opts.Address = prx
+	}
+
 	// return remote address
 	if len(opts.Address) > 0 {
+		address := opts.Address
+		port := 0
+
+		host, sport, err := net.SplitHostPort(opts.Address)
+		if err == nil {
+			address = host
+			port, _ = strconv.Atoi(sport)
+		}
+
 		return func() (*registry.Node, error) {
 			return &registry.Node{
-				Address: opts.Address,
+				Address: address,
+				Port:    port,
 			}, nil
 		}, nil
 	}
 
 	// get next nodes from the selector
-	next, err := r.opts.Selector.Select(request.Service(), opts.SelectOptions...)
+	next, err := r.opts.Selector.Select(service, opts.SelectOptions...)
 	if err != nil && err == selector.ErrNotFound {
-		return nil, errors.NotFound("go.micro.client", "service %s: %v", request.Service(), err.Error())
+		return nil, errors.NotFound("go.micro.client", "service %s: %v", service, err.Error())
 	} else if err != nil {
-		return nil, errors.InternalServerError("go.micro.client", "error selecting %s node: %v", request.Service(), err.Error())
+		return nil, errors.InternalServerError("go.micro.client", "error selecting %s node: %v", service, err.Error())
 	}
 
 	return next, nil
@@ -307,19 +376,13 @@ func (r *rpcClient) Call(ctx context.Context, request Request, response interfac
 			return errors.InternalServerError("go.micro.client", "error getting next %s node: %v", request.Service(), err.Error())
 		}
 
-		// set the address
-		address := node.Address
-		if node.Port > 0 {
-			address = fmt.Sprintf("%s:%d", address, node.Port)
-		}
-
 		// make the call
-		err = rcall(ctx, address, request, response, callOpts)
+		err = rcall(ctx, node, request, response, callOpts)
 		r.opts.Selector.Mark(request.Service(), node, err)
 		return err
 	}
 
-	ch := make(chan error, callOpts.Retries)
+	ch := make(chan error, callOpts.Retries+1)
 	var gerr error
 
 	for i := 0; i <= callOpts.Retries; i++ {
@@ -390,12 +453,7 @@ func (r *rpcClient) Stream(ctx context.Context, request Request, opts ...CallOpt
 			return nil, errors.InternalServerError("go.micro.client", "error getting next %s node: %v", request.Service(), err.Error())
 		}
 
-		address := node.Address
-		if node.Port > 0 {
-			address = fmt.Sprintf("%s:%d", address, node.Port)
-		}
-
-		stream, err := r.stream(ctx, address, request, callOpts)
+		stream, err := r.stream(ctx, node, request, callOpts)
 		r.opts.Selector.Mark(request.Service(), node, err)
 		return stream, err
 	}
@@ -405,14 +463,14 @@ func (r *rpcClient) Stream(ctx context.Context, request Request, opts ...CallOpt
 		err    error
 	}
 
-	ch := make(chan response, callOpts.Retries)
+	ch := make(chan response, callOpts.Retries+1)
 	var grr error
 
 	for i := 0; i <= callOpts.Retries; i++ {
-		go func() {
+		go func(i int) {
 			s, err := call(i)
 			ch <- response{s, err}
-		}()
+		}(i)
 
 		select {
 		case <-ctx.Done():
@@ -440,11 +498,35 @@ func (r *rpcClient) Stream(ctx context.Context, request Request, opts ...CallOpt
 }
 
 func (r *rpcClient) Publish(ctx context.Context, msg Message, opts ...PublishOption) error {
+	options := PublishOptions{
+		Context: context.Background(),
+	}
+	for _, o := range opts {
+		o(&options)
+	}
+
 	md, ok := metadata.FromContext(ctx)
 	if !ok {
 		md = make(map[string]string)
 	}
+
+	id := uuid.New().String()
 	md["Content-Type"] = msg.ContentType()
+	md["Micro-Topic"] = msg.Topic()
+	md["Micro-Id"] = id
+
+	// set the topic
+	topic := msg.Topic()
+
+	// get proxy
+	if prx := os.Getenv("MICRO_PROXY"); len(prx) > 0 {
+		options.Exchange = prx
+	}
+
+	// get the exchange
+	if len(options.Exchange) > 0 {
+		topic = options.Exchange
+	}
 
 	// encode message body
 	cf, err := r.newCodec(msg.ContentType())
@@ -452,14 +534,21 @@ func (r *rpcClient) Publish(ctx context.Context, msg Message, opts ...PublishOpt
 		return errors.InternalServerError("go.micro.client", err.Error())
 	}
 	b := &buffer{bytes.NewBuffer(nil)}
-	if err := cf(b).Write(&codec.Message{Type: codec.Publication}, msg.Payload()); err != nil {
+	if err := cf(b).Write(&codec.Message{
+		Target: topic,
+		Type:   codec.Publication,
+		Header: map[string]string{
+			"Micro-Id":    id,
+			"Micro-Topic": msg.Topic(),
+		},
+	}, msg.Payload()); err != nil {
 		return errors.InternalServerError("go.micro.client", err.Error())
 	}
 	r.once.Do(func() {
 		r.opts.Broker.Connect()
 	})
 
-	return r.opts.Broker.Publish(msg.Topic(), &broker.Message{
+	return r.opts.Broker.Publish(topic, &broker.Message{
 		Header: md,
 		Body:   b.Bytes(),
 	})

client/rpc_client_test.go

@@ -7,30 +7,41 @@ import (
 
 	"github.com/micro/go-micro/errors"
 	"github.com/micro/go-micro/registry"
-	"github.com/micro/go-micro/registry/mock"
+	"github.com/micro/go-micro/registry/memory"
 	"github.com/micro/go-micro/selector"
 )
 
+func newTestRegistry() registry.Registry {
+	r := memory.NewRegistry()
+	r.(*memory.Registry).Setup()
+	return r
+}
+
 func TestCallAddress(t *testing.T) {
 	var called bool
 	service := "test.service"
-	method := "Test.Method"
-	address := "10.1.10.1:8080"
+	endpoint := "Test.Endpoint"
+	address := "10.1.10.1"
+	port := 8080
 
 	wrap := func(cf CallFunc) CallFunc {
-		return func(ctx context.Context, addr string, req Request, rsp interface{}, opts CallOptions) error {
+		return func(ctx context.Context, node *registry.Node, req Request, rsp interface{}, opts CallOptions) error {
 			called = true
 
 			if req.Service() != service {
 				return fmt.Errorf("expected service: %s got %s", service, req.Service())
 			}
 
-			if req.Method() != method {
-				return fmt.Errorf("expected service: %s got %s", method, req.Method())
+			if req.Endpoint() != endpoint {
+				return fmt.Errorf("expected service: %s got %s", endpoint, req.Endpoint())
 			}
 
-			if addr != address {
-				return fmt.Errorf("expected address: %s got %s", address, addr)
+			if node.Address != address {
+				return fmt.Errorf("expected address: %s got %s", address, node.Address)
+			}
+
+			if node.Port != port {
+				return fmt.Errorf("expected address: %d got %d", port, node.Port)
 			}
 
 			// don't do the call
@@ -38,17 +49,17 @@ func TestCallAddress(t *testing.T) {
 		}
 	}
 
-	r := mock.NewRegistry()
+	r := newTestRegistry()
 	c := NewClient(
 		Registry(r),
 		WrapCall(wrap),
 	)
 	c.Options().Selector.Init(selector.Registry(r))
 
-	req := c.NewRequest(service, method, nil)
+	req := c.NewRequest(service, endpoint, nil)
 
 	// test calling remote address
-	if err := c.Call(context.Background(), req, nil, WithAddress(address)); err != nil {
+	if err := c.Call(context.Background(), req, nil, WithAddress(fmt.Sprintf("%s:%d", address, port))); err != nil {
 		t.Fatal("call with address error", err)
 	}
 
@@ -60,13 +71,13 @@ func TestCallAddress(t *testing.T) {
 
 func TestCallRetry(t *testing.T) {
 	service := "test.service"
-	method := "Test.Method"
-	address := "10.1.10.1:8080"
+	endpoint := "Test.Endpoint"
+	address := "10.1.10.1"
 
 	var called int
 
 	wrap := func(cf CallFunc) CallFunc {
-		return func(ctx context.Context, addr string, req Request, rsp interface{}, opts CallOptions) error {
+		return func(ctx context.Context, node *registry.Node, req Request, rsp interface{}, opts CallOptions) error {
 			called++
 			if called == 1 {
 				return errors.InternalServerError("test.error", "retry request")
@@ -77,14 +88,14 @@ func TestCallRetry(t *testing.T) {
 		}
 	}
 
-	r := mock.NewRegistry()
+	r := newTestRegistry()
 	c := NewClient(
 		Registry(r),
 		WrapCall(wrap),
 	)
 	c.Options().Selector.Init(selector.Registry(r))
 
-	req := c.NewRequest(service, method, nil)
+	req := c.NewRequest(service, endpoint, nil)
 
 	// test calling remote address
 	if err := c.Call(context.Background(), req, nil, WithAddress(address)); err != nil {
@@ -101,25 +112,24 @@ func TestCallWrapper(t *testing.T) {
 	var called bool
 	id := "test.1"
 	service := "test.service"
-	method := "Test.Method"
-	host := "10.1.10.1"
+	endpoint := "Test.Endpoint"
+	address := "10.1.10.1"
 	port := 8080
-	address := "10.1.10.1:8080"
 
 	wrap := func(cf CallFunc) CallFunc {
-		return func(ctx context.Context, addr string, req Request, rsp interface{}, opts CallOptions) error {
+		return func(ctx context.Context, node *registry.Node, req Request, rsp interface{}, opts CallOptions) error {
 			called = true
 
 			if req.Service() != service {
 				return fmt.Errorf("expected service: %s got %s", service, req.Service())
 			}
 
-			if req.Method() != method {
-				return fmt.Errorf("expected service: %s got %s", method, req.Method())
+			if req.Endpoint() != endpoint {
+				return fmt.Errorf("expected service: %s got %s", endpoint, req.Endpoint())
 			}
 
-			if addr != address {
-				return fmt.Errorf("expected address: %s got %s", address, addr)
+			if node.Address != address {
+				return fmt.Errorf("expected address: %s got %s", address, node.Address)
 			}
 
 			// don't do the call
@@ -127,7 +137,7 @@ func TestCallWrapper(t *testing.T) {
 		}
 	}
 
-	r := mock.NewRegistry()
+	r := newTestRegistry()
 	c := NewClient(
 		Registry(r),
 		WrapCall(wrap),
@@ -140,13 +150,13 @@ func TestCallWrapper(t *testing.T) {
 		Nodes: []*registry.Node{
 			&registry.Node{
 				Id:      id,
-				Address: host,
+				Address: address,
 				Port:    port,
 			},
 		},
 	})
 
-	req := c.NewRequest(service, method, nil)
+	req := c.NewRequest(service, endpoint, nil)
 	if err := c.Call(context.Background(), req, nil); err != nil {
 		t.Fatal("call wrapper error", err)
 	}

client/rpc_codec.go

@@ -5,9 +5,14 @@ import (
 	errs "errors"
 
 	"github.com/micro/go-micro/codec"
+	raw "github.com/micro/go-micro/codec/bytes"
+	"github.com/micro/go-micro/codec/grpc"
+	"github.com/micro/go-micro/codec/json"
 	"github.com/micro/go-micro/codec/jsonrpc"
+	"github.com/micro/go-micro/codec/proto"
 	"github.com/micro/go-micro/codec/protorpc"
 	"github.com/micro/go-micro/errors"
+	"github.com/micro/go-micro/registry"
 	"github.com/micro/go-micro/transport"
 )
 
@@ -28,7 +33,7 @@ var (
 	errShutdown = errs.New("connection is shut down")
 )
 
-type rpcPlusCodec struct {
+type rpcCodec struct {
 	client transport.Client
 	codec  codec.Codec
 
@@ -41,31 +46,21 @@ type readWriteCloser struct {
 	rbuf *bytes.Buffer
 }
 
-type clientCodec interface {
-	WriteRequest(*request, interface{}) error
-	ReadResponseHeader(*response) error
-	ReadResponseBody(interface{}) error
-
-	Close() error
-}
-
-type request struct {
-	Service       string
-	ServiceMethod string   // format: "Service.Method"
-	Seq           uint64   // sequence number chosen by client
-	next          *request // for free list in Server
-}
-
-type response struct {
-	ServiceMethod string    // echoes that of the Request
-	Seq           uint64    // echoes that of the request
-	Error         string    // error, if any.
-	next          *response // for free list in Server
-}
-
 var (
-	defaultContentType = "application/octet-stream"
+	DefaultContentType = "application/protobuf"
 
+	DefaultCodecs = map[string]codec.NewCodec{
+		"application/grpc":         grpc.NewCodec,
+		"application/grpc+json":    grpc.NewCodec,
+		"application/grpc+proto":   grpc.NewCodec,
+		"application/protobuf":     proto.NewCodec,
+		"application/json":         json.NewCodec,
+		"application/json-rpc":     jsonrpc.NewCodec,
+		"application/proto-rpc":    protorpc.NewCodec,
+		"application/octet-stream": raw.NewCodec,
+	}
+
+	// TODO: remove legacy codec list
 	defaultCodecs = map[string]codec.NewCodec{
 		"application/json":         jsonrpc.NewCodec,
 		"application/json-rpc":     jsonrpc.NewCodec,
@@ -89,12 +84,77 @@ func (rwc *readWriteCloser) Close() error {
 	return nil
 }
 
-func newRpcPlusCodec(req *transport.Message, client transport.Client, c codec.NewCodec) *rpcPlusCodec {
+func getHeaders(m *codec.Message) {
+	get := func(hdr string) string {
+		if hd := m.Header[hdr]; len(hd) > 0 {
+			return hd
+		}
+		// old
+		return m.Header["X-"+hdr]
+	}
+
+	// check error in header
+	if len(m.Error) == 0 {
+		m.Error = get("Micro-Error")
+	}
+
+	// check endpoint in header
+	if len(m.Endpoint) == 0 {
+		m.Endpoint = get("Micro-Endpoint")
+	}
+
+	// check method in header
+	if len(m.Method) == 0 {
+		m.Method = get("Micro-Method")
+	}
+
+	if len(m.Id) == 0 {
+		m.Id = get("Micro-Id")
+	}
+}
+
+func setHeaders(m *codec.Message) {
+	set := func(hdr, v string) {
+		if len(v) == 0 {
+			return
+		}
+		m.Header[hdr] = v
+		m.Header["X-"+hdr] = v
+	}
+
+	set("Micro-Id", m.Id)
+	set("Micro-Service", m.Target)
+	set("Micro-Method", m.Method)
+	set("Micro-Endpoint", m.Endpoint)
+}
+
+// setupProtocol sets up the old protocol
+func setupProtocol(msg *transport.Message, node *registry.Node) codec.NewCodec {
+	protocol := node.Metadata["protocol"]
+
+	// got protocol
+	if len(protocol) > 0 {
+		return nil
+	}
+
+	// no protocol use old codecs
+	switch msg.Header["Content-Type"] {
+	case "application/json":
+		msg.Header["Content-Type"] = "application/json-rpc"
+	case "application/protobuf":
+		msg.Header["Content-Type"] = "application/proto-rpc"
+	}
+
+	// now return codec
+	return defaultCodecs[msg.Header["Content-Type"]]
+}
+
+func newRpcCodec(req *transport.Message, client transport.Client, c codec.NewCodec) codec.Codec {
 	rwc := &readWriteCloser{
 		wbuf: bytes.NewBuffer(nil),
 		rbuf: bytes.NewBuffer(nil),
 	}
-	r := &rpcPlusCodec{
+	r := &rpcCodec{
 		buf:    rwc,
 		client: client,
 		codec:  c(rwc),
@@ -103,58 +163,90 @@ func newRpcPlusCodec(req *transport.Message, client transport.Client, c codec.Ne
 	return r
 }
 
-func (c *rpcPlusCodec) WriteRequest(req *request, body interface{}) error {
+func (c *rpcCodec) Write(m *codec.Message, body interface{}) error {
 	c.buf.wbuf.Reset()
 
-	m := &codec.Message{
-		Id:     req.Seq,
-		Target: req.Service,
-		Method: req.ServiceMethod,
-		Type:   codec.Request,
-		Header: map[string]string{
-			"X-Micro-Target": req.Service,
-			"X-Micro-Method": req.ServiceMethod,
-		},
+	// create header
+	if m.Header == nil {
+		m.Header = map[string]string{}
 	}
-	if err := c.codec.Write(m, body); err != nil {
-		return errors.InternalServerError("go.micro.client.codec", err.Error())
+
+	// copy original header
+	for k, v := range c.req.Header {
+		m.Header[k] = v
 	}
-	c.req.Body = c.buf.wbuf.Bytes()
-	for k, v := range m.Header {
-		c.req.Header[k] = v
+
+	// set the mucp headers
+	setHeaders(m)
+
+	// if body is bytes Frame don't encode
+	if body != nil {
+		b, ok := body.(*raw.Frame)
+		if ok {
+			// set body
+			m.Body = b.Data
+			body = nil
+		}
 	}
-	if err := c.client.Send(c.req); err != nil {
+
+	if len(m.Body) == 0 {
+		// write to codec
+		if err := c.codec.Write(m, body); err != nil {
+			return errors.InternalServerError("go.micro.client.codec", err.Error())
+		}
+		// set body
+		m.Body = c.buf.wbuf.Bytes()
+	}
+
+	// create new transport message
+	msg := transport.Message{
+		Header: m.Header,
+		Body:   m.Body,
+	}
+	// send the request
+	if err := c.client.Send(&msg); err != nil {
 		return errors.InternalServerError("go.micro.client.transport", err.Error())
 	}
 	return nil
 }
 
-func (c *rpcPlusCodec) ReadResponseHeader(r *response) error {
-	var m transport.Message
-	if err := c.client.Recv(&m); err != nil {
+func (c *rpcCodec) ReadHeader(m *codec.Message, r codec.MessageType) error {
+	var tm transport.Message
+
+	// read message from transport
+	if err := c.client.Recv(&tm); err != nil {
 		return errors.InternalServerError("go.micro.client.transport", err.Error())
 	}
+
 	c.buf.rbuf.Reset()
-	c.buf.rbuf.Write(m.Body)
-	var me codec.Message
-	err := c.codec.ReadHeader(&me, codec.Response)
-	r.ServiceMethod = me.Method
-	r.Seq = me.Id
-	r.Error = me.Error
+	c.buf.rbuf.Write(tm.Body)
+
+	// set headers from transport
+	m.Header = tm.Header
+
+	// read header
+	err := c.codec.ReadHeader(m, r)
+
+	// get headers
+	getHeaders(m)
+
+	// return header error
 	if err != nil {
 		return errors.InternalServerError("go.micro.client.codec", err.Error())
 	}
+
 	return nil
 }
 
-func (c *rpcPlusCodec) ReadResponseBody(b interface{}) error {
+func (c *rpcCodec) ReadBody(b interface{}) error {
+	// read body
 	if err := c.codec.ReadBody(b); err != nil {
 		return errors.InternalServerError("go.micro.client.codec", err.Error())
 	}
 	return nil
 }
 
-func (c *rpcPlusCodec) Close() error {
+func (c *rpcCodec) Close() error {
 	c.buf.Close()
 	c.codec.Close()
 	if err := c.client.Close(); err != nil {
@@ -162,3 +254,7 @@ func (c *rpcPlusCodec) Close() error {
 	}
 	return nil
 }
+
+func (c *rpcCodec) String() string {
+	return "rpc"
+}

client/rpc_pool_test.go

@@ -5,7 +5,7 @@ import (
 	"time"
 
 	"github.com/micro/go-micro/transport"
-	"github.com/micro/go-micro/transport/mock"
+	"github.com/micro/go-micro/transport/memory"
 )
 
 func testPool(t *testing.T, size int, ttl time.Duration) {
@@ -13,7 +13,7 @@ func testPool(t *testing.T, size int, ttl time.Duration) {
 	p := newPool(size, ttl)
 
 	// mock transport
-	tr := mock.NewTransport()
+	tr := memory.NewTransport()
 
 	// listen
 	l, err := tr.Listen(":0")

client/rpc_request.go

@@ -1,14 +1,20 @@
 package client
 
+import (
+	"github.com/micro/go-micro/codec"
+)
+
 type rpcRequest struct {
 	service     string
 	method      string
+	endpoint    string
 	contentType string
-	request     interface{}
+	codec       codec.Codec
+	body        interface{}
 	opts        RequestOptions
 }
 
-func newRequest(service, method string, request interface{}, contentType string, reqOpts ...RequestOption) Request {
+func newRequest(service, endpoint string, request interface{}, contentType string, reqOpts ...RequestOption) Request {
 	var opts RequestOptions
 
 	for _, o := range reqOpts {
@@ -22,8 +28,9 @@ func newRequest(service, method string, request interface{}, contentType string,
 
 	return &rpcRequest{
 		service:     service,
-		method:      method,
-		request:     request,
+		method:      endpoint,
+		endpoint:    endpoint,
+		body:        request,
 		contentType: contentType,
 		opts:        opts,
 	}
@@ -41,8 +48,16 @@ func (r *rpcRequest) Method() string {
 	return r.method
 }
 
-func (r *rpcRequest) Request() interface{} {
-	return r.request
+func (r *rpcRequest) Endpoint() string {
+	return r.endpoint
+}
+
+func (r *rpcRequest) Body() interface{} {
+	return r.body
+}
+
+func (r *rpcRequest) Codec() codec.Writer {
+	return r.codec
 }
 
 func (r *rpcRequest) Stream() bool {

client/rpc_request_test.go

@@ -5,19 +5,19 @@ import (
 )
 
 func TestRequestOptions(t *testing.T) {
-	r := newRequest("service", "method", nil, "application/json")
+	r := newRequest("service", "endpoint", nil, "application/json")
 	if r.Service() != "service" {
 		t.Fatalf("expected 'service' got %s", r.Service())
 	}
-	if r.Method() != "method" {
-		t.Fatalf("expected 'method' got %s", r.Method())
+	if r.Endpoint() != "endpoint" {
+		t.Fatalf("expected 'endpoint' got %s", r.Endpoint())
 	}
 	if r.ContentType() != "application/json" {
-		t.Fatalf("expected 'method' got %s", r.ContentType())
+		t.Fatalf("expected 'endpoint' got %s", r.ContentType())
 	}
 
-	r2 := newRequest("service", "method", nil, "application/json", WithContentType("application/protobuf"))
+	r2 := newRequest("service", "endpoint", nil, "application/json", WithContentType("application/protobuf"))
 	if r2.ContentType() != "application/protobuf" {
-		t.Fatalf("expected 'method' got %s", r2.ContentType())
+		t.Fatalf("expected 'endpoint' got %s", r2.ContentType())
 	}
 }

client/rpc_response.go

@@ -0,0 +1,35 @@
+package client
+
+import (
+	"github.com/micro/go-micro/codec"
+	"github.com/micro/go-micro/transport"
+)
+
+type rpcResponse struct {
+	header map[string]string
+	body   []byte
+	socket transport.Socket
+	codec  codec.Codec
+}
+
+func (r *rpcResponse) Codec() codec.Reader {
+	return r.codec
+}
+
+func (r *rpcResponse) Header() map[string]string {
+	return r.header
+}
+
+func (r *rpcResponse) Read() ([]byte, error) {
+	var msg transport.Message
+
+	if err := r.socket.Recv(&msg); err != nil {
+		return nil, err
+	}
+
+	// set internals
+	r.header = msg.Header
+	r.body = msg.Body
+
+	return msg.Body, nil
+}

client/rpc_stream.go

@@ -4,17 +4,20 @@ import (
 	"context"
 	"io"
 	"sync"
+
+	"github.com/micro/go-micro/codec"
 )
 
 // Implements the streamer interface
 type rpcStream struct {
 	sync.RWMutex
-	seq     uint64
-	closed  chan bool
-	err     error
-	request Request
-	codec   clientCodec
-	context context.Context
+	id       string
+	closed   chan bool
+	err      error
+	request  Request
+	response Response
+	codec    codec.Codec
+	context  context.Context
 }
 
 func (r *rpcStream) isClosed() bool {
@@ -34,6 +37,10 @@ func (r *rpcStream) Request() Request {
 	return r.request
 }
 
+func (r *rpcStream) Response() Response {
+	return r.response
+}
+
 func (r *rpcStream) Send(msg interface{}) error {
 	r.Lock()
 	defer r.Unlock()
@@ -43,18 +50,19 @@ func (r *rpcStream) Send(msg interface{}) error {
 		return errShutdown
 	}
 
-	seq := r.seq
-
-	req := request{
-		Service:       r.request.Service(),
-		Seq:           seq,
-		ServiceMethod: r.request.Method(),
+	req := codec.Message{
+		Id:       r.id,
+		Target:   r.request.Service(),
+		Method:   r.request.Method(),
+		Endpoint: r.request.Endpoint(),
+		Type:     codec.Request,
 	}
 
-	if err := r.codec.WriteRequest(&req, msg); err != nil {
+	if err := r.codec.Write(&req, msg); err != nil {
 		r.err = err
 		return err
 	}
+
 	return nil
 }
 
@@ -67,8 +75,9 @@ func (r *rpcStream) Recv(msg interface{}) error {
 		return errShutdown
 	}
 
-	var resp response
-	if err := r.codec.ReadResponseHeader(&resp); err != nil {
+	var resp codec.Message
+
+	if err := r.codec.ReadHeader(&resp, codec.Response); err != nil {
 		if err == io.EOF && !r.isClosed() {
 			r.err = io.ErrUnexpectedEOF
 			return io.ErrUnexpectedEOF
@@ -87,11 +96,11 @@ func (r *rpcStream) Recv(msg interface{}) error {
 		} else {
 			r.err = io.EOF
 		}
-		if err := r.codec.ReadResponseBody(nil); err != nil {
+		if err := r.codec.ReadBody(nil); err != nil {
 			r.err = err
 		}
 	default:
-		if err := r.codec.ReadResponseBody(msg); err != nil {
+		if err := r.codec.ReadBody(msg); err != nil {
 			r.err = err
 		}
 	}

client/wrapper.go

@@ -2,10 +2,12 @@ package client
 
 import (
 	"context"
+
+	"github.com/micro/go-micro/registry"
 )
 
 // CallFunc represents the individual call func
-type CallFunc func(ctx context.Context, address string, req Request, rsp interface{}, opts CallOptions) error
+type CallFunc func(ctx context.Context, node *registry.Node, req Request, rsp interface{}, opts CallOptions) error
 
 // CallWrapper is a low level wrapper for the CallFunc
 type CallWrapper func(CallFunc) CallFunc

cmd/cmd.go

@@ -10,26 +10,32 @@ import (
 	"time"
 
 	"github.com/micro/cli"
-	"github.com/micro/go-log"
 	"github.com/micro/go-micro/client"
 	"github.com/micro/go-micro/server"
+	"github.com/micro/go-micro/util/log"
 
 	// brokers
 	"github.com/micro/go-micro/broker"
 	"github.com/micro/go-micro/broker/http"
+	"github.com/micro/go-micro/broker/memory"
+	"github.com/micro/go-micro/broker/nats"
 
 	// registries
 	"github.com/micro/go-micro/registry"
 	"github.com/micro/go-micro/registry/consul"
 	"github.com/micro/go-micro/registry/gossip"
 	"github.com/micro/go-micro/registry/mdns"
+	rmem "github.com/micro/go-micro/registry/memory"
 
 	// selectors
 	"github.com/micro/go-micro/selector"
+	"github.com/micro/go-micro/selector/dns"
+	"github.com/micro/go-micro/selector/static"
 
 	// transports
 	"github.com/micro/go-micro/transport"
 	thttp "github.com/micro/go-micro/transport/http"
+	tmem "github.com/micro/go-micro/transport/memory"
 )
 
 type Cmd interface {
@@ -66,6 +72,7 @@ var (
 		cli.IntFlag{
 			Name:   "client_retries",
 			EnvVar: "MICRO_CLIENT_RETRIES",
+			Value:  client.DefaultRetries,
 			Usage:  "Sets the client retries. Default: 1",
 		},
 		cli.IntFlag{
@@ -162,7 +169,9 @@ var (
 	}
 
 	DefaultBrokers = map[string]func(...broker.Option) broker.Broker{
-		"http": http.NewBroker,
+		"http":   http.NewBroker,
+		"memory": memory.NewBroker,
+		"nats":   nats.NewBroker,
 	}
 
 	DefaultClients = map[string]func(...client.Option) client.Client{
@@ -173,11 +182,14 @@ var (
 		"consul": consul.NewRegistry,
 		"gossip": gossip.NewRegistry,
 		"mdns":   mdns.NewRegistry,
+		"memory": rmem.NewRegistry,
 	}
 
 	DefaultSelectors = map[string]func(...selector.Option) selector.Selector{
 		"default": selector.NewSelector,
+		"dns":     dns.NewSelector,
 		"cache":   selector.NewSelector,
+		"static":  static.NewSelector,
 	}
 
 	DefaultServers = map[string]func(...server.Option) server.Server{
@@ -185,15 +197,16 @@ var (
 	}
 
 	DefaultTransports = map[string]func(...transport.Option) transport.Transport{
-		"http": thttp.NewTransport,
+		"memory": tmem.NewTransport,
+		"http":   thttp.NewTransport,
 	}
 
 	// used for default selection as the fall back
 	defaultClient    = "rpc"
 	defaultServer    = "rpc"
 	defaultBroker    = "http"
-	defaultRegistry  = "consul"
-	defaultSelector  = "cache"
+	defaultRegistry  = "mdns"
+	defaultSelector  = "registry"
 	defaultTransport = "http"
 )
 
@@ -394,6 +407,10 @@ func (c *cmd) Before(ctx *cli.Context) error {
 		serverOpts = append(serverOpts, server.RegisterTTL(ttl*time.Second))
 	}
 
+	if val := time.Duration(ctx.GlobalInt("register_interval")); val > 0 {
+		serverOpts = append(serverOpts, server.RegisterInterval(val*time.Second))
+	}
+
 	// client opts
 	if r := ctx.Int("client_retries"); r >= 0 {
 		clientOpts = append(clientOpts, client.Retries(r))

codec/bytes/bytes.go

@@ -0,0 +1,75 @@
+// Package bytes provides a bytes codec which does not encode or decode anything
+package bytes
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+
+	"github.com/micro/go-micro/codec"
+)
+
+type Codec struct {
+	Conn io.ReadWriteCloser
+}
+
+// Frame gives us the ability to define raw data to send over the pipes
+type Frame struct {
+	Data []byte
+}
+
+func (c *Codec) ReadHeader(m *codec.Message, t codec.MessageType) error {
+	return nil
+}
+
+func (c *Codec) ReadBody(b interface{}) error {
+	// read bytes
+	buf, err := ioutil.ReadAll(c.Conn)
+	if err != nil {
+		return err
+	}
+
+	switch b.(type) {
+	case *[]byte:
+		v := b.(*[]byte)
+		*v = buf
+	case *Frame:
+		v := b.(*Frame)
+		v.Data = buf
+	default:
+		return fmt.Errorf("failed to read body: %v is not type of *[]byte", b)
+	}
+
+	return nil
+}
+
+func (c *Codec) Write(m *codec.Message, b interface{}) error {
+	var v []byte
+	switch b.(type) {
+	case *Frame:
+		v = b.(*Frame).Data
+	case *[]byte:
+		ve := b.(*[]byte)
+		v = *ve
+	case []byte:
+		v = b.([]byte)
+	default:
+		return fmt.Errorf("failed to write: %v is not type of *[]byte or []byte", b)
+	}
+	_, err := c.Conn.Write(v)
+	return err
+}
+
+func (c *Codec) Close() error {
+	return c.Conn.Close()
+}
+
+func (c *Codec) String() string {
+	return "bytes"
+}
+
+func NewCodec(c io.ReadWriteCloser) codec.Codec {
+	return &Codec{
+		Conn: c,
+	}
+}

codec/bytes/marshaler.go

@@ -0,0 +1,41 @@
+package bytes
+
+import (
+	"errors"
+)
+
+type Marshaler struct{}
+
+type Message struct {
+	Header map[string]string
+	Body   []byte
+}
+
+func (n Marshaler) Marshal(v interface{}) ([]byte, error) {
+	switch v.(type) {
+	case *[]byte:
+		ve := v.(*[]byte)
+		return *ve, nil
+	case []byte:
+		return v.([]byte), nil
+	case *Message:
+		return v.(*Message).Body, nil
+	}
+	return nil, errors.New("invalid message")
+}
+
+func (n Marshaler) Unmarshal(d []byte, v interface{}) error {
+	switch v.(type) {
+	case *[]byte:
+		ve := v.(*[]byte)
+		*ve = d
+	case *Message:
+		ve := v.(*Message)
+		ve.Body = d
+	}
+	return errors.New("invalid message")
+}
+
+func (n Marshaler) String() string {
+	return "bytes"
+}

codec/codec.go

@@ -23,10 +23,26 @@ type NewCodec func(io.ReadWriteCloser) Codec
 // connection. ReadBody may be called with a nil argument to force the
 // body to be read and discarded.
 type Codec interface {
+	Reader
+	Writer
+	Close() error
+	String() string
+}
+
+type Reader interface {
 	ReadHeader(*Message, MessageType) error
 	ReadBody(interface{}) error
+}
+
+type Writer interface {
 	Write(*Message, interface{}) error
-	Close() error
+}
+
+// Marshaler is a simple encoding interface used for the broker/transport
+// where headers are not supported by the underlying implementation.
+type Marshaler interface {
+	Marshal(interface{}) ([]byte, error)
+	Unmarshal([]byte, interface{}) error
 	String() string
 }
 
@@ -34,10 +50,14 @@ type Codec interface {
 // the communication, likely followed by the body.
 // In the case of an error, body may be nil.
 type Message struct {
-	Id     uint64
-	Type   MessageType
-	Target string
-	Method string
-	Error  string
+	Id       string
+	Type     MessageType
+	Target   string
+	Method   string
+	Endpoint string
+	Error    string
+
+	// The values read from the socket
 	Header map[string]string
+	Body   []byte
 }

codec/grpc/grpc.go

@@ -0,0 +1,132 @@
+// Package grpc provides a grpc codec
+package grpc
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"strings"
+
+	"github.com/golang/protobuf/proto"
+	"github.com/micro/go-micro/codec"
+)
+
+type Codec struct {
+	Conn        io.ReadWriteCloser
+	ContentType string
+}
+
+func (c *Codec) ReadHeader(m *codec.Message, t codec.MessageType) error {
+	if ct := m.Header["Content-Type"]; len(ct) > 0 {
+		c.ContentType = ct
+	}
+
+	if ct := m.Header["content-type"]; len(ct) > 0 {
+		c.ContentType = ct
+	}
+
+	// service method
+	path := m.Header[":path"]
+	if len(path) == 0 || path[0] != '/' {
+		m.Target = m.Header["Micro-Service"]
+		m.Endpoint = m.Header["Micro-Endpoint"]
+	} else {
+		// [ , a.package.Foo, Bar]
+		parts := strings.Split(path, "/")
+		if len(parts) != 3 {
+			return errors.New("Unknown request path")
+		}
+		service := strings.Split(parts[1], ".")
+		m.Endpoint = strings.Join([]string{service[len(service)-1], parts[2]}, ".")
+		m.Target = strings.Join(service[:len(service)-1], ".")
+	}
+
+	return nil
+}
+
+func (c *Codec) ReadBody(b interface{}) error {
+	// no body
+	if b == nil {
+		return nil
+	}
+
+	_, buf, err := decode(c.Conn)
+	if err != nil {
+		return err
+	}
+
+	switch c.ContentType {
+	case "application/grpc+json":
+		return json.Unmarshal(buf, b)
+	case "application/grpc+proto", "application/grpc":
+		return proto.Unmarshal(buf, b.(proto.Message))
+	}
+
+	return errors.New("Unsupported Content-Type")
+}
+
+func (c *Codec) Write(m *codec.Message, b interface{}) error {
+	var buf []byte
+	var err error
+
+	if ct := m.Header["Content-Type"]; len(ct) > 0 {
+		c.ContentType = ct
+	}
+
+	if ct := m.Header["content-type"]; len(ct) > 0 {
+		c.ContentType = ct
+	}
+
+	switch m.Type {
+	case codec.Request:
+		parts := strings.Split(m.Endpoint, ".")
+		m.Header[":method"] = "POST"
+		m.Header[":path"] = fmt.Sprintf("/%s.%s/%s", m.Target, parts[0], parts[1])
+		m.Header[":proto"] = "HTTP/2.0"
+		m.Header["te"] = "trailers"
+		m.Header["user-agent"] = "grpc-go/1.0.0"
+		m.Header[":authority"] = m.Target
+		m.Header["content-type"] = c.ContentType
+	case codec.Response:
+		m.Header["Trailer"] = "grpc-status, grpc-message"
+		m.Header["grpc-status"] = "0"
+		m.Header["grpc-message"] = ""
+	}
+
+	// marshal content
+	switch c.ContentType {
+	case "application/grpc+json":
+		buf, err = json.Marshal(b)
+	case "application/grpc+proto", "application/grpc":
+		pb, ok := b.(proto.Message)
+		if ok {
+			buf, err = proto.Marshal(pb)
+		}
+	default:
+		err = errors.New("Unsupported Content-Type")
+	}
+	// check error
+	if err != nil {
+		m.Header["grpc-status"] = "8"
+		m.Header["grpc-message"] = err.Error()
+		return err
+	}
+
+	return encode(0, buf, c.Conn)
+}
+
+func (c *Codec) Close() error {
+	return c.Conn.Close()
+}
+
+func (c *Codec) String() string {
+	return "grpc"
+}
+
+func NewCodec(c io.ReadWriteCloser) codec.Codec {
+	return &Codec{
+		Conn:        c,
+		ContentType: "application/grpc",
+	}
+}

codec/grpc/util.go

@@ -0,0 +1,70 @@
+package grpc
+
+import (
+	"encoding/binary"
+	"fmt"
+	"io"
+)
+
+var (
+	maxMessageSize = 1024 * 1024 * 4
+	maxInt         = int(^uint(0) >> 1)
+)
+
+func decode(r io.Reader) (uint8, []byte, error) {
+	header := make([]byte, 5)
+
+	// read the header
+	if _, err := r.Read(header[:]); err != nil {
+		return uint8(0), nil, err
+	}
+
+	// get encoding format e.g compressed
+	cf := uint8(header[0])
+
+	// get message length
+	length := binary.BigEndian.Uint32(header[1:])
+
+	// no encoding format
+	if length == 0 {
+		return cf, nil, nil
+	}
+
+	//
+	if int64(length) > int64(maxInt) {
+		return cf, nil, fmt.Errorf("grpc: received message larger than max length allowed on current machine (%d vs. %d)", length, maxInt)
+	}
+	if int(length) > maxMessageSize {
+		return cf, nil, fmt.Errorf("grpc: received message larger than max (%d vs. %d)", length, maxMessageSize)
+	}
+
+	msg := make([]byte, int(length))
+
+	if _, err := r.Read(msg); err != nil {
+		if err == io.EOF {
+			err = io.ErrUnexpectedEOF
+		}
+		return cf, nil, err
+	}
+
+	return cf, msg, nil
+}
+
+func encode(cf uint8, buf []byte, w io.Writer) error {
+	header := make([]byte, 5)
+
+	// set compression
+	header[0] = byte(cf)
+
+	// write length as header
+	binary.BigEndian.PutUint32(header[1:], uint32(len(buf)))
+
+	// read the header
+	if _, err := w.Write(header[:]); err != nil {
+		return err
+	}
+
+	// write the buffer
+	_, err := w.Write(buf)
+	return err
+}

codec/json/json.go

@@ -0,0 +1,49 @@
+// Package json provides a json codec
+package json
+
+import (
+	"encoding/json"
+	"io"
+
+	"github.com/micro/go-micro/codec"
+)
+
+type Codec struct {
+	Conn    io.ReadWriteCloser
+	Encoder *json.Encoder
+	Decoder *json.Decoder
+}
+
+func (c *Codec) ReadHeader(m *codec.Message, t codec.MessageType) error {
+	return nil
+}
+
+func (c *Codec) ReadBody(b interface{}) error {
+	if b == nil {
+		return nil
+	}
+	return c.Decoder.Decode(b)
+}
+
+func (c *Codec) Write(m *codec.Message, b interface{}) error {
+	if b == nil {
+		return nil
+	}
+	return c.Encoder.Encode(b)
+}
+
+func (c *Codec) Close() error {
+	return c.Conn.Close()
+}
+
+func (c *Codec) String() string {
+	return "json"
+}
+
+func NewCodec(c io.ReadWriteCloser) codec.Codec {
+	return &Codec{
+		Conn:    c,
+		Decoder: json.NewDecoder(c),
+		Encoder: json.NewEncoder(c),
+	}
+}

codec/json/marshaler.go

@@ -0,0 +1,19 @@
+package json
+
+import (
+	"encoding/json"
+)
+
+type Marshaler struct{}
+
+func (j Marshaler) Marshal(v interface{}) ([]byte, error) {
+	return json.Marshal(v)
+}
+
+func (j Marshaler) Unmarshal(d []byte, v interface{}) error {
+	return json.Unmarshal(d, v)
+}
+
+func (j Marshaler) String() string {
+	return "json"
+}

codec/jsonrpc/client.go

@@ -19,17 +19,17 @@ type clientCodec struct {
 	resp clientResponse
 
 	sync.Mutex
-	pending map[uint64]string
+	pending map[interface{}]string
 }
 
 type clientRequest struct {
 	Method string         `json:"method"`
 	Params [1]interface{} `json:"params"`
-	ID     uint64         `json:"id"`
+	ID     interface{}    `json:"id"`
 }
 
 type clientResponse struct {
-	ID     uint64           `json:"id"`
+	ID     interface{}      `json:"id"`
 	Result *json.RawMessage `json:"result"`
 	Error  interface{}      `json:"error"`
 }
@@ -39,7 +39,7 @@ func newClientCodec(conn io.ReadWriteCloser) *clientCodec {
 		dec:     json.NewDecoder(conn),
 		enc:     json.NewEncoder(conn),
 		c:       conn,
-		pending: make(map[uint64]string),
+		pending: make(map[interface{}]string),
 	}
 }
 
@@ -71,7 +71,7 @@ func (c *clientCodec) ReadHeader(m *codec.Message) error {
 	c.Unlock()
 
 	m.Error = ""
-	m.Id = c.resp.ID
+	m.Id = fmt.Sprintf("%v", c.resp.ID)
 	if c.resp.Error != nil {
 		x, ok := c.resp.Error.(string)
 		if !ok {

codec/jsonrpc/jsonrpc.go

@@ -31,7 +31,7 @@ func (j *jsonCodec) Write(m *codec.Message, b interface{}) error {
 	switch m.Type {
 	case codec.Request:
 		return j.c.Write(m, b)
-	case codec.Response:
+	case codec.Response, codec.Error:
 		return j.s.Write(m, b)
 	case codec.Publication:
 		data, err := json.Marshal(b)

codec/jsonrpc/server.go

@@ -2,9 +2,8 @@ package jsonrpc
 
 import (
 	"encoding/json"
-	"errors"
+	"fmt"
 	"io"
-	"sync"
 
 	"github.com/micro/go-micro/codec"
 )
@@ -17,30 +16,25 @@ type serverCodec struct {
 	// temporary work space
 	req  serverRequest
 	resp serverResponse
-
-	sync.Mutex
-	seq     uint64
-	pending map[uint64]*json.RawMessage
 }
 
 type serverRequest struct {
 	Method string           `json:"method"`
 	Params *json.RawMessage `json:"params"`
-	ID     *json.RawMessage `json:"id"`
+	ID     interface{}      `json:"id"`
 }
 
 type serverResponse struct {
-	ID     *json.RawMessage `json:"id"`
-	Result interface{}      `json:"result"`
-	Error  interface{}      `json:"error"`
+	ID     interface{} `json:"id"`
+	Result interface{} `json:"result"`
+	Error  interface{} `json:"error"`
 }
 
 func newServerCodec(conn io.ReadWriteCloser) *serverCodec {
 	return &serverCodec{
-		dec:     json.NewDecoder(conn),
-		enc:     json.NewEncoder(conn),
-		c:       conn,
-		pending: make(map[uint64]*json.RawMessage),
+		dec: json.NewDecoder(conn),
+		enc: json.NewEncoder(conn),
+		c:   conn,
 	}
 }
 
@@ -50,7 +44,7 @@ func (r *serverRequest) reset() {
 		*r.Params = (*r.Params)[0:0]
 	}
 	if r.ID != nil {
-		*r.ID = (*r.ID)[0:0]
+		r.ID = nil
 	}
 }
 
@@ -60,14 +54,8 @@ func (c *serverCodec) ReadHeader(m *codec.Message) error {
 		return err
 	}
 	m.Method = c.req.Method
-
-	c.Lock()
-	c.seq++
-	c.pending[c.seq] = c.req.ID
+	m.Id = fmt.Sprintf("%v", c.req.ID)
 	c.req.ID = nil
-	m.Id = c.seq
-	c.Unlock()
-
 	return nil
 }
 
@@ -84,19 +72,7 @@ var null = json.RawMessage([]byte("null"))
 
 func (c *serverCodec) Write(m *codec.Message, x interface{}) error {
 	var resp serverResponse
-	c.Lock()
-	b, ok := c.pending[m.Id]
-	if !ok {
-		c.Unlock()
-		return errors.New("invalid sequence number in response")
-	}
-	c.Unlock()
-
-	if b == nil {
-		// Invalid request so no id.  Use JSON null.
-		b = &null
-	}
-	resp.ID = b
+	resp.ID = m.Id
 	resp.Result = x
 	if m.Error == "" {
 		resp.Error = nil

codec/proto/marshaler.go

@@ -0,0 +1,19 @@
+package proto
+
+import (
+	"github.com/golang/protobuf/proto"
+)
+
+type Marshaler struct{}
+
+func (Marshaler) Marshal(v interface{}) ([]byte, error) {
+	return proto.Marshal(v.(proto.Message))
+}
+
+func (Marshaler) Unmarshal(data []byte, v interface{}) error {
+	return proto.Unmarshal(data, v.(proto.Message))
+}
+
+func (Marshaler) Name() string {
+	return "proto"
+}

codec/proto/proto.go

@@ -0,0 +1,56 @@
+// Package proto provides a proto codec
+package proto
+
+import (
+	"io"
+	"io/ioutil"
+
+	"github.com/golang/protobuf/proto"
+	"github.com/micro/go-micro/codec"
+)
+
+type Codec struct {
+	Conn io.ReadWriteCloser
+}
+
+func (c *Codec) ReadHeader(m *codec.Message, t codec.MessageType) error {
+	return nil
+}
+
+func (c *Codec) ReadBody(b interface{}) error {
+	if b == nil {
+		return nil
+	}
+	buf, err := ioutil.ReadAll(c.Conn)
+	if err != nil {
+		return err
+	}
+	return proto.Unmarshal(buf, b.(proto.Message))
+}
+
+func (c *Codec) Write(m *codec.Message, b interface{}) error {
+	p, ok := b.(proto.Message)
+	if !ok {
+		return nil
+	}
+	buf, err := proto.Marshal(p)
+	if err != nil {
+		return err
+	}
+	_, err = c.Conn.Write(buf)
+	return err
+}
+
+func (c *Codec) Close() error {
+	return c.Conn.Close()
+}
+
+func (c *Codec) String() string {
+	return "proto"
+}
+
+func NewCodec(c io.ReadWriteCloser) codec.Codec {
+	return &Codec{
+		Conn: c,
+	}
+}

codec/protorpc/protorpc.go

@@ -5,6 +5,7 @@ import (
 	"bytes"
 	"fmt"
 	"io"
+	"strconv"
 	"sync"
 
 	"github.com/golang/protobuf/proto"
@@ -31,13 +32,22 @@ func (c *protoCodec) String() string {
 	return "proto-rpc"
 }
 
+func id(id string) *uint64 {
+	p, err := strconv.ParseInt(id, 10, 64)
+	if err != nil {
+		p = 0
+	}
+	i := uint64(p)
+	return &i
+}
+
 func (c *protoCodec) Write(m *codec.Message, b interface{}) error {
 	switch m.Type {
 	case codec.Request:
 		c.Lock()
 		defer c.Unlock()
 		// This is protobuf, of course we copy it.
-		pbr := &Request{ServiceMethod: &m.Method, Seq: &m.Id}
+		pbr := &Request{ServiceMethod: &m.Method, Seq: id(m.Id)}
 		data, err := proto.Marshal(pbr)
 		if err != nil {
 			return err
@@ -60,10 +70,10 @@ func (c *protoCodec) Write(m *codec.Message, b interface{}) error {
 				return err
 			}
 		}
-	case codec.Response:
+	case codec.Response, codec.Error:
 		c.Lock()
 		defer c.Unlock()
-		rtmp := &Response{ServiceMethod: &m.Method, Seq: &m.Id, Error: &m.Error}
+		rtmp := &Response{ServiceMethod: &m.Method, Seq: id(m.Id), Error: &m.Error}
 		data, err := proto.Marshal(rtmp)
 		if err != nil {
 			return err
@@ -117,7 +127,7 @@ func (c *protoCodec) ReadHeader(m *codec.Message, mt codec.MessageType) error {
 			return err
 		}
 		m.Method = rtmp.GetServiceMethod()
-		m.Id = rtmp.GetSeq()
+		m.Id = fmt.Sprintf("%d", rtmp.GetSeq())
 	case codec.Response:
 		data, err := ReadNetString(c.rwc)
 		if err != nil {
@@ -129,7 +139,7 @@ func (c *protoCodec) ReadHeader(m *codec.Message, mt codec.MessageType) error {
 			return err
 		}
 		m.Method = rtmp.GetServiceMethod()
-		m.Id = rtmp.GetSeq()
+		m.Id = fmt.Sprintf("%d", rtmp.GetSeq())
 		m.Error = rtmp.GetError()
 	case codec.Publication:
 		_, err := io.Copy(c.buf, c.rwc)

codec/text/text.go

@@ -0,0 +1,75 @@
+// Package text reads any text/* content-type
+package text
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+
+	"github.com/micro/go-micro/codec"
+)
+
+type Codec struct {
+	Conn io.ReadWriteCloser
+}
+
+// Frame gives us the ability to define raw data to send over the pipes
+type Frame struct {
+	Data []byte
+}
+
+func (c *Codec) ReadHeader(m *codec.Message, t codec.MessageType) error {
+	return nil
+}
+
+func (c *Codec) ReadBody(b interface{}) error {
+	// read bytes
+	buf, err := ioutil.ReadAll(c.Conn)
+	if err != nil {
+		return err
+	}
+
+	switch b.(type) {
+	case *[]byte:
+		v := b.(*[]byte)
+		*v = buf
+	case *Frame:
+		v := b.(*Frame)
+		v.Data = buf
+	default:
+		return fmt.Errorf("failed to read body: %v is not type of *[]byte", b)
+	}
+
+	return nil
+}
+
+func (c *Codec) Write(m *codec.Message, b interface{}) error {
+	var v []byte
+	switch b.(type) {
+	case *Frame:
+		v = b.(*Frame).Data
+	case *[]byte:
+		ve := b.(*[]byte)
+		v = *ve
+	case []byte:
+		v = b.([]byte)
+	default:
+		return fmt.Errorf("failed to write: %v is not type of *[]byte or []byte", b)
+	}
+	_, err := c.Conn.Write(v)
+	return err
+}
+
+func (c *Codec) Close() error {
+	return c.Conn.Close()
+}
+
+func (c *Codec) String() string {
+	return "bytes"
+}
+
+func NewCodec(c io.ReadWriteCloser) codec.Codec {
+	return &Codec{
+		Conn: c,
+	}
+}

config/README.md

@@ -0,0 +1,29 @@
+# Config [![GoDoc](https://godoc.org/github.com/micro/go-micro/config?status.svg)](https://godoc.org/github.com/micro/go-micro/config)
+
+Go Config is a pluggable dynamic config library.
+
+Most config in applications are statically configured or include complex logic to load from multiple sources. 
+Go Config makes this easy, pluggable and mergeable. You'll never have to deal with config in the same way again.
+
+## Features
+
+- **Dynamic Loading** - Load configuration from multiple source as and when needed. Go Config manages watching config sources 
+in the background and automatically merges and updates an in memory view. 
+
+- **Pluggable Sources** - Choose from any number of sources to load and merge config. The backend source is abstracted away into 
+a standard format consumed internally and decoded via encoders. Sources can be env vars, flags, file, etcd, k8s configmap, etc.
+
+- **Mergeable Config** - If you specify multiple sources of config, regardless of format, they will be merged and presented in 
+a single view. This massively simplifies priority order loading and changes based on environment.
+
+- **Observe Changes** - Optionally watch the config for changes to specific values. Hot reload your app using Go Config's watcher. 
+You don't have to handle ad-hoc hup reloading or whatever else, just keep reading the config and watch for changes if you need 
+to be notified.
+
+- **Sane Defaults** - In case config loads badly or is completely wiped away for some unknown reason, you can specify fallback 
+values when accessing any config values directly. This ensures you'll always be reading some sane default in the event of a problem.
+
+## Getting Started
+
+For detailed information or architecture, installation and general usage see the [docs](https://micro.mu/docs/go-config.html)
+

config/config.go

@@ -0,0 +1,94 @@
+// Package config is an interface for dynamic configuration.
+package config
+
+import (
+	"context"
+
+	"github.com/micro/go-micro/config/loader"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/source"
+	"github.com/micro/go-micro/config/source/file"
+)
+
+// Config is an interface abstraction for dynamic configuration
+type Config interface {
+	// provide the reader.Values interface
+	reader.Values
+	// Stop the config loader/watcher
+	Close() error
+	// Load config sources
+	Load(source ...source.Source) error
+	// Force a source changeset sync
+	Sync() error
+	// Watch a value for changes
+	Watch(path ...string) (Watcher, error)
+}
+
+// Watcher is the config watcher
+type Watcher interface {
+	Next() (reader.Value, error)
+	Stop() error
+}
+
+type Options struct {
+	Loader loader.Loader
+	Reader reader.Reader
+	Source []source.Source
+
+	// for alternative data
+	Context context.Context
+}
+
+type Option func(o *Options)
+
+var (
+	// Default Config Manager
+	DefaultConfig = NewConfig()
+)
+
+// NewConfig returns new config
+func NewConfig(opts ...Option) Config {
+	return newConfig(opts...)
+}
+
+// Return config as raw json
+func Bytes() []byte {
+	return DefaultConfig.Bytes()
+}
+
+// Return config as a map
+func Map() map[string]interface{} {
+	return DefaultConfig.Map()
+}
+
+// Scan values to a go type
+func Scan(v interface{}) error {
+	return DefaultConfig.Scan(v)
+}
+
+// Force a source changeset sync
+func Sync() error {
+	return DefaultConfig.Sync()
+}
+
+// Get a value from the config
+func Get(path ...string) reader.Value {
+	return DefaultConfig.Get(path...)
+}
+
+// Load config sources
+func Load(source ...source.Source) error {
+	return DefaultConfig.Load(source...)
+}
+
+// Watch a value for changes
+func Watch(path ...string) (Watcher, error) {
+	return DefaultConfig.Watch(path...)
+}
+
+// LoadFile is short hand for creating a file source and loading it
+func LoadFile(path string) error {
+	return Load(file.NewSource(
+		file.WithPath(path),
+	))
+}

config/default.go

@@ -0,0 +1,253 @@
+package config
+
+import (
+	"bytes"
+	"sync"
+	"time"
+
+	"github.com/micro/go-micro/config/loader"
+	"github.com/micro/go-micro/config/loader/memory"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/reader/json"
+	"github.com/micro/go-micro/config/source"
+)
+
+type config struct {
+	exit chan bool
+	opts Options
+
+	sync.RWMutex
+	// the current snapshot
+	snap *loader.Snapshot
+	// the current values
+	vals reader.Values
+}
+
+type watcher struct {
+	lw    loader.Watcher
+	rd    reader.Reader
+	path  []string
+	value reader.Value
+}
+
+func newConfig(opts ...Option) Config {
+	options := Options{
+		Loader: memory.NewLoader(),
+		Reader: json.NewReader(),
+	}
+
+	for _, o := range opts {
+		o(&options)
+	}
+
+	options.Loader.Load(options.Source...)
+	snap, _ := options.Loader.Snapshot()
+	vals, _ := options.Reader.Values(snap.ChangeSet)
+
+	c := &config{
+		exit: make(chan bool),
+		opts: options,
+		snap: snap,
+		vals: vals,
+	}
+
+	go c.run()
+
+	return c
+}
+
+func (c *config) run() {
+	watch := func(w loader.Watcher) error {
+		for {
+			// get changeset
+			snap, err := w.Next()
+			if err != nil {
+				return err
+			}
+
+			c.Lock()
+
+			// save
+			c.snap = snap
+
+			// set values
+			c.vals, _ = c.opts.Reader.Values(snap.ChangeSet)
+
+			c.Unlock()
+		}
+	}
+
+	for {
+		w, err := c.opts.Loader.Watch()
+		if err != nil {
+			time.Sleep(time.Second)
+			continue
+		}
+
+		done := make(chan bool)
+
+		// the stop watch func
+		go func() {
+			select {
+			case <-done:
+			case <-c.exit:
+			}
+			w.Stop()
+		}()
+
+		// block watch
+		if err := watch(w); err != nil {
+			// do something better
+			time.Sleep(time.Second)
+		}
+
+		// close done chan
+		close(done)
+
+		// if the config is closed exit
+		select {
+		case <-c.exit:
+			return
+		default:
+		}
+	}
+}
+
+func (c *config) Map() map[string]interface{} {
+	c.RLock()
+	defer c.RUnlock()
+	return c.vals.Map()
+}
+
+func (c *config) Scan(v interface{}) error {
+	c.RLock()
+	defer c.RUnlock()
+	return c.vals.Scan(v)
+}
+
+// sync loads all the sources, calls the parser and updates the config
+func (c *config) Sync() error {
+	if err := c.opts.Loader.Sync(); err != nil {
+		return err
+	}
+
+	snap, err := c.opts.Loader.Snapshot()
+	if err != nil {
+		return err
+	}
+
+	c.Lock()
+	defer c.Unlock()
+
+	c.snap = snap
+	vals, err := c.opts.Reader.Values(snap.ChangeSet)
+	if err != nil {
+		return err
+	}
+	c.vals = vals
+
+	return nil
+}
+
+func (c *config) Close() error {
+	select {
+	case <-c.exit:
+		return nil
+	default:
+		close(c.exit)
+	}
+	return nil
+}
+
+func (c *config) Get(path ...string) reader.Value {
+	c.RLock()
+	defer c.RUnlock()
+
+	// did sync actually work?
+	if c.vals != nil {
+		return c.vals.Get(path...)
+	}
+
+	// no value
+	return newValue()
+}
+
+func (c *config) Bytes() []byte {
+	c.RLock()
+	defer c.RUnlock()
+
+	if c.vals == nil {
+		return []byte{}
+	}
+
+	return c.vals.Bytes()
+}
+
+func (c *config) Load(sources ...source.Source) error {
+	if err := c.opts.Loader.Load(sources...); err != nil {
+		return err
+	}
+
+	snap, err := c.opts.Loader.Snapshot()
+	if err != nil {
+		return err
+	}
+
+	c.Lock()
+	defer c.Unlock()
+
+	c.snap = snap
+	vals, err := c.opts.Reader.Values(snap.ChangeSet)
+	if err != nil {
+		return err
+	}
+	c.vals = vals
+
+	return nil
+}
+
+func (c *config) Watch(path ...string) (Watcher, error) {
+	value := c.Get(path...)
+
+	w, err := c.opts.Loader.Watch(path...)
+	if err != nil {
+		return nil, err
+	}
+
+	return &watcher{
+		lw:    w,
+		rd:    c.opts.Reader,
+		path:  path,
+		value: value,
+	}, nil
+}
+
+func (c *config) String() string {
+	return "config"
+}
+
+func (w *watcher) Next() (reader.Value, error) {
+	for {
+		s, err := w.lw.Next()
+		if err != nil {
+			return nil, err
+		}
+
+		// only process changes
+		if bytes.Equal(w.value.Bytes(), s.ChangeSet.Data) {
+			continue
+		}
+
+		v, err := w.rd.Values(s.ChangeSet)
+		if err != nil {
+			return nil, err
+		}
+
+		w.value = v.Get()
+		return w.value, nil
+	}
+}
+
+func (w *watcher) Stop() error {
+	return w.lw.Stop()
+}

config/default_test.go

@@ -0,0 +1,101 @@
+package config
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/micro/go-micro/config/source/file"
+)
+
+func createFileForTest(t *testing.T) *os.File {
+	data := []byte(`{"foo": "bar"}`)
+	path := filepath.Join(os.TempDir(), fmt.Sprintf("file.%d", time.Now().UnixNano()))
+	fh, err := os.Create(path)
+	if err != nil {
+		t.Error(err)
+	}
+	_, err = fh.Write(data)
+	if err != nil {
+		t.Error(err)
+	}
+
+	return fh
+}
+
+func TestLoadWithGoodFile(t *testing.T) {
+	fh := createFileForTest(t)
+	path := fh.Name()
+	defer func() {
+		fh.Close()
+		os.Remove(path)
+	}()
+
+	// Create new config
+	conf := NewConfig()
+	// Load file source
+	if err := conf.Load(file.NewSource(
+		file.WithPath(path),
+	)); err != nil {
+		t.Fatalf("Expected no error but got %v", err)
+	}
+}
+
+func TestLoadWithInvalidFile(t *testing.T) {
+	fh := createFileForTest(t)
+	path := fh.Name()
+	defer func() {
+		fh.Close()
+		os.Remove(path)
+	}()
+
+	// Create new config
+	conf := NewConfig()
+	// Load file source
+	err := conf.Load(file.NewSource(
+		file.WithPath(path),
+		file.WithPath("/i/do/not/exists.json"),
+	))
+
+	if err == nil {
+		t.Fatal("Expected error but none !")
+	}
+	if !strings.Contains(fmt.Sprintf("%v", err), "/i/do/not/exists.json") {
+		t.Fatalf("Expected error to contain the unexisting file but got %v", err)
+	}
+}
+
+func TestConsul(t *testing.T) {
+	/*consulSource := consul.NewSource(
+		// optionally specify consul address; default to localhost:8500
+		consul.WithAddress("131.150.38.111:8500"),
+		// optionally specify prefix; defaults to /micro/config
+		consul.WithPrefix("/project"),
+		// optionally strip the provided prefix from the keys, defaults to false
+		consul.StripPrefix(true),
+		consul.WithDatacenter("dc1"),
+		consul.WithToken("xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"),
+	)
+
+	// Create new config
+	conf := NewConfig()
+
+	// Load file source
+	err := conf.Load(consulSource)
+	if err != nil {
+		t.Error(err)
+		return
+	}
+
+	m := conf.Map()
+	t.Log("m: ", m)
+
+	v := conf.Get("project", "dc111", "port")
+
+	t.Log("v: ", v.Int(13))*/
+
+	t.Log("OK")
+}

config/encoder/encoder.go

@@ -0,0 +1,8 @@
+// Package encoder handles source encoding formats
+package encoder
+
+type Encoder interface {
+	Encode(interface{}) ([]byte, error)
+	Decode([]byte, interface{}) error
+	String() string
+}

config/encoder/hcl/hcl.go

@@ -0,0 +1,26 @@
+package hcl
+
+import (
+	"encoding/json"
+
+	"github.com/hashicorp/hcl"
+	"github.com/micro/go-micro/config/encoder"
+)
+
+type hclEncoder struct{}
+
+func (h hclEncoder) Encode(v interface{}) ([]byte, error) {
+	return json.Marshal(v)
+}
+
+func (h hclEncoder) Decode(d []byte, v interface{}) error {
+	return hcl.Unmarshal(d, v)
+}
+
+func (h hclEncoder) String() string {
+	return "hcl"
+}
+
+func NewEncoder() encoder.Encoder {
+	return hclEncoder{}
+}

config/encoder/json/json.go

@@ -0,0 +1,25 @@
+package json
+
+import (
+	"encoding/json"
+
+	"github.com/micro/go-micro/config/encoder"
+)
+
+type jsonEncoder struct{}
+
+func (j jsonEncoder) Encode(v interface{}) ([]byte, error) {
+	return json.Marshal(v)
+}
+
+func (j jsonEncoder) Decode(d []byte, v interface{}) error {
+	return json.Unmarshal(d, v)
+}
+
+func (j jsonEncoder) String() string {
+	return "json"
+}
+
+func NewEncoder() encoder.Encoder {
+	return jsonEncoder{}
+}

config/encoder/toml/toml.go

@@ -0,0 +1,32 @@
+package toml
+
+import (
+	"bytes"
+
+	"github.com/BurntSushi/toml"
+	"github.com/micro/go-micro/config/encoder"
+)
+
+type tomlEncoder struct{}
+
+func (t tomlEncoder) Encode(v interface{}) ([]byte, error) {
+	b := bytes.NewBuffer(nil)
+	defer b.Reset()
+	err := toml.NewEncoder(b).Encode(v)
+	if err != nil {
+		return nil, err
+	}
+	return b.Bytes(), nil
+}
+
+func (t tomlEncoder) Decode(d []byte, v interface{}) error {
+	return toml.Unmarshal(d, v)
+}
+
+func (t tomlEncoder) String() string {
+	return "toml"
+}
+
+func NewEncoder() encoder.Encoder {
+	return tomlEncoder{}
+}

config/encoder/xml/xml.go

@@ -0,0 +1,25 @@
+package xml
+
+import (
+	"encoding/xml"
+
+	"github.com/micro/go-micro/config/encoder"
+)
+
+type xmlEncoder struct{}
+
+func (x xmlEncoder) Encode(v interface{}) ([]byte, error) {
+	return xml.Marshal(v)
+}
+
+func (x xmlEncoder) Decode(d []byte, v interface{}) error {
+	return xml.Unmarshal(d, v)
+}
+
+func (x xmlEncoder) String() string {
+	return "xml"
+}
+
+func NewEncoder() encoder.Encoder {
+	return xmlEncoder{}
+}

config/encoder/yaml/yaml.go

@@ -0,0 +1,24 @@
+package yaml
+
+import (
+	"github.com/ghodss/yaml"
+	"github.com/micro/go-micro/config/encoder"
+)
+
+type yamlEncoder struct{}
+
+func (y yamlEncoder) Encode(v interface{}) ([]byte, error) {
+	return yaml.Marshal(v)
+}
+
+func (y yamlEncoder) Decode(d []byte, v interface{}) error {
+	return yaml.Unmarshal(d, v)
+}
+
+func (y yamlEncoder) String() string {
+	return "yaml"
+}
+
+func NewEncoder() encoder.Encoder {
+	return yamlEncoder{}
+}

config/issue18_test.go

@@ -0,0 +1,60 @@
+package config
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"github.com/micro/go-micro/config/source/env"
+	"github.com/micro/go-micro/config/source/file"
+)
+
+func createFileForIssue18(t *testing.T, content string) *os.File {
+	data := []byte(content)
+	path := filepath.Join(os.TempDir(), fmt.Sprintf("file.%d", time.Now().UnixNano()))
+	fh, err := os.Create(path)
+	if err != nil {
+		t.Error(err)
+	}
+	_, err = fh.Write(data)
+	if err != nil {
+		t.Error(err)
+	}
+
+	return fh
+}
+
+func TestIssue18(t *testing.T) {
+	fh := createFileForIssue18(t, `{
+  "amqp": {
+    "host": "rabbit.platform",
+    "port": 80
+  },
+  "handler": {
+    "exchange": "springCloudBus"
+  }
+}`)
+	path := fh.Name()
+	defer func() {
+		fh.Close()
+		os.Remove(path)
+	}()
+	os.Setenv("AMQP_HOST", "rabbit.testing.com")
+
+	conf := NewConfig()
+	conf.Load(
+		file.NewSource(
+			file.WithPath(path),
+		),
+		env.NewSource(),
+	)
+
+	actualHost := conf.Get("amqp", "host").String("backup")
+	if actualHost != "rabbit.testing.com" {
+		t.Fatalf("Expected %v but got %v",
+			"rabbit.testing.com",
+			actualHost)
+	}
+}

config/loader/loader.go

@@ -0,0 +1,63 @@
+// package loader manages loading from multiple sources
+package loader
+
+import (
+	"context"
+
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/source"
+)
+
+// Loader manages loading sources
+type Loader interface {
+	// Stop the loader
+	Close() error
+	// Load the sources
+	Load(...source.Source) error
+	// A Snapshot of loaded config
+	Snapshot() (*Snapshot, error)
+	// Force sync of sources
+	Sync() error
+	// Watch for changes
+	Watch(...string) (Watcher, error)
+	// Name of loader
+	String() string
+}
+
+// Watcher lets you watch sources and returns a merged ChangeSet
+type Watcher interface {
+	// First call to next may return the current Snapshot
+	// If you are watching a path then only the data from
+	// that path is returned.
+	Next() (*Snapshot, error)
+	// Stop watching for changes
+	Stop() error
+}
+
+// Snapshot is a merged ChangeSet
+type Snapshot struct {
+	// The merged ChangeSet
+	ChangeSet *source.ChangeSet
+	// Deterministic and comparable version of the snapshot
+	Version string
+}
+
+type Options struct {
+	Reader reader.Reader
+	Source []source.Source
+
+	// for alternative data
+	Context context.Context
+}
+
+type Option func(o *Options)
+
+// Copy snapshot
+func Copy(s *Snapshot) *Snapshot {
+	cs := *(s.ChangeSet)
+
+	return &Snapshot{
+		ChangeSet: &cs,
+		Version:   s.Version,
+	}
+}

config/loader/memory/memory.go

@@ -0,0 +1,415 @@
+package memory
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/micro/go-micro/config/loader"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/reader/json"
+	"github.com/micro/go-micro/config/source"
+)
+
+type memory struct {
+	exit chan bool
+	opts loader.Options
+
+	sync.RWMutex
+	// the current snapshot
+	snap *loader.Snapshot
+	// the current values
+	vals reader.Values
+	// all the sets
+	sets []*source.ChangeSet
+	// all the sources
+	sources []source.Source
+
+	idx      int
+	watchers map[int]*watcher
+}
+
+type watcher struct {
+	exit    chan bool
+	path    []string
+	value   reader.Value
+	reader  reader.Reader
+	updates chan reader.Value
+}
+
+func (m *memory) watch(idx int, s source.Source) {
+	m.Lock()
+	m.sets = append(m.sets, &source.ChangeSet{Source: s.String()})
+	m.Unlock()
+
+	// watches a source for changes
+	watch := func(idx int, s source.Watcher) error {
+		for {
+			// get changeset
+			cs, err := s.Next()
+			if err != nil {
+				return err
+			}
+
+			m.Lock()
+
+			// save
+			m.sets[idx] = cs
+
+			// merge sets
+			set, err := m.opts.Reader.Merge(m.sets...)
+			if err != nil {
+				m.Unlock()
+				return err
+			}
+
+			// set values
+			m.vals, _ = m.opts.Reader.Values(set)
+			m.snap = &loader.Snapshot{
+				ChangeSet: set,
+				Version:   fmt.Sprintf("%d", time.Now().Unix()),
+			}
+			m.Unlock()
+
+			// send watch updates
+			m.update()
+		}
+	}
+
+	for {
+		// watch the source
+		w, err := s.Watch()
+		if err != nil {
+			time.Sleep(time.Second)
+			continue
+		}
+
+		done := make(chan bool)
+
+		// the stop watch func
+		go func() {
+			select {
+			case <-done:
+			case <-m.exit:
+			}
+			w.Stop()
+		}()
+
+		// block watch
+		if err := watch(idx, w); err != nil {
+			// do something better
+			time.Sleep(time.Second)
+		}
+
+		// close done chan
+		close(done)
+
+		// if the config is closed exit
+		select {
+		case <-m.exit:
+			return
+		default:
+		}
+	}
+}
+
+func (m *memory) loaded() bool {
+	var loaded bool
+	m.RLock()
+	if m.vals != nil {
+		loaded = true
+	}
+	m.RUnlock()
+	return loaded
+}
+
+// reload reads the sets and creates new values
+func (m *memory) reload() error {
+	m.Lock()
+
+	// merge sets
+	set, err := m.opts.Reader.Merge(m.sets...)
+	if err != nil {
+		m.Unlock()
+		return err
+	}
+
+	// set values
+	m.vals, _ = m.opts.Reader.Values(set)
+	m.snap = &loader.Snapshot{
+		ChangeSet: set,
+		Version:   fmt.Sprintf("%d", time.Now().Unix()),
+	}
+
+	m.Unlock()
+
+	// update watchers
+	m.update()
+
+	return nil
+}
+
+func (m *memory) update() {
+	var watchers []*watcher
+
+	m.RLock()
+	for _, w := range m.watchers {
+		watchers = append(watchers, w)
+	}
+	m.RUnlock()
+
+	for _, w := range watchers {
+		select {
+		case w.updates <- m.vals.Get(w.path...):
+		default:
+		}
+	}
+}
+
+// Snapshot returns a snapshot of the current loaded config
+func (m *memory) Snapshot() (*loader.Snapshot, error) {
+	if m.loaded() {
+		m.RLock()
+		snap := loader.Copy(m.snap)
+		m.RUnlock()
+		return snap, nil
+	}
+
+	// not loaded, sync
+	if err := m.Sync(); err != nil {
+		return nil, err
+	}
+
+	// make copy
+	m.RLock()
+	snap := loader.Copy(m.snap)
+	m.RUnlock()
+
+	return snap, nil
+}
+
+// Sync loads all the sources, calls the parser and updates the config
+func (m *memory) Sync() error {
+	var sets []*source.ChangeSet
+
+	m.Lock()
+
+	// read the source
+	var gerr []string
+
+	for _, source := range m.sources {
+		ch, err := source.Read()
+		if err != nil {
+			gerr = append(gerr, err.Error())
+			continue
+		}
+		sets = append(sets, ch)
+	}
+
+	// merge sets
+	set, err := m.opts.Reader.Merge(sets...)
+	if err != nil {
+		m.Unlock()
+		return err
+	}
+
+	// set values
+	vals, err := m.opts.Reader.Values(set)
+	if err != nil {
+		m.Unlock()
+		return err
+	}
+	m.vals = vals
+	m.snap = &loader.Snapshot{
+		ChangeSet: set,
+		Version:   fmt.Sprintf("%d", time.Now().Unix()),
+	}
+
+	m.Unlock()
+
+	// update watchers
+	m.update()
+
+	if len(gerr) > 0 {
+		return fmt.Errorf("source loading errors: %s", strings.Join(gerr, "\n"))
+	}
+
+	return nil
+}
+
+func (m *memory) Close() error {
+	select {
+	case <-m.exit:
+		return nil
+	default:
+		close(m.exit)
+	}
+	return nil
+}
+
+func (m *memory) Get(path ...string) (reader.Value, error) {
+	if !m.loaded() {
+		if err := m.Sync(); err != nil {
+			return nil, err
+		}
+	}
+
+	m.Lock()
+	defer m.Unlock()
+
+	// did sync actually work?
+	if m.vals != nil {
+		return m.vals.Get(path...), nil
+	}
+
+	// assuming vals is nil
+	// create new vals
+
+	ch := m.snap.ChangeSet
+
+	// we are truly screwed, trying to load in a hacked way
+	v, err := m.opts.Reader.Values(ch)
+	if err != nil {
+		return nil, err
+	}
+
+	// lets set it just because
+	m.vals = v
+
+	if m.vals != nil {
+		return m.vals.Get(path...), nil
+	}
+
+	// ok we're going hardcore now
+	return nil, errors.New("no values")
+}
+
+func (m *memory) Load(sources ...source.Source) error {
+	var gerrors []string
+
+	for _, source := range sources {
+		set, err := source.Read()
+		if err != nil {
+			gerrors = append(gerrors,
+				fmt.Sprintf("error loading source %s: %v",
+					source,
+					err))
+			// continue processing
+			continue
+		}
+		m.Lock()
+		m.sources = append(m.sources, source)
+		m.sets = append(m.sets, set)
+		idx := len(m.sets) - 1
+		m.Unlock()
+		go m.watch(idx, source)
+	}
+
+	if err := m.reload(); err != nil {
+		gerrors = append(gerrors, err.Error())
+	}
+
+	// Return errors
+	if len(gerrors) != 0 {
+		return errors.New(strings.Join(gerrors, "\n"))
+	}
+	return nil
+}
+
+func (m *memory) Watch(path ...string) (loader.Watcher, error) {
+	value, err := m.Get(path...)
+	if err != nil {
+		return nil, err
+	}
+
+	m.Lock()
+
+	w := &watcher{
+		exit:    make(chan bool),
+		path:    path,
+		value:   value,
+		reader:  m.opts.Reader,
+		updates: make(chan reader.Value, 1),
+	}
+
+	id := m.idx
+	m.watchers[id] = w
+	m.idx++
+
+	m.Unlock()
+
+	go func() {
+		<-w.exit
+		m.Lock()
+		delete(m.watchers, id)
+		m.Unlock()
+	}()
+
+	return w, nil
+}
+
+func (m *memory) String() string {
+	return "memory"
+}
+
+func (w *watcher) Next() (*loader.Snapshot, error) {
+	for {
+		select {
+		case <-w.exit:
+			return nil, errors.New("watcher stopped")
+		case v := <-w.updates:
+			if bytes.Equal(w.value.Bytes(), v.Bytes()) {
+				continue
+			}
+			w.value = v
+
+			cs := &source.ChangeSet{
+				Data:      v.Bytes(),
+				Format:    w.reader.String(),
+				Source:    "memory",
+				Timestamp: time.Now(),
+			}
+			cs.Sum()
+
+			return &loader.Snapshot{
+				ChangeSet: cs,
+				Version:   fmt.Sprintf("%d", time.Now().Unix()),
+			}, nil
+		}
+	}
+}
+
+func (w *watcher) Stop() error {
+	select {
+	case <-w.exit:
+	default:
+		close(w.exit)
+	}
+	return nil
+}
+
+func NewLoader(opts ...loader.Option) loader.Loader {
+	options := loader.Options{
+		Reader: json.NewReader(),
+	}
+
+	for _, o := range opts {
+		o(&options)
+	}
+
+	m := &memory{
+		exit:     make(chan bool),
+		opts:     options,
+		watchers: make(map[int]*watcher),
+		sources:  options.Source,
+	}
+
+	for i, s := range options.Source {
+		go m.watch(i, s)
+	}
+
+	return m
+}

config/loader/memory/options.go

@@ -0,0 +1,21 @@
+package memory
+
+import (
+	"github.com/micro/go-micro/config/loader"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/source"
+)
+
+// WithSource appends a source to list of sources
+func WithSource(s source.Source) loader.Option {
+	return func(o *loader.Options) {
+		o.Source = append(o.Source, s)
+	}
+}
+
+// WithReader sets the config reader
+func WithReader(r reader.Reader) loader.Option {
+	return func(o *loader.Options) {
+		o.Reader = r
+	}
+}

config/options.go

@@ -0,0 +1,28 @@
+package config
+
+import (
+	"github.com/micro/go-micro/config/loader"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/source"
+)
+
+// WithLoader sets the loader for manager config
+func WithLoader(l loader.Loader) Option {
+	return func(o *Options) {
+		o.Loader = l
+	}
+}
+
+// WithSource appends a source to list of sources
+func WithSource(s source.Source) Option {
+	return func(o *Options) {
+		o.Source = append(o.Source, s)
+	}
+}
+
+// WithReader sets the config reader
+func WithReader(r reader.Reader) Option {
+	return func(o *Options) {
+		o.Reader = r
+	}
+}

config/reader/json/json.go

@@ -0,0 +1,83 @@
+package json
+
+import (
+	"errors"
+	"time"
+
+	"github.com/imdario/mergo"
+	"github.com/micro/go-micro/config/encoder"
+	"github.com/micro/go-micro/config/encoder/json"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/source"
+)
+
+type jsonReader struct {
+	opts reader.Options
+	json encoder.Encoder
+}
+
+func (j *jsonReader) Merge(changes ...*source.ChangeSet) (*source.ChangeSet, error) {
+	var merged map[string]interface{}
+
+	for _, m := range changes {
+		if m == nil {
+			continue
+		}
+
+		if len(m.Data) == 0 {
+			continue
+		}
+
+		codec, ok := j.opts.Encoding[m.Format]
+		if !ok {
+			// fallback
+			codec = j.json
+		}
+
+		var data map[string]interface{}
+		if err := codec.Decode(m.Data, &data); err != nil {
+			return nil, err
+		}
+		if err := mergo.Map(&merged, data, mergo.WithOverride); err != nil {
+			return nil, err
+		}
+	}
+
+	b, err := j.json.Encode(merged)
+	if err != nil {
+		return nil, err
+	}
+
+	cs := &source.ChangeSet{
+		Timestamp: time.Now(),
+		Data:      b,
+		Source:    "json",
+		Format:    j.json.String(),
+	}
+	cs.Checksum = cs.Sum()
+
+	return cs, nil
+}
+
+func (j *jsonReader) Values(ch *source.ChangeSet) (reader.Values, error) {
+	if ch == nil {
+		return nil, errors.New("changeset is nil")
+	}
+	if ch.Format != "json" {
+		return nil, errors.New("unsupported format")
+	}
+	return newValues(ch)
+}
+
+func (j *jsonReader) String() string {
+	return "json"
+}
+
+// NewReader creates a json reader
+func NewReader(opts ...reader.Option) reader.Reader {
+	options := reader.NewOptions(opts...)
+	return &jsonReader{
+		json: json.NewEncoder(),
+		opts: options,
+	}
+}

config/reader/json/json_test.go

@@ -0,0 +1,43 @@
+package json
+
+import (
+	"testing"
+
+	"github.com/micro/go-micro/config/source"
+)
+
+func TestReader(t *testing.T) {
+	data := []byte(`{"foo": "bar", "baz": {"bar": "cat"}}`)
+
+	testData := []struct {
+		path  []string
+		value string
+	}{
+		{
+			[]string{"foo"},
+			"bar",
+		},
+		{
+			[]string{"baz", "bar"},
+			"cat",
+		},
+	}
+
+	r := NewReader()
+
+	c, err := r.Merge(&source.ChangeSet{Data: data}, &source.ChangeSet{})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	values, err := r.Values(c)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for _, test := range testData {
+		if v := values.Get(test.path...).String(""); v != test.value {
+			t.Fatalf("Expected %s got %s for path %v", test.value, v, test.path)
+		}
+	}
+}

config/reader/json/values.go

@@ -0,0 +1,208 @@
+package json
+
+import (
+	"encoding/json"
+	"fmt"
+	"strconv"
+	"strings"
+	"time"
+
+	simple "github.com/bitly/go-simplejson"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/source"
+)
+
+type jsonValues struct {
+	ch *source.ChangeSet
+	sj *simple.Json
+}
+
+type jsonValue struct {
+	*simple.Json
+}
+
+func newValues(ch *source.ChangeSet) (reader.Values, error) {
+	sj := simple.New()
+	data, _ := reader.ReplaceEnvVars(ch.Data)
+	if err := sj.UnmarshalJSON(data); err != nil {
+		sj.SetPath(nil, string(ch.Data))
+	}
+	return &jsonValues{ch, sj}, nil
+}
+
+func newValue(s *simple.Json) reader.Value {
+	if s == nil {
+		s = simple.New()
+	}
+	return &jsonValue{s}
+}
+
+func (j *jsonValues) Get(path ...string) reader.Value {
+	return &jsonValue{j.sj.GetPath(path...)}
+}
+
+func (j *jsonValues) Del(path ...string) {
+	// delete the tree?
+	if len(path) == 0 {
+		j.sj = simple.New()
+		return
+	}
+
+	if len(path) == 1 {
+		j.sj.Del(path[0])
+		return
+	}
+
+	vals := j.sj.GetPath(path[:len(path)-1]...)
+	vals.Del(path[len(path)-1])
+	j.sj.SetPath(path[:len(path)-1], vals.Interface())
+	return
+}
+
+func (j *jsonValues) Set(val interface{}, path ...string) {
+	j.sj.SetPath(path, val)
+}
+
+func (j *jsonValues) Bytes() []byte {
+	b, _ := j.sj.MarshalJSON()
+	return b
+}
+
+func (j *jsonValues) Map() map[string]interface{} {
+	m, _ := j.sj.Map()
+	return m
+}
+
+func (j *jsonValues) Scan(v interface{}) error {
+	b, err := j.sj.MarshalJSON()
+	if err != nil {
+		return err
+	}
+	return json.Unmarshal(b, v)
+}
+
+func (j *jsonValues) String() string {
+	return "json"
+}
+
+func (j *jsonValue) Bool(def bool) bool {
+	b, err := j.Json.Bool()
+	if err == nil {
+		return b
+	}
+
+	str, ok := j.Interface().(string)
+	if !ok {
+		return def
+	}
+
+	b, err = strconv.ParseBool(str)
+	if err != nil {
+		return def
+	}
+
+	return b
+}
+
+func (j *jsonValue) Int(def int) int {
+	i, err := j.Json.Int()
+	if err == nil {
+		return i
+	}
+
+	str, ok := j.Interface().(string)
+	if !ok {
+		return def
+	}
+
+	i, err = strconv.Atoi(str)
+	if err != nil {
+		return def
+	}
+
+	return i
+}
+
+func (j *jsonValue) String(def string) string {
+	return j.Json.MustString(def)
+}
+
+func (j *jsonValue) Float64(def float64) float64 {
+	f, err := j.Json.Float64()
+	if err == nil {
+		return f
+	}
+
+	str, ok := j.Interface().(string)
+	if !ok {
+		return def
+	}
+
+	f, err = strconv.ParseFloat(str, 64)
+	if err != nil {
+		return def
+	}
+
+	return f
+}
+
+func (j *jsonValue) Duration(def time.Duration) time.Duration {
+	v, err := j.Json.String()
+	if err != nil {
+		return def
+	}
+
+	value, err := time.ParseDuration(v)
+	if err != nil {
+		return def
+	}
+
+	return value
+}
+
+func (j *jsonValue) StringSlice(def []string) []string {
+	v, err := j.Json.String()
+	if err == nil {
+		sl := strings.Split(v, ",")
+		if len(sl) > 1 {
+			return sl
+		}
+	}
+	return j.Json.MustStringArray(def)
+}
+
+func (j *jsonValue) StringMap(def map[string]string) map[string]string {
+	m, err := j.Json.Map()
+	if err != nil {
+		return def
+	}
+
+	res := map[string]string{}
+
+	for k, v := range m {
+		res[k] = fmt.Sprintf("%v", v)
+	}
+
+	return res
+}
+
+func (j *jsonValue) Scan(v interface{}) error {
+	b, err := j.Json.MarshalJSON()
+	if err != nil {
+		return err
+	}
+	return json.Unmarshal(b, v)
+}
+
+func (j *jsonValue) Bytes() []byte {
+	b, err := j.Json.Bytes()
+	if err != nil {
+		// try return marshalled
+		b, err = j.Json.MarshalJSON()
+		if err != nil {
+			return []byte{}
+		}
+		return b
+	}
+	return b
+}

config/reader/json/values_test.go

@@ -0,0 +1,39 @@
+package json
+
+import (
+	"testing"
+
+	"github.com/micro/go-micro/config/source"
+)
+
+func TestValues(t *testing.T) {
+	data := []byte(`{"foo": "bar", "baz": {"bar": "cat"}}`)
+
+	testData := []struct {
+		path  []string
+		value string
+	}{
+		{
+			[]string{"foo"},
+			"bar",
+		},
+		{
+			[]string{"baz", "bar"},
+			"cat",
+		},
+	}
+
+	values, err := newValues(&source.ChangeSet{
+		Data: data,
+	})
+
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for _, test := range testData {
+		if v := values.Get(test.path...).String(""); v != test.value {
+			t.Fatalf("Expected %s got %s for path %v", test.value, v, test.path)
+		}
+	}
+}

config/reader/options.go

@@ -0,0 +1,42 @@
+package reader
+
+import (
+	"github.com/micro/go-micro/config/encoder"
+	"github.com/micro/go-micro/config/encoder/hcl"
+	"github.com/micro/go-micro/config/encoder/json"
+	"github.com/micro/go-micro/config/encoder/toml"
+	"github.com/micro/go-micro/config/encoder/xml"
+	"github.com/micro/go-micro/config/encoder/yaml"
+)
+
+type Options struct {
+	Encoding map[string]encoder.Encoder
+}
+
+type Option func(o *Options)
+
+func NewOptions(opts ...Option) Options {
+	options := Options{
+		Encoding: map[string]encoder.Encoder{
+			"json": json.NewEncoder(),
+			"yaml": yaml.NewEncoder(),
+			"toml": toml.NewEncoder(),
+			"xml":  xml.NewEncoder(),
+			"hcl":  hcl.NewEncoder(),
+			"yml":  yaml.NewEncoder(),
+		},
+	}
+	for _, o := range opts {
+		o(&options)
+	}
+	return options
+}
+
+func WithEncoder(e encoder.Encoder) Option {
+	return func(o *Options) {
+		if o.Encoding == nil {
+			o.Encoding = make(map[string]encoder.Encoder)
+		}
+		o.Encoding[e.String()] = e
+	}
+}

config/reader/preprocessor.go

@@ -0,0 +1,23 @@
+package reader
+
+import (
+	"os"
+	"regexp"
+)
+
+func ReplaceEnvVars(raw []byte) ([]byte, error) {
+	re := regexp.MustCompile(`\$\{([A-Za-z0-9_]+)\}`)
+	if re.Match(raw) {
+		dataS := string(raw)
+		res := re.ReplaceAllStringFunc(dataS, replaceEnvVars)
+		return []byte(res), nil
+	} else {
+		return raw, nil
+	}
+}
+
+func replaceEnvVars(element string) string {
+	v := element[2 : len(element)-1]
+	el := os.Getenv(v)
+	return el
+}

config/reader/preprocessor_test.go

@@ -0,0 +1,73 @@
+package reader
+
+import (
+	"os"
+	"strings"
+	"testing"
+)
+
+func TestReplaceEnvVars(t *testing.T) {
+	os.Setenv("myBar", "cat")
+	os.Setenv("MYBAR", "cat")
+	os.Setenv("my_Bar", "cat")
+	os.Setenv("myBar_", "cat")
+
+	testData := []struct {
+		expected string
+		data     []byte
+	}{
+		// Right use cases
+		{
+			`{"foo": "bar", "baz": {"bar": "cat"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${myBar}"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "cat"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${MYBAR}"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "cat"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${my_Bar}"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "cat"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${myBar_}"}}`),
+		},
+		// Wrong use cases
+		{
+			`{"foo": "bar", "baz": {"bar": "${myBar-}"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${myBar-}"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "${}"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${}"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "$sss}"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "$sss}"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "${sss"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${sss"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "{something}"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "{something}"}}`),
+		},
+		// Use cases without replace env vars
+		{
+			`{"foo": "bar", "baz": {"bar": "cat"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "cat"}}`),
+		},
+	}
+
+	for _, test := range testData {
+		res, err := ReplaceEnvVars(test.data)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if strings.Compare(test.expected, string(res)) != 0 {
+			t.Fatalf("Expected %s got %s", test.expected, res)
+		}
+	}
+}

config/reader/reader.go

@@ -0,0 +1,36 @@
+// Package reader parses change sets and provides config values
+package reader
+
+import (
+	"time"
+
+	"github.com/micro/go-micro/config/source"
+)
+
+// Reader is an interface for merging changesets
+type Reader interface {
+	Merge(...*source.ChangeSet) (*source.ChangeSet, error)
+	Values(*source.ChangeSet) (Values, error)
+	String() string
+}
+
+// Values is returned by the reader
+type Values interface {
+	Bytes() []byte
+	Get(path ...string) Value
+	Map() map[string]interface{}
+	Scan(v interface{}) error
+}
+
+// Value represents a value of any type
+type Value interface {
+	Bool(def bool) bool
+	Int(def int) int
+	String(def string) string
+	Float64(def float64) float64
+	Duration(def time.Duration) time.Duration
+	StringSlice(def []string) []string
+	StringMap(def map[string]string) map[string]string
+	Scan(val interface{}) error
+	Bytes() []byte
+}

config/source/changeset.go

@@ -0,0 +1,13 @@
+package source
+
+import (
+	"crypto/md5"
+	"fmt"
+)
+
+// Sum returns the md5 checksum of the ChangeSet data
+func (c *ChangeSet) Sum() string {
+	h := md5.New()
+	h.Write(c.Data)
+	return fmt.Sprintf("%x", h.Sum(nil))
+}

config/source/cli/README.md

@@ -0,0 +1,71 @@
+# cli Source
+
+The cli source reads config from parsed flags via a cli.Context.
+
+## Format
+
+We expect the use of the `micro/cli` package. Upper case flags will be lower cased. Dashes will be used as delimiters for nesting.
+
+### Example
+
+```go
+micro.Flags(
+    cli.StringFlag{
+        Name: "database-address",
+        Value: "127.0.0.1",
+        Usage: "the db address",
+    },
+    cli.IntFlag{
+        Name: "database-port",
+        Value: 3306,
+        Usage: "the db port",
+    },
+)
+```
+
+Becomes
+
+```json
+{
+    "database": {
+        "address": "127.0.0.1",
+        "port": 3306
+    }
+}
+```
+
+## New and Load Source
+
+Because a cli.Context is needed to retrieve the flags and their values, it is recommended to build your source from within a cli.Action.
+
+```go
+
+func main() {
+    // New Service
+    service := micro.NewService(
+        micro.Name("example"),
+        micro.Flags(
+            cli.StringFlag{
+                Name: "database-address",
+                Value: "127.0.0.1",
+                Usage: "the db address",
+            },
+        ),
+    )
+
+    var clisrc source.Source
+
+    service.Init(
+        micro.Action(func(c *cli.Context) {
+            clisrc = cli.NewSource(
+                cli.Context(c),
+	    )
+            // Alternatively, just setup your config right here
+        }),
+    )
+    
+    // ... Load and use that source ...
+    conf := config.NewConfig()
+    conf.Load(clisrc)
+}
+```

config/source/cli/cli.go

@@ -0,0 +1,146 @@
+package cli
+
+import (
+	"flag"
+	"io/ioutil"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/imdario/mergo"
+	"github.com/micro/cli"
+	"github.com/micro/go-micro/cmd"
+	"github.com/micro/go-micro/config/source"
+)
+
+type cliSource struct {
+	opts source.Options
+	ctx  *cli.Context
+}
+
+func (c *cliSource) Read() (*source.ChangeSet, error) {
+	var changes map[string]interface{}
+
+	for _, name := range c.ctx.GlobalFlagNames() {
+		tmp := toEntry(name, c.ctx.GlobalGeneric(name))
+		mergo.Map(&changes, tmp) // need to sort error handling
+	}
+
+	for _, name := range c.ctx.FlagNames() {
+		tmp := toEntry(name, c.ctx.Generic(name))
+		mergo.Map(&changes, tmp) // need to sort error handling
+	}
+
+	b, err := c.opts.Encoder.Encode(changes)
+	if err != nil {
+		return nil, err
+	}
+
+	cs := &source.ChangeSet{
+		Format:    c.opts.Encoder.String(),
+		Data:      b,
+		Timestamp: time.Now(),
+		Source:    c.String(),
+	}
+	cs.Checksum = cs.Sum()
+
+	return cs, nil
+}
+
+func toEntry(name string, v interface{}) map[string]interface{} {
+	n := strings.ToLower(name)
+	keys := strings.FieldsFunc(n, split)
+	reverse(keys)
+	tmp := make(map[string]interface{})
+	for i, k := range keys {
+		if i == 0 {
+			tmp[k] = v
+			continue
+		}
+
+		tmp = map[string]interface{}{k: tmp}
+	}
+	return tmp
+}
+
+func reverse(ss []string) {
+	for i := len(ss)/2 - 1; i >= 0; i-- {
+		opp := len(ss) - 1 - i
+		ss[i], ss[opp] = ss[opp], ss[i]
+	}
+}
+
+func split(r rune) bool {
+	return r == '-' || r == '_'
+}
+
+func (c *cliSource) Watch() (source.Watcher, error) {
+	return source.NewNoopWatcher()
+}
+
+func (c *cliSource) String() string {
+	return "cli"
+}
+
+// NewSource returns a config source for integrating parsed flags from a micro/cli.Context.
+// Hyphens are delimiters for nesting, and all keys are lowercased. The assumption is that
+// command line flags have already been parsed.
+//
+// Example:
+//      cli.StringFlag{Name: "db-host"},
+//
+//
+//      {
+//          "database": {
+//              "host": "localhost"
+//          }
+//      }
+func NewSource(opts ...source.Option) source.Source {
+	options := source.NewOptions(opts...)
+
+	var ctx *cli.Context
+
+	c, ok := options.Context.Value(contextKey{}).(*cli.Context)
+	if ok {
+		ctx = c
+	}
+
+	// no context
+	if ctx == nil {
+		// get the default app/flags
+		app := cmd.App()
+		flags := app.Flags
+
+		// create flagset
+		set := flag.NewFlagSet(app.Name, flag.ContinueOnError)
+
+		// apply flags to set
+		for _, f := range flags {
+			f.Apply(set)
+		}
+
+		// parse flags
+		set.SetOutput(ioutil.Discard)
+		set.Parse(os.Args[1:])
+
+		// normalise flags
+		normalizeFlags(app.Flags, set)
+
+		// create context
+		ctx = cli.NewContext(app, set, nil)
+	}
+
+	return &cliSource{
+		ctx:  ctx,
+		opts: options,
+	}
+}
+
+// WithContext returns a new source with the context specified.
+// The assumption is that Context is retrieved within an app.Action function.
+func WithContext(ctx *cli.Context, opts ...source.Option) source.Source {
+	return &cliSource{
+		ctx:  ctx,
+		opts: source.NewOptions(opts...),
+	}
+}

config/source/cli/cli_test.go

@@ -0,0 +1,65 @@
+package cli
+
+import (
+	"encoding/json"
+	"os"
+	"testing"
+
+	"github.com/micro/cli"
+	"github.com/micro/go-micro/cmd"
+	"github.com/micro/go-micro/config/source"
+)
+
+func test(t *testing.T, withContext bool) {
+	var src source.Source
+
+	// setup app
+	app := cmd.App()
+	app.Name = "testapp"
+	app.Flags = []cli.Flag{
+		cli.StringFlag{Name: "db-host"},
+	}
+
+	// with context
+	if withContext {
+		// set action
+		app.Action = func(c *cli.Context) {
+			src = WithContext(c)
+		}
+
+		// run app
+		app.Run([]string{"run", "-db-host", "localhost"})
+		// no context
+	} else {
+		// set args
+		os.Args = []string{"run", "-db-host", "localhost"}
+		src = NewSource()
+	}
+
+	// test config
+	c, err := src.Read()
+	if err != nil {
+		t.Error(err)
+	}
+
+	var actual map[string]interface{}
+	if err := json.Unmarshal(c.Data, &actual); err != nil {
+		t.Error(err)
+	}
+
+	actualDB := actual["db"].(map[string]interface{})
+	if actualDB["host"] != "localhost" {
+		t.Errorf("expected localhost, got %v", actualDB["name"])
+	}
+
+}
+
+func TestCliSource(t *testing.T) {
+	// without context
+	test(t, false)
+}
+
+func TestCliSourceWithContext(t *testing.T) {
+	// with context
+	test(t, true)
+}

config/source/cli/options.go

@@ -0,0 +1,20 @@
+package cli
+
+import (
+	"context"
+
+	"github.com/micro/cli"
+	"github.com/micro/go-micro/config/source"
+)
+
+type contextKey struct{}
+
+// Context sets the cli context
+func Context(c *cli.Context) source.Option {
+	return func(o *source.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, contextKey{}, c)
+	}
+}

config/source/cli/util.go

@@ -0,0 +1,50 @@
+package cli
+
+import (
+	"errors"
+	"flag"
+	"strings"
+
+	"github.com/micro/cli"
+)
+
+func copyFlag(name string, ff *flag.Flag, set *flag.FlagSet) {
+	switch ff.Value.(type) {
+	case *cli.StringSlice:
+	default:
+		set.Set(name, ff.Value.String())
+	}
+}
+
+func normalizeFlags(flags []cli.Flag, set *flag.FlagSet) error {
+	visited := make(map[string]bool)
+	set.Visit(func(f *flag.Flag) {
+		visited[f.Name] = true
+	})
+	for _, f := range flags {
+		parts := strings.Split(f.GetName(), ",")
+		if len(parts) == 1 {
+			continue
+		}
+		var ff *flag.Flag
+		for _, name := range parts {
+			name = strings.Trim(name, " ")
+			if visited[name] {
+				if ff != nil {
+					return errors.New("Cannot use two forms of the same flag: " + name + " " + ff.Name)
+				}
+				ff = set.Lookup(name)
+			}
+		}
+		if ff == nil {
+			continue
+		}
+		for _, name := range parts {
+			name = strings.Trim(name, " ")
+			if !visited[name] {
+				copyFlag(name, ff, set)
+			}
+		}
+	}
+	return nil
+}

config/source/consul/README.md

@@ -0,0 +1,49 @@
+# Consul Source
+
+The consul source reads config from consul key/values
+
+## Consul Format
+
+The consul source expects keys under the default prefix `/micro/config`
+
+Values are expected to be json
+
+```
+// set database
+consul kv put micro/config/database '{"address": "10.0.0.1", "port": 3306}'
+// set cache
+consul kv put micro/config/cache '{"address": "10.0.0.2", "port": 6379}'
+```
+
+Keys are split on `/` so access becomes
+
+```
+conf.Get("micro", "config", "database")
+```
+
+## New Source
+
+Specify source with data
+
+```go
+consulSource := consul.NewSource(
+	// optionally specify consul address; default to localhost:8500
+	consul.WithAddress("10.0.0.10:8500"),
+	// optionally specify prefix; defaults to /micro/config
+	consul.WithPrefix("/my/prefix"),
+  // optionally strip the provided prefix from the keys, defaults to false
+  consul.StripPrefix(true),
+)
+```
+
+## Load Source
+
+Load the source into config
+
+```go
+// Create new config
+conf := config.NewConfig()
+
+// Load file source
+conf.Load(consulSource)
+```

config/source/consul/consul.go

@@ -0,0 +1,121 @@
+package consul
+
+import (
+	"fmt"
+	"net"
+	"time"
+
+	"github.com/hashicorp/consul/api"
+	"github.com/micro/go-micro/config/source"
+)
+
+// Currently a single consul reader
+type consul struct {
+	prefix      string
+	stripPrefix string
+	addr        string
+	opts        source.Options
+	client      *api.Client
+}
+
+var (
+	// DefaultPrefix is the prefix that consul keys will be assumed to have if you
+	// haven't specified one
+	DefaultPrefix = "/micro/config/"
+)
+
+func (c *consul) Read() (*source.ChangeSet, error) {
+	kv, _, err := c.client.KV().List(c.prefix, nil)
+	if err != nil {
+		return nil, err
+	}
+
+	if kv == nil || len(kv) == 0 {
+		return nil, fmt.Errorf("source not found: %s", c.prefix)
+	}
+
+	data, err := makeMap(c.opts.Encoder, kv, c.stripPrefix)
+	if err != nil {
+		return nil, fmt.Errorf("error reading data: %v", err)
+	}
+
+	b, err := c.opts.Encoder.Encode(data)
+	if err != nil {
+		return nil, fmt.Errorf("error reading source: %v", err)
+	}
+
+	cs := &source.ChangeSet{
+		Timestamp: time.Now(),
+		Format:    c.opts.Encoder.String(),
+		Source:    c.String(),
+		Data:      b,
+	}
+	cs.Checksum = cs.Sum()
+
+	return cs, nil
+}
+
+func (c *consul) String() string {
+	return "consul"
+}
+
+func (c *consul) Watch() (source.Watcher, error) {
+	w, err := newWatcher(c.prefix, c.addr, c.String(), c.stripPrefix, c.opts.Encoder)
+	if err != nil {
+		return nil, err
+	}
+	return w, nil
+}
+
+// NewSource creates a new consul source
+func NewSource(opts ...source.Option) source.Source {
+	options := source.NewOptions(opts...)
+
+	// use default config
+	config := api.DefaultConfig()
+
+	// check if there are any addrs
+	a, ok := options.Context.Value(addressKey{}).(string)
+	if ok {
+		addr, port, err := net.SplitHostPort(a)
+		if ae, ok := err.(*net.AddrError); ok && ae.Err == "missing port in address" {
+			port = "8500"
+			addr = a
+			config.Address = fmt.Sprintf("%s:%s", addr, port)
+		} else if err == nil {
+			config.Address = fmt.Sprintf("%s:%s", addr, port)
+		}
+	}
+
+	dc, ok := options.Context.Value(dcKey{}).(string)
+	if ok {
+		config.Datacenter = dc
+	}
+
+	token, ok := options.Context.Value(tokenKey{}).(string)
+	if ok {
+		config.Token = token
+	}
+
+	// create the client
+	client, _ := api.NewClient(config)
+
+	prefix := DefaultPrefix
+	sp := ""
+	f, ok := options.Context.Value(prefixKey{}).(string)
+	if ok {
+		prefix = f
+	}
+
+	if b, ok := options.Context.Value(stripPrefixKey{}).(bool); ok && b {
+		sp = prefix
+	}
+
+	return &consul{
+		prefix:      prefix,
+		stripPrefix: sp,
+		addr:        config.Address,
+		opts:        options,
+		client:      client,
+	}
+}

config/source/consul/options.go

@@ -0,0 +1,63 @@
+package consul
+
+import (
+	"context"
+
+	"github.com/micro/go-micro/config/source"
+)
+
+type addressKey struct{}
+type prefixKey struct{}
+type stripPrefixKey struct{}
+type dcKey struct{}
+type tokenKey struct{}
+
+// WithAddress sets the consul address
+func WithAddress(a string) source.Option {
+	return func(o *source.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, addressKey{}, a)
+	}
+}
+
+// WithPrefix sets the key prefix to use
+func WithPrefix(p string) source.Option {
+	return func(o *source.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, prefixKey{}, p)
+	}
+}
+
+// StripPrefix indicates whether to remove the prefix from config entries, or leave it in place.
+func StripPrefix(strip bool) source.Option {
+	return func(o *source.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+
+		o.Context = context.WithValue(o.Context, stripPrefixKey{}, strip)
+	}
+}
+
+func WithDatacenter(p string) source.Option {
+	return func(o *source.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, dcKey{}, p)
+	}
+}
+
+// WithToken sets the key token to use
+func WithToken(p string) source.Option {
+	return func(o *source.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, tokenKey{}, p)
+	}
+}

config/source/consul/util.go

@@ -0,0 +1,49 @@
+package consul
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/hashicorp/consul/api"
+	"github.com/micro/go-micro/config/encoder"
+)
+
+func makeMap(e encoder.Encoder, kv api.KVPairs, stripPrefix string) (map[string]interface{}, error) {
+	data := make(map[string]interface{})
+
+	// consul guarantees lexicographic order, so no need to sort
+	for _, v := range kv {
+		pathString := strings.TrimPrefix(strings.TrimPrefix(v.Key, stripPrefix), "/")
+		var val map[string]interface{}
+
+		// ensure a valid value is stored at this location
+		if len(v.Value) > 0 {
+			if err := e.Decode(v.Value, &val); err != nil {
+				return nil, fmt.Errorf("faild decode value. path: %s, error: %s", pathString, err)
+			}
+		}
+
+		// set target at the root
+		target := data
+
+		// then descend to the target location, creating as we go, if need be
+		if pathString != "" {
+			path := strings.Split(pathString, "/")
+			// find (or create) the location we want to put this value at
+			for _, dir := range path {
+				if _, ok := target[dir]; !ok {
+					target[dir] = make(map[string]interface{})
+				}
+				target = target[dir].(map[string]interface{})
+			}
+
+		}
+
+		// copy over the keys from the value
+		for k := range val {
+			target[k] = val[k]
+		}
+	}
+
+	return data, nil
+}

config/source/consul/watcher.go

@@ -0,0 +1,96 @@
+package consul
+
+import (
+	"errors"
+	"time"
+
+	"github.com/hashicorp/consul/api"
+	"github.com/hashicorp/consul/api/watch"
+	"github.com/micro/go-micro/config/encoder"
+	"github.com/micro/go-micro/config/source"
+)
+
+type watcher struct {
+	e           encoder.Encoder
+	name        string
+	stripPrefix string
+
+	wp   *watch.Plan
+	ch   chan *source.ChangeSet
+	exit chan bool
+}
+
+func newWatcher(key, addr, name, stripPrefix string, e encoder.Encoder) (source.Watcher, error) {
+	w := &watcher{
+		e:           e,
+		name:        name,
+		stripPrefix: stripPrefix,
+		ch:          make(chan *source.ChangeSet),
+		exit:        make(chan bool),
+	}
+
+	wp, err := watch.Parse(map[string]interface{}{"type": "keyprefix", "prefix": key})
+	if err != nil {
+		return nil, err
+	}
+
+	wp.Handler = w.handle
+
+	// wp.Run is a blocking call and will prevent newWatcher from returning
+	go wp.Run(addr)
+
+	w.wp = wp
+
+	return w, nil
+}
+
+func (w *watcher) handle(idx uint64, data interface{}) {
+	if data == nil {
+		return
+	}
+
+	kvs, ok := data.(api.KVPairs)
+	if !ok {
+		return
+	}
+
+	d, err := makeMap(w.e, kvs, w.stripPrefix)
+	if err != nil {
+		return
+	}
+
+	b, err := w.e.Encode(d)
+	if err != nil {
+		return
+	}
+
+	cs := &source.ChangeSet{
+		Timestamp: time.Now(),
+		Format:    w.e.String(),
+		Source:    w.name,
+		Data:      b,
+	}
+	cs.Checksum = cs.Sum()
+
+	w.ch <- cs
+}
+
+func (w *watcher) Next() (*source.ChangeSet, error) {
+	select {
+	case cs := <-w.ch:
+		return cs, nil
+	case <-w.exit:
+		return nil, errors.New("watcher stopped")
+	}
+}
+
+func (w *watcher) Stop() error {
+	select {
+	case <-w.exit:
+		return nil
+	default:
+		w.wp.Stop()
+		close(w.exit)
+	}
+	return nil
+}

config/source/env/README.md

@@ -0,0 +1,96 @@
+# Env Source
+
+The env source reads config from environment variables
+
+## Format
+
+We expect environment variables to be in the standard format of FOO=bar
+
+Keys are converted to lowercase and split on underscore.
+
+
+### Example
+
+```
+DATABASE_ADDRESS=127.0.0.1
+DATABASE_PORT=3306
+```
+
+Becomes
+
+```json
+{
+    "database": {
+        "address": "127.0.0.1",
+        "port": 3306
+    }
+}
+```
+
+## Prefixes
+
+Environment variables can be namespaced so we only have access to a subset. Two options are available:
+
+```
+WithPrefix(p ...string)
+WithStrippedPrefix(p ...string)
+```
+
+The former will preserve the prefix and make it a top level key in the config. The latter eliminates the prefix, reducing the nesting by one. 
+
+#### Example:
+
+Given ENVs of:
+
+```
+APP_DATABASE_ADDRESS=127.0.0.1
+APP_DATABASE_PORT=3306
+VAULT_ADDR=vault:1337
+```
+
+and a source initialized as follows:
+
+```
+src := env.NewSource(
+    env.WithPrefix("VAULT"),
+    env.WithStrippedPrefix("APP"),
+)
+```
+
+The resulting config will be:
+
+```
+{
+    "database": {
+        "address": "127.0.0.1",
+        "port": 3306
+    },
+    "vault": {
+        "addr": "vault:1337"
+    }
+}
+```
+
+
+## New Source
+
+Specify source with data
+
+```go
+src := env.NewSource(
+	// optionally specify prefix
+	env.WithPrefix("MICRO"),
+)
+```
+
+## Load Source
+
+Load the source into config
+
+```go
+// Create new config
+conf := config.NewConfig()
+
+// Load file source
+conf.Load(src)
+```

config/source/env/env.go

@@ -0,0 +1,142 @@
+package env
+
+import (
+	"os"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/imdario/mergo"
+	"github.com/micro/go-micro/config/source"
+)
+
+var (
+	DefaultPrefixes = []string{}
+)
+
+type env struct {
+	prefixes         []string
+	strippedPrefixes []string
+	opts             source.Options
+}
+
+func (e *env) Read() (*source.ChangeSet, error) {
+	var changes map[string]interface{}
+
+	for _, env := range os.Environ() {
+
+		if len(e.prefixes) > 0 || len(e.strippedPrefixes) > 0 {
+			notFound := true
+
+			if _, ok := matchPrefix(e.prefixes, env); ok {
+				notFound = false
+			}
+
+			if match, ok := matchPrefix(e.strippedPrefixes, env); ok {
+				env = strings.TrimPrefix(env, match)
+				notFound = false
+			}
+
+			if notFound {
+				continue
+			}
+		}
+
+		pair := strings.SplitN(env, "=", 2)
+		value := pair[1]
+		keys := strings.Split(strings.ToLower(pair[0]), "_")
+		reverse(keys)
+
+		tmp := make(map[string]interface{})
+		for i, k := range keys {
+			if i == 0 {
+				if intValue, err := strconv.Atoi(value); err == nil {
+					tmp[k] = intValue
+				} else if boolValue, err := strconv.ParseBool(value); err == nil {
+					tmp[k] = boolValue
+				} else {
+					tmp[k] = value
+				}
+				continue
+			}
+
+			tmp = map[string]interface{}{k: tmp}
+		}
+
+		if err := mergo.Map(&changes, tmp); err != nil {
+			return nil, err
+		}
+	}
+
+	b, err := e.opts.Encoder.Encode(changes)
+	if err != nil {
+		return nil, err
+	}
+
+	cs := &source.ChangeSet{
+		Format:    e.opts.Encoder.String(),
+		Data:      b,
+		Timestamp: time.Now(),
+		Source:    e.String(),
+	}
+	cs.Checksum = cs.Sum()
+
+	return cs, nil
+}
+
+func matchPrefix(pre []string, s string) (string, bool) {
+	for _, p := range pre {
+		if strings.HasPrefix(s, p) {
+			return p, true
+		}
+	}
+
+	return "", false
+}
+
+func reverse(ss []string) {
+	for i := len(ss)/2 - 1; i >= 0; i-- {
+		opp := len(ss) - 1 - i
+		ss[i], ss[opp] = ss[opp], ss[i]
+	}
+}
+
+func (e *env) Watch() (source.Watcher, error) {
+	return newWatcher()
+}
+
+func (e *env) String() string {
+	return "env"
+}
+
+// NewSource returns a config source for parsing ENV variables.
+// Underscores are delimiters for nesting, and all keys are lowercased.
+//
+// Example:
+//      "DATABASE_SERVER_HOST=localhost" will convert to
+//
+//      {
+//          "database": {
+//              "server": {
+//                  "host": "localhost"
+//              }
+//          }
+//      }
+func NewSource(opts ...source.Option) source.Source {
+	options := source.NewOptions(opts...)
+
+	var sp []string
+	var pre []string
+	if p, ok := options.Context.Value(strippedPrefixKey{}).([]string); ok {
+		sp = p
+	}
+
+	if p, ok := options.Context.Value(prefixKey{}).([]string); ok {
+		pre = p
+	}
+
+	if len(sp) > 0 || len(pre) > 0 {
+		pre = append(pre, DefaultPrefixes...)
+	}
+	return &env{prefixes: pre, strippedPrefixes: sp, opts: options}
+}

config/source/env/env_test.go

@@ -0,0 +1,112 @@
+package env
+
+import (
+	"encoding/json"
+	"os"
+	"testing"
+	"time"
+
+	"github.com/micro/go-micro/config/source"
+)
+
+func TestEnv_Read(t *testing.T) {
+	expected := map[string]map[string]string{
+		"database": {
+			"host":       "localhost",
+			"password":   "password",
+			"datasource": "user:password@tcp(localhost:port)/db?charset=utf8mb4&parseTime=True&loc=Local",
+		},
+	}
+
+	os.Setenv("DATABASE_HOST", "localhost")
+	os.Setenv("DATABASE_PASSWORD", "password")
+	os.Setenv("DATABASE_DATASOURCE", "user:password@tcp(localhost:port)/db?charset=utf8mb4&parseTime=True&loc=Local")
+
+	source := NewSource()
+	c, err := source.Read()
+	if err != nil {
+		t.Error(err)
+	}
+
+	var actual map[string]interface{}
+	if err := json.Unmarshal(c.Data, &actual); err != nil {
+		t.Error(err)
+	}
+
+	actualDB := actual["database"].(map[string]interface{})
+
+	for k, v := range expected["database"] {
+		a := actualDB[k]
+
+		if a != v {
+			t.Errorf("expected %v got %v", v, a)
+		}
+	}
+}
+
+func TestEnvvar_Prefixes(t *testing.T) {
+	os.Setenv("APP_DATABASE_HOST", "localhost")
+	os.Setenv("APP_DATABASE_PASSWORD", "password")
+	os.Setenv("VAULT_ADDR", "vault:1337")
+	os.Setenv("MICRO_REGISTRY", "mdns")
+
+	var prefixtests = []struct {
+		prefixOpts   []source.Option
+		expectedKeys []string
+	}{
+		{[]source.Option{WithPrefix("APP", "MICRO")}, []string{"app", "micro"}},
+		{[]source.Option{WithPrefix("MICRO"), WithStrippedPrefix("APP")}, []string{"database", "micro"}},
+		{[]source.Option{WithPrefix("MICRO"), WithStrippedPrefix("APP")}, []string{"database", "micro"}},
+	}
+
+	for _, pt := range prefixtests {
+		source := NewSource(pt.prefixOpts...)
+
+		c, err := source.Read()
+		if err != nil {
+			t.Error(err)
+		}
+
+		var actual map[string]interface{}
+		if err := json.Unmarshal(c.Data, &actual); err != nil {
+			t.Error(err)
+		}
+
+		// assert other prefixes ignored
+		if l := len(actual); l != len(pt.expectedKeys) {
+			t.Errorf("expected %v top keys, got %v", len(pt.expectedKeys), l)
+		}
+
+		for _, k := range pt.expectedKeys {
+			if !containsKey(actual, k) {
+				t.Errorf("expected key %v, not found", k)
+			}
+		}
+	}
+}
+
+func TestEnvvar_WatchNextNoOpsUntilStop(t *testing.T) {
+	source := NewSource(WithStrippedPrefix("GOMICRO_"))
+	w, err := source.Watch()
+	if err != nil {
+		t.Error(err)
+	}
+
+	go func() {
+		time.Sleep(50 * time.Millisecond)
+		w.Stop()
+	}()
+
+	if _, err := w.Next(); err.Error() != "watcher stopped" {
+		t.Errorf("expected watcher stopped error, got %v", err)
+	}
+}
+
+func containsKey(m map[string]interface{}, s string) bool {
+	for k := range m {
+		if k == s {
+			return true
+		}
+	}
+	return false
+}

config/source/env/options.go

@@ -0,0 +1,49 @@
+package env
+
+import (
+	"context"
+
+	"strings"
+
+	"github.com/micro/go-micro/config/source"
+)
+
+type strippedPrefixKey struct{}
+type prefixKey struct{}
+
+// WithStrippedPrefix sets the environment variable prefixes to scope to.
+// These prefixes will be removed from the actual config entries.
+func WithStrippedPrefix(p ...string) source.Option {
+	return func(o *source.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+
+		o.Context = context.WithValue(o.Context, strippedPrefixKey{}, appendUnderscore(p))
+	}
+}
+
+// WithPrefix sets the environment variable prefixes to scope to.
+// These prefixes will not be removed. Each prefix will be considered a top level config entry.
+func WithPrefix(p ...string) source.Option {
+	return func(o *source.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, prefixKey{}, appendUnderscore(p))
+	}
+}
+
+func appendUnderscore(prefixes []string) []string {
+	var result []string
+	for _, p := range prefixes {
+		if !strings.HasSuffix(p, "_") {
+			result = append(result, p+"_")
+			continue
+		}
+
+		result = append(result, p)
+	}
+
+	return result
+}

config/source/env/watcher.go

@@ -0,0 +1,26 @@
+package env
+
+import (
+	"errors"
+
+	"github.com/micro/go-micro/config/source"
+)
+
+type watcher struct {
+	exit chan struct{}
+}
+
+func (w *watcher) Next() (*source.ChangeSet, error) {
+	<-w.exit
+
+	return nil, errors.New("watcher stopped")
+}
+
+func (w *watcher) Stop() error {
+	close(w.exit)
+	return nil
+}
+
+func newWatcher() (source.Watcher, error) {
+	return &watcher{exit: make(chan struct{})}, nil
+}

config/source/etcd/README.md

@@ -0,0 +1,51 @@
+# Etcd Source
+
+The etcd source reads config from etcd key/values
+
+This source supports etcd version 3 and beyond.
+
+## Etcd Format
+
+The etcd source expects keys under the default prefix `/micro/config` (prefix can be changed)
+
+Values are expected to be JSON
+
+```
+// set database
+etcdctl put /micro/config/database '{"address": "10.0.0.1", "port": 3306}'
+// set cache
+etcdctl put /micro/config/cache '{"address": "10.0.0.2", "port": 6379}'
+```
+
+Keys are split on `/` so access becomes
+
+```
+conf.Get("micro", "config", "database")
+```
+
+## New Source
+
+Specify source with data
+
+```go
+etcdSource := etcd.NewSource(
+	// optionally specify etcd address; default to localhost:8500
+	etcd.WithAddress("10.0.0.10:8500"),
+	// optionally specify prefix; defaults to /micro/config
+	etcd.WithPrefix("/my/prefix"),
+	// optionally strip the provided prefix from the keys, defaults to false
+	etcd.StripPrefix(true),
+)
+```
+
+## Load Source
+
+Load the source into config
+
+```go
+// Create new config
+conf := config.NewConfig()
+
+// Load file source
+conf.Load(etcdSource)
+```

config/source/etcd/etcd.go

@@ -0,0 +1,134 @@
+package etcd
+
+import (
+	"context"
+	"fmt"
+	"net"
+	"time"
+
+	"github.com/micro/go-micro/config/source"
+	cetcd "go.etcd.io/etcd/clientv3"
+	"go.etcd.io/etcd/mvcc/mvccpb"
+)
+
+// Currently a single etcd reader
+type etcd struct {
+	prefix      string
+	stripPrefix string
+	opts        source.Options
+	client      *cetcd.Client
+	cerr        error
+}
+
+var (
+	DefaultPrefix = "/micro/config/"
+)
+
+func (c *etcd) Read() (*source.ChangeSet, error) {
+	if c.cerr != nil {
+		return nil, c.cerr
+	}
+
+	rsp, err := c.client.Get(context.Background(), c.prefix, cetcd.WithPrefix())
+	if err != nil {
+		return nil, err
+	}
+
+	if rsp == nil || len(rsp.Kvs) == 0 {
+		return nil, fmt.Errorf("source not found: %s", c.prefix)
+	}
+
+	var kvs []*mvccpb.KeyValue
+	for _, v := range rsp.Kvs {
+		kvs = append(kvs, (*mvccpb.KeyValue)(v))
+	}
+
+	data := makeMap(c.opts.Encoder, kvs, c.stripPrefix)
+
+	b, err := c.opts.Encoder.Encode(data)
+	if err != nil {
+		return nil, fmt.Errorf("error reading source: %v", err)
+	}
+
+	cs := &source.ChangeSet{
+		Timestamp: time.Now(),
+		Source:    c.String(),
+		Data:      b,
+		Format:    c.opts.Encoder.String(),
+	}
+	cs.Checksum = cs.Sum()
+
+	return cs, nil
+}
+
+func (c *etcd) String() string {
+	return "etcd"
+}
+
+func (c *etcd) Watch() (source.Watcher, error) {
+	if c.cerr != nil {
+		return nil, c.cerr
+	}
+	cs, err := c.Read()
+	if err != nil {
+		return nil, err
+	}
+	return newWatcher(c.prefix, c.stripPrefix, c.client.Watcher, cs, c.opts)
+}
+
+func NewSource(opts ...source.Option) source.Source {
+	options := source.NewOptions(opts...)
+
+	var endpoints []string
+
+	// check if there are any addrs
+	addrs, ok := options.Context.Value(addressKey{}).([]string)
+	if ok {
+		for _, a := range addrs {
+			addr, port, err := net.SplitHostPort(a)
+			if ae, ok := err.(*net.AddrError); ok && ae.Err == "missing port in address" {
+				port = "2379"
+				addr = a
+				endpoints = append(endpoints, fmt.Sprintf("%s:%s", addr, port))
+			} else if err == nil {
+				endpoints = append(endpoints, fmt.Sprintf("%s:%s", addr, port))
+			}
+		}
+	}
+
+	if len(endpoints) == 0 {
+		endpoints = []string{"localhost:2379"}
+	}
+
+	config := cetcd.Config{
+		Endpoints: endpoints,
+	}
+
+	u, ok := options.Context.Value(authKey{}).(*authCreds)
+	if ok {
+		config.Username = u.Username
+		config.Password = u.Password
+	}
+
+	// use default config
+	client, err := cetcd.New(config)
+
+	prefix := DefaultPrefix
+	sp := ""
+	f, ok := options.Context.Value(prefixKey{}).(string)
+	if ok {
+		prefix = f
+	}
+
+	if b, ok := options.Context.Value(stripPrefixKey{}).(bool); ok && b {
+		sp = prefix
+	}
+
+	return &etcd{
+		prefix:      prefix,
+		stripPrefix: sp,
+		opts:        options,
+		client:      client,
+		cerr:        err,
+	}
+}