gofmt

Update marshaler.go of codec/proto.

.travis.yml

@@ -1,7 +1,9 @@
 language: go
 go:
-- 1.8.5
-- 1.9.2
+- 1.11.x
+- 1.12.x
+env:
+  - GO111MODULE=on
 notifications:
   slack:
     secure: aEvhLbhujaGaKSrOokiG3//PaVHTIrc3fBpoRbCRqfZpyq6WREoapJJhF+tIpWWOwaC9GmChbD6aHo/jMUgwKXVyPSaNjiEL87YzUUpL8B2zslNp1rgfTg/LrzthOx3Q1TYwpaAl3to0fuHUVFX4yMeC2vuThq7WSXgMMxFCtbc=

README.md

@@ -1,10 +1,16 @@
 # Go Micro [![License](https://img.shields.io/:license-apache-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![GoDoc](https://godoc.org/github.com/micro/go-micro?status.svg)](https://godoc.org/github.com/micro/go-micro) [![Travis CI](https://api.travis-ci.org/micro/go-micro.svg?branch=master)](https://travis-ci.org/micro/go-micro) [![Go Report Card](https://goreportcard.com/badge/micro/go-micro)](https://goreportcard.com/report/github.com/micro/go-micro)
 
-Go Micro is a pluggable RPC framework for distributed systems development.
+Go Micro is a framework for microservice development.
 
-The **Micro** philosophy is sane defaults with a pluggable architecture. We provide defaults to get you started quickly but everything can be easily swapped out. It comes with built in support for {json,proto}-rpc encoding, consul or multicast dns for service discovery, http for communication and random hashed client side load balancing.
+## Overview
 
-Everything in go-micro is **pluggable**. You can find and contribute to plugins at [github.com/micro/go-plugins](https://github.com/micro/go-plugins).
+Go Micro provides the core requirements for distributed systems development including RPC and Event driven communication. 
+The **micro** philosophy is sane defaults with a pluggable architecture. We provide defaults to get you started quickly 
+but everything can be easily swapped out. 
+
+<img src="https://micro.mu/docs/images/go-micro.svg" />
+
+Plugins are available at [github.com/micro/go-plugins](https://github.com/micro/go-plugins).
 
 Follow us on [Twitter](https://twitter.com/microhq) or join the [Slack](http://slack.micro.mu/) community.
 
@@ -12,405 +18,32 @@ Follow us on [Twitter](https://twitter.com/microhq) or join the [Slack](http://s
 
 Go Micro abstracts away the details of distributed systems. Here are the main features.
 
-- **Service Discovery** - Automatic service registration and name resolution
-- **Load Balancing** - Client side load balancing built on discovery
-- **Sync Comms** - RPC based communication with support for bidirectional streaming
-- **Async Comms** - Native PubSub messaging built in for event driven architectures
-- **Message Encoding** - Dynamic encoding based on content-type with protobuf and json out of the box
-- **Service Interface** - All features are wrapped up in a simple high level interface
+- **Service Discovery** - Automatic service registration and name resolution. Service discovery is at the core of micro service 
+development. When service A needs to speak to service B it needs the location of that service. The default discovery mechanism is 
+multicast DNS (mdns), a zeroconf system. You can optionally set gossip using the SWIM protocol for p2p networks or consul for a
+resilient cloud-native setup.
 
-Go Micro supports both the Service and Function programming models. Read on to learn more.
+- **Load Balancing** - Client side load balancing built on service discovery. Once we have the addresses of any number of instances 
+of a service we now need a way to decide which node to route to. We use random hashed load balancing to provide even distribution 
+across the services and retry a different node if there's a problem. 
 
-## Docs
+- **Message Encoding** - Dynamic message encoding based on content-type. The client and server will use codecs along with content-type 
+to seamlessly encode and decode Go types for you. Any variety of messages could be encoded and sent from different clients. The client 
+and server handle this by default. This includes protobuf and json by default.
 
-For more detailed information on the architecture, installation and use of go-micro checkout the [docs](https://micro.mu/docs).
+- **Request/Response** - RPC based request/response with support for bidirectional streaming. We provide an abstraction for synchronous 
+communication. A request made to a service will be automatically resolved, load balanced, dialled and streamed. The default 
+transport is http/1.1 or http2 when tls is enabled.
 
-## Learn By Example
+- **Async Messaging** - PubSub is built in as a first class citizen for asynchronous communication and event driven architectures. 
+Event notifications are a core pattern in micro service development. The default messaging is point-to-point http/1.1 or http2 when tls 
+is enabled.
 
-An example service can be found in [**examples/service**](https://github.com/micro/examples/tree/master/service) and function in [**examples/function**](https://github.com/micro/examples/tree/master/function). 
+- **Pluggable Interfaces** - Go Micro makes use of Go interfaces for each distributed system abstraction. Because of this these interfaces 
+are pluggable and allows Go Micro to be runtime agnostic. You can plugin any underlying technology. Find plugins in 
+[github.com/micro/go-plugins](https://github.com/micro/go-plugins).
 
-The [**examples**](https://github.com/micro/examples) directory contains examples for using things such as middleware/wrappers, selector filters, pub/sub, grpc, plugins and much more. For the complete greeter example look at [**examples/greeter**](https://github.com/micro/examples/tree/master/greeter). Other examples can be found throughout the GitHub repository.
+## Getting Started
 
-Watch the [Golang UK Conf 2016](https://www.youtube.com/watch?v=xspaDovwk34) video for a high level overview.
+See the [docs](https://micro.mu/docs/go-micro.html) for detailed information on the architecture, installation and use of go-micro.
 
-## Getting started
-
-- [Service Discovery](#service-discovery)
-- [Writing a Service](#writing-a-service)
-- [Writing a Function](#writing-a-function)
-- [Plugins](#plugins)
-- [Wrappers](#wrappers)
-
-## Service Discovery
-
-Service discovery is used to resolve service names to addresses. It's the only dependency of go-micro.
-
-### Consul
-
-[Consul](https://www.consul.io/) is used as the default service discovery system. 
-
-Discovery is pluggable. Find plugins for etcd, kubernetes, zookeeper and more in the [micro/go-plugins](https://github.com/micro/go-plugins) repo.
-
-[Install guide](https://www.consul.io/intro/getting-started/install.html)
-
-### Multicast DNS
-
-[Multicast DNS](https://en.wikipedia.org/wiki/Multicast_DNS) is a built in service discovery plugin for a zero dependency configuration. 
-
-Pass `--registry=mdns` to any command or the enviroment variable `MICRO_REGISTRY=mdns`
-
-```
-go run main.go --registry=mdns
-```
-
-## Writing a service
-
-This is a simple greeter RPC service example
-
-Find this example at [examples/service](https://github.com/micro/examples/tree/master/service).
-
-### Create service proto
-
-One of the key requirements of microservices is strongly defined interfaces. Micro uses protobuf to achieve this.
-
-Here we define the Greeter handler with the method Hello. It takes a HelloRequest and HelloResponse both with one string arguments.
-
-```proto
-syntax = "proto3";
-
-service Greeter {
-	rpc Hello(HelloRequest) returns (HelloResponse) {}
-}
-
-message HelloRequest {
-	string name = 1;
-}
-
-message HelloResponse {
-	string greeting = 2;
-}
-```
-
-### Install protobuf
-
-Install [protobuf](https://developers.google.com/protocol-buffers/)
-
-Now install the micro fork of protoc-gen-go. The protobuf compiler for Go. 
-
-```shell
-go get github.com/micro/protobuf/{proto,protoc-gen-go}
-```
-
-### Generate the proto
-
-After writing the proto definition we must compile it using protoc with the micro plugin.
-
-```shell
-protoc -I$GOPATH/src --go_out=plugins=micro:$GOPATH/src \
-	$GOPATH/src/github.com/micro/examples/service/proto/greeter.proto
-```
-
-### Write the service
-
-Below is the code for the greeter service. 
-
-It does the following:
-
-1. Implements the interface defined for the Greeter handler
-2. Initialises a micro.Service
-3. Registers the Greeter handler
-4. Runs the service
-
-```go
-package main
-
-import (
-	"fmt"
-
-	micro "github.com/micro/go-micro"
-	proto "github.com/micro/examples/service/proto"
-	"golang.org/x/net/context"
-)
-
-type Greeter struct{}
-
-func (g *Greeter) Hello(ctx context.Context, req *proto.HelloRequest, rsp *proto.HelloResponse) error {
-	rsp.Greeting = "Hello " + req.Name
-	return nil
-}
-
-func main() {
-	// Create a new service. Optionally include some options here.
-	service := micro.NewService(
-		micro.Name("greeter"),
-	)
-
-	// Init will parse the command line flags.
-	service.Init()
-
-	// Register handler
-	proto.RegisterGreeterHandler(service.Server(), new(Greeter))
-
-	// Run the server
-	if err := service.Run(); err != nil {
-		fmt.Println(err)
-	}
-}
-```
-
-### Run service
-```
-go run examples/service/main.go
-```
-
-Output
-```
-2016/03/14 10:59:14 Listening on [::]:50137
-2016/03/14 10:59:14 Broker Listening on [::]:50138
-2016/03/14 10:59:14 Registering node: greeter-ca62b017-e9d3-11e5-9bbb-68a86d0d36b6
-```
-
-### Define a client
-
-Below is the client code to query the greeter service. 
-
-The generated proto includes a greeter client to reduce boilerplate code.
-
-```go
-package main
-
-import (
-	"fmt"
-
-	micro "github.com/micro/go-micro"
-	proto "github.com/micro/examples/service/proto"
-	"golang.org/x/net/context"
-)
-
-
-func main() {
-	// Create a new service. Optionally include some options here.
-	service := micro.NewService(micro.Name("greeter.client"))
-
-	// Create new greeter client
-	greeter := proto.NewGreeterClient("greeter", service.Client())
-
-	// Call the greeter
-	rsp, err := greeter.Hello(context.TODO(), &proto.HelloRequest{Name: "John"})
-	if err != nil {
-		fmt.Println(err)
-	}
-
-	// Print response
-	fmt.Println(rsp.Greeting)
-}
-```
-
-### Run the client
-
-```shell
-go run client.go
-```
-
-Output
-```
-Hello John
-```
-
-## Writing a Function
-
-Go Micro includes the Function programming model. 
-
-A Function is a one time executing Service which exits after completing a request. 
-
-### Defining a Function
-
-```go
-package main
-
-import (
-	proto "github.com/micro/examples/function/proto"
-	"github.com/micro/go-micro"
-	"golang.org/x/net/context"
-)
-
-type Greeter struct{}
-
-func (g *Greeter) Hello(ctx context.Context, req *proto.HelloRequest, rsp *proto.HelloResponse) error {
-	rsp.Greeting = "Hello " + req.Name
-	return nil
-}
-
-func main() {
-	// create a new function
-	fnc := micro.NewFunction(
-		micro.Name("go.micro.fnc.greeter"),
-	)
-
-	// init the command line
-	fnc.Init()
-
-	// register a handler
-	fnc.Handle(new(Greeter))
-
-	// run the function
-	fnc.Run()
-}
-```
-
-It's that simple.
-
-## Plugins
-
-By default go-micro only provides a few implementation of each interface at the core but it's completely pluggable. There's already dozens of plugins which are available at [github.com/micro/go-plugins](https://github.com/micro/go-plugins). Contributions are welcome!
-
-### Build with plugins
-
-If you want to integrate plugins simply link them in a separate file and rebuild
-
-Create a plugins.go file
-
-```go
-import (
-        // etcd v3 registry
-        _ "github.com/micro/go-plugins/registry/etcdv3"
-        // nats transport
-        _ "github.com/micro/go-plugins/transport/nats"
-        // kafka broker
-        _ "github.com/micro/go-plugins/broker/kafka"
-)
-```
-
-Build binary
-
-```shell
-// For local use
-go build -i -o service ./main.go ./plugins.go
-```
-
-Flag usage of plugins
-```shell
-service --registry=etcdv3 --transport=nats --broker=kafka
-```
-
-### Plugin as option
-
-Alternatively you can set the plugin as an option to a service
-
-```go
-
-import (
-        "github.com/micro/go-micro" 
-        // etcd v3 registry
-        "github.com/micro/go-plugins/registry/etcdv3"
-        // nats transport
-        "github.com/micro/go-plugins/transport/nats"
-        // kafka broker
-        "github.com/micro/go-plugins/broker/kafka"
-)
-
-func main() {
-	registry := etcdv3.NewRegistry()
-	broker := kafka.NewBroker()
-	transport := nats.NewTransport()
-
-        service := micro.NewService(
-                micro.Name("greeter"),
-                micro.Registry(registry),
-                micro.Broker(broker),
-                micro.Transport(transport),
-        )
-
-	service.Init()
-	service.Run()
-}
-```
-
-### Write plugins
-
-Plugins are a concept built on Go's interface. Each package maintains a high level interface abstraction. 
-Simply implement the interface and pass it in as an option to the service.
-
-The service discovery interface is called [Registry](https://godoc.org/github.com/micro/go-micro/registry#Registry). 
-Anything which implements this interface can be used as a registry. The same applies to the other packages.
-
-```go
-type Registry interface {
-    Register(*Service, ...RegisterOption) error
-    Deregister(*Service) error
-    GetService(string) ([]*Service, error)
-    ListServices() ([]*Service, error)
-    Watch() (Watcher, error)
-    String() string
-}
-```
-
-Browse [go-plugins](https://github.com/micro/go-plugins) to get a better idea of implementation details.
-
-## Wrappers
-
-Go-micro includes the notion of middleware as wrappers. The client or handlers can be wrapped using the decorator pattern.
-
-### Handler
-
-Here's an example service handler wrapper which logs the incoming request
-
-```go
-// implements the server.HandlerWrapper
-func logWrapper(fn server.HandlerFunc) server.HandlerFunc {
-	return func(ctx context.Context, req server.Request, rsp interface{}) error {
-		fmt.Printf("[%v] server request: %s", time.Now(), req.Method())
-		return fn(ctx, req, rsp)
-	}
-}
-```
-
-It can be initialised when creating the service
-
-```go
-service := micro.NewService(
-	micro.Name("greeter"),
-	// wrap the handler
-	micro.WrapHandler(logWrapper),
-)
-```
-
-### Client
-
-Here's an example of a client wrapper which logs requests made
-
-```go
-type logWrapper struct {
-	client.Client
-}
-
-func (l *logWrapper) Call(ctx context.Context, req client.Request, rsp interface{}, opts ...client.CallOption) error {
-	fmt.Printf("[wrapper] client request to service: %s method: %s\n", req.Service(), req.Method())
-	return l.Client.Call(ctx, req, rsp)
-}
-
-// implements client.Wrapper as logWrapper
-func logWrap(c client.Client) client.Client {
-	return &logWrapper{c}
-}
-```
-
-It can be initialised when creating the service
-
-```go
-service := micro.NewService(
-	micro.Name("greeter"),
-	// wrap the client
-	micro.WrapClient(logWrap),
-)
-```
-
-## Other Languages
-
-Check out [ja-micro](https://github.com/Sixt/ja-micro) to write services in Java
-
-## Sponsors
-
-Open source development of Micro is sponsored by Sixt
-
-<a href="https://micro.mu/blog/2016/04/25/announcing-sixt-sponsorship.html"><img src="https://micro.mu/sixt_logo.png" width=150px height="auto" /></a>

README.zh-cn.md

@@ -0,0 +1,36 @@
+# Go Micro [![License](https://img.shields.io/:license-apache-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![GoDoc](https://godoc.org/github.com/micro/go-micro?status.svg)](https://godoc.org/github.com/micro/go-micro) [![Travis CI](https://api.travis-ci.org/micro/go-micro.svg?branch=master)](https://travis-ci.org/micro/go-micro) [![Go Report Card](https://goreportcard.com/badge/micro/go-micro)](https://goreportcard.com/report/github.com/micro/go-micro)
+
+Go Micro是基于Golang的微服务开发框架。
+
+## 概览
+
+Go Micro提供分布式系统开发的核心库，包含RPC与事件驱动的通信机制。
+
+**micro**的设计哲学是可插拔的架构理念，她提供可快速构建系统的组件，并且可以根据自身的需求剥离默认实现并自行定制。
+
+<img src="https://micro.mu/docs/images/go-micro.svg" />
+
+所有插件可在仓库[github.com/micro/go-plugins](https://github.com/micro/go-plugins)中找到。
+
+可以订阅我们的[Twitter](https://twitter.com/microhq)或者加入[Slack](http://slack.micro.mu/)论坛。
+
+## 特性
+
+Go Micro把分布式系统的各种细节抽象出来。下面是它的主要特性。
+
+- **服务发现（Service Discovery）** - 自动服务注册与名称解析。服务发现是微服务开发中的核心。当服务A要与服务B协作时，它得知道B在哪里。默认的服务发现系统是Consul，而multicast DNS (mdns，组播)机制作为本地解决方案，或者零依赖的P2P网络中的SWIM协议（gossip）。 
+
+- **负载均衡（Load Balancing）** - 在服务发现之上构建了负载均衡机制。当我们得到一个服务的任意多个的实例节点时，我们要一个机制去决定要路由到哪一个节点。我们使用随机处理过的哈希负载均衡机制来保证对服务请求颁发的均匀分布，并且在发生问题时进行重试。
+
+- **消息编码（Message Encoding）** - 支持基于内容类型（content-type）动态编码消息。客户端和服务端会一起使用content-type的格式来对Go进行无缝编/解码。各种各样的消息被编码会发送到不同的客户端，客户端服服务端默认会处理这些消息。content-type默认包含proto-rpc和json-rpc。
+
+- **Request/Response** - RPC通信基于支持双向流的请求/响应方式，我们提供有抽象的同步通信机制。请求发送到服务时，会自动解析、负载均衡、拨号、转成字节流。默认的传输协议是http/1.1，而tls下使用http2协议。
+
+- **异步消息（Async Messaging）** - 发布订阅（PubSub）头等功能内置在异步通信与事件驱动架构中。事件通知在微服务开发中处于核心位置。默认的消息传送使用点到点http/1.1，激活tls时则使用http2。
+
+- **可插拔接口（Pluggable Interfaces）** - Go Micro为每个分布式系统抽象出接口。因此，Go Micro的接口都是可插拔的，允许其在运行时不可知的情况下仍可支持。所以只要实现接口，可以在内部使用任何的技术。更多插件请参考：[github.com/micro/go-plugins](https://github.com/micro/go-plugins)。
+
+## 快速上手
+
+更多关于架构、安装的资料可以查看[文档](https://micro.mu/docs/go-micro_cn.html)。
+

agent/README.md

@@ -0,0 +1,197 @@
+# Agent 
+
+Agent is a library used to create commands, inputs and robot services
+
+## Getting Started
+
+- [Commands](#commands) - Commands are functions executed by the bot based on text based pattern matching.
+- [Inputs](#inputs) - Inputs are plugins for communication e.g Slack, Telegram, IRC, etc.
+- [Services](#services) - Write bots as micro services
+
+## Commands
+
+Commands are functions executed by the bot based on text based pattern matching.
+
+### Write a Command
+
+```go
+import "github.com/micro/go-micro/agent/command"
+
+func Ping() command.Command {
+	usage := "ping"
+	description := "Returns pong"
+
+	return command.NewCommand("ping", usage, desc, func(args ...string) ([]byte, error) {
+		return []byte("pong"), nil
+	})
+}
+```
+
+### Register the command
+
+Add the command to the Commands map with a pattern key that can be matched by golang/regexp.Match
+
+```go
+import "github.com/micro/go-micro/agent/command"
+
+func init() {
+	command.Commands["^ping$"] = Ping()
+}
+```
+
+### Rebuild Micro
+
+Build binary
+```shell
+cd github.com/micro/micro
+
+// For local use
+go build -i -o micro ./main.go
+
+// For docker image
+CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o micro ./main.go
+```
+
+## Inputs
+
+Inputs are plugins for communication e.g Slack, HipChat, XMPP, IRC, SMTP, etc, etc. 
+
+New inputs can be added in the following way.
+
+### Write an Input
+
+Write an input that satisfies the Input interface.
+
+```go
+type Input interface {
+	// Provide cli flags
+	Flags() []cli.Flag
+	// Initialise input using cli context
+	Init(*cli.Context) error
+	// Stream events from the input
+	Stream() (Conn, error)
+	// Start the input
+	Start() error
+	// Stop the input
+	Stop() error
+	// name of the input
+	String() string
+}
+```
+
+### Register the input
+
+Add the input to the Inputs map.
+
+```go
+import "github.com/micro/micro/bot/input"
+
+func init() {
+	input.Inputs["name"] = MyInput
+}
+```
+
+### Rebuild Micro
+
+Build binary
+```shell
+cd github.com/micro/micro
+
+// For local use
+go build -i -o micro ./main.go
+
+// For docker image
+CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' -i -o micro ./main.go
+```
+
+## Services
+
+The micro bot supports the ability to create commands as micro services. 
+
+### How does it work?
+
+The bot watches the service registry for services with it's namespace. The default namespace is `go.micro.bot`. 
+Any service within this namespace will automatically be added to the list of available commands. When a command 
+is executed, the bot will call the service with method `Command.Exec`. It also expects the method `Command.Help` 
+to exist for usage info.
+
+
+The service interface is as follows and can be found at [go-micro/agent/proto](https://github.com/micro/go-micro/agent/blob/master/proto/bot.proto)
+
+```
+syntax = "proto3";
+
+package go.micro.bot;
+
+service Command {
+	rpc Help(HelpRequest) returns (HelpResponse) {};
+	rpc Exec(ExecRequest) returns (ExecResponse) {};
+}
+
+message HelpRequest {
+}
+
+message HelpResponse {
+	string usage = 1;
+	string description = 2;
+}
+
+message ExecRequest {
+	repeated string args = 1;
+}
+
+message ExecResponse {
+	bytes result = 1;
+	string error = 2;
+}
+```
+
+### Example
+
+Here's an example echo command as a microservice
+
+```go
+package main
+
+import (
+	"fmt"
+	"strings"
+
+	"github.com/micro/go-micro"
+	"golang.org/x/net/context"
+
+	proto "github.com/micro/go-micro/agent/proto"
+)
+
+type Command struct{}
+
+// Help returns the command usage
+func (c *Command) Help(ctx context.Context, req *proto.HelpRequest, rsp *proto.HelpResponse) error {
+	// Usage should include the name of the command
+	rsp.Usage = "echo"
+	rsp.Description = "This is an example bot command as a micro service which echos the message"
+	return nil
+}
+
+// Exec executes the command
+func (c *Command) Exec(ctx context.Context, req *proto.ExecRequest, rsp *proto.ExecResponse) error {
+	rsp.Result = []byte(strings.Join(req.Args, " "))
+	// rsp.Error could be set to return an error instead
+	// the function error would only be used for service level issues
+	return nil
+}
+
+func main() {
+	service := micro.NewService(
+		micro.Name("go.micro.bot.echo"),
+	)
+
+	service.Init()
+
+	proto.RegisterCommandHandler(service.Server(), new(Command))
+
+	if err := service.Run(); err != nil {
+		fmt.Println(err)
+	}
+}
+```

agent/agent.go

@@ -0,0 +1,2 @@
+// Package agent provides an interface for building robots
+package agent

agent/command/command.go

@@ -0,0 +1,54 @@
+// Package command is an interface for defining bot commands
+package command
+
+var (
+	// Commmands keyed by golang/regexp patterns
+	// regexp.Match(key, input) is used to match
+	Commands = map[string]Command{}
+)
+
+// Command is the interface for specific named
+// commands executed via plugins or the bot.
+type Command interface {
+	// Executes the command with args passed in
+	Exec(args ...string) ([]byte, error)
+	// Usage of the command
+	Usage() string
+	// Description of the command
+	Description() string
+	// Name of the command
+	String() string
+}
+
+type cmd struct {
+	name        string
+	usage       string
+	description string
+	exec        func(args ...string) ([]byte, error)
+}
+
+func (c *cmd) Description() string {
+	return c.description
+}
+
+func (c *cmd) Exec(args ...string) ([]byte, error) {
+	return c.exec(args...)
+}
+
+func (c *cmd) Usage() string {
+	return c.usage
+}
+
+func (c *cmd) String() string {
+	return c.name
+}
+
+// NewCommand helps quickly create a new command
+func NewCommand(name, usage, description string, exec func(args ...string) ([]byte, error)) Command {
+	return &cmd{
+		name:        name,
+		usage:       usage,
+		description: description,
+		exec:        exec,
+	}
+}

agent/command/command_test.go

@@ -0,0 +1,65 @@
+package command
+
+import (
+	"testing"
+)
+
+func TestCommand(t *testing.T) {
+	c := &cmd{
+		name:        "test",
+		usage:       "test usage",
+		description: "test description",
+		exec: func(args ...string) ([]byte, error) {
+			return []byte("test"), nil
+		},
+	}
+
+	if c.String() != c.name {
+		t.Fatalf("expected name %s got %s", c.name, c.String())
+	}
+
+	if c.Usage() != c.usage {
+		t.Fatalf("expected usage %s got %s", c.usage, c.Usage())
+	}
+
+	if c.Description() != c.description {
+		t.Fatalf("expected description %s got %s", c.description, c.Description())
+	}
+
+	if r, err := c.Exec(); err != nil {
+		t.Fatal(err)
+	} else if string(r) != "test" {
+		t.Fatalf("expected exec result test got %s", string(r))
+	}
+}
+
+func TestNewCommand(t *testing.T) {
+	c := &cmd{
+		name:        "test",
+		usage:       "test usage",
+		description: "test description",
+		exec: func(args ...string) ([]byte, error) {
+			return []byte("test"), nil
+		},
+	}
+
+	nc := NewCommand(c.name, c.usage, c.description, c.exec)
+
+	if nc.String() != c.name {
+		t.Fatalf("expected name %s got %s", c.name, nc.String())
+	}
+
+	if nc.Usage() != c.usage {
+		t.Fatalf("expected usage %s got %s", c.usage, nc.Usage())
+	}
+
+	if nc.Description() != c.description {
+		t.Fatalf("expected description %s got %s", c.description, nc.Description())
+	}
+
+	if r, err := nc.Exec(); err != nil {
+		t.Fatal(err)
+	} else if string(r) != "test" {
+		t.Fatalf("expected exec result test got %s", string(r))
+	}
+}

agent/input/discord/README.md

@@ -0,0 +1,22 @@
+# Discord input for micro-bot
+[Discord](https://discordapp.com) support for micro bot based on [discordgo](github.com/bwmarrin/discordgo).
+
+This was originally written by Aleksandr Tihomirov (@zet4) and can be found at https://github.com/zet4/micro-misc/.
+
+## Options
+### discord_token
+
+You have to supply an application token via `--discord_token`.
+
+Head over to Discord's [developer introduction](https://discordapp.com/developers/docs/intro)
+to learn how to create applications and how the API works.
+
+### discord_prefix
+
+Set a command prefix with `--discord_prefix`. The default prefix is `Micro `.
+You can mention the bot or use the prefix to run a command.
+
+### discord_whitelist
+
+Pass a list of comma-separated user IDs with `--discord_whitelist`. Only allow
+these users to use the bot.

agent/input/discord/conn.go

@@ -0,0 +1,94 @@
+package discord
+
+import (
+	"errors"
+	"strings"
+	"sync"
+
+	"github.com/bwmarrin/discordgo"
+	"github.com/micro/go-micro/agent/input"
+	"github.com/micro/go-micro/util/log"
+)
+
+type discordConn struct {
+	master *discordInput
+	exit   chan struct{}
+	recv   chan *discordgo.Message
+
+	sync.Mutex
+}
+
+func newConn(master *discordInput) *discordConn {
+	conn := &discordConn{
+		master: master,
+		exit:   make(chan struct{}),
+		recv:   make(chan *discordgo.Message),
+	}
+
+	conn.master.session.AddHandler(func(s *discordgo.Session, m *discordgo.MessageCreate) {
+		if m.Author.ID == master.botID {
+			return
+		}
+
+		whitelisted := false
+		for _, ID := range conn.master.whitelist {
+			if m.Author.ID == ID {
+				whitelisted = true
+				break
+			}
+		}
+
+		if len(master.whitelist) > 0 && !whitelisted {
+			return
+		}
+
+		var valid bool
+		m.Message.Content, valid = conn.master.prefixfn(m.Message.Content)
+		if !valid {
+			return
+		}
+
+		conn.recv <- m.Message
+	})
+
+	return conn
+}
+
+func (dc *discordConn) Recv(event *input.Event) error {
+	for {
+		select {
+		case <-dc.exit:
+			return errors.New("connection closed")
+		case msg := <-dc.recv:
+
+			event.From = msg.ChannelID + ":" + msg.Author.ID
+			event.To = dc.master.botID
+			event.Type = input.TextEvent
+			event.Data = []byte(msg.Content)
+			return nil
+		}
+	}
+}
+
+func (dc *discordConn) Send(e *input.Event) error {
+	fields := strings.Split(e.To, ":")
+	_, err := dc.master.session.ChannelMessageSend(fields[0], string(e.Data))
+	if err != nil {
+		log.Log("[bot][loop][send]", err)
+	}
+	return nil
+}
+
+func (dc *discordConn) Close() error {
+	if err := dc.master.session.Close(); err != nil {
+		return err
+	}
+
+	select {
+	case <-dc.exit:
+		return nil
+	default:
+		close(dc.exit)
+	}
+	return nil
+}

agent/input/discord/discord.go

@@ -0,0 +1,153 @@
+package discord
+
+import (
+	"fmt"
+	"sync"
+
+	"errors"
+	"strings"
+
+	"github.com/bwmarrin/discordgo"
+	"github.com/micro/cli"
+	"github.com/micro/go-micro/agent/input"
+)
+
+func init() {
+	input.Inputs["discord"] = newInput()
+}
+
+func newInput() *discordInput {
+	return &discordInput{}
+}
+
+type discordInput struct {
+	token     string
+	whitelist []string
+	prefix    string
+	prefixfn  func(string) (string, bool)
+	botID     string
+
+	session *discordgo.Session
+
+	sync.Mutex
+	running bool
+	exit    chan struct{}
+}
+
+func (d *discordInput) Flags() []cli.Flag {
+	return []cli.Flag{
+		cli.StringFlag{
+			Name:   "discord_token",
+			EnvVar: "MICRO_DISCORD_TOKEN",
+			Usage:  "Discord token (prefix with Bot if it's for bot account)",
+		},
+		cli.StringFlag{
+			Name:   "discord_whitelist",
+			EnvVar: "MICRO_DISCORD_WHITELIST",
+			Usage:  "Discord Whitelist (seperated by ,)",
+		},
+		cli.StringFlag{
+			Name:   "discord_prefix",
+			Usage:  "Discord Prefix",
+			EnvVar: "MICRO_DISCORD_PREFIX",
+			Value:  "Micro ",
+		},
+	}
+}
+
+func (d *discordInput) Init(ctx *cli.Context) error {
+	token := ctx.String("discord_token")
+	whitelist := ctx.String("discord_whitelist")
+	prefix := ctx.String("discord_prefix")
+
+	if len(token) == 0 {
+		return errors.New("require token")
+	}
+
+	d.token = token
+	d.prefix = prefix
+
+	if len(whitelist) > 0 {
+		d.whitelist = strings.Split(whitelist, ",")
+	}
+
+	return nil
+}
+
+func (d *discordInput) Start() error {
+	if len(d.token) == 0 {
+		return errors.New("missing discord configuration")
+	}
+
+	d.Lock()
+	defer d.Unlock()
+
+	if d.running {
+		return nil
+	}
+
+	var err error
+	d.session, err = discordgo.New(d.token)
+	if err != nil {
+		return err
+	}
+
+	u, err := d.session.User("@me")
+	if err != nil {
+		return err
+	}
+
+	d.botID = u.ID
+	d.prefixfn = CheckPrefixFactory(fmt.Sprintf("<@%s> ", d.botID), fmt.Sprintf("<@!%s> ", d.botID), d.prefix)
+
+	d.exit = make(chan struct{})
+	d.running = true
+
+	return nil
+}
+
+func (d *discordInput) Stream() (input.Conn, error) {
+	d.Lock()
+	defer d.Unlock()
+	if !d.running {
+		return nil, errors.New("not running")
+	}
+
+	//Fire-n-forget close just in case...
+	d.session.Close()
+
+	conn := newConn(d)
+	if err := d.session.Open(); err != nil {
+		return nil, err
+	}
+	return conn, nil
+}
+
+func (d *discordInput) Stop() error {
+	d.Lock()
+	defer d.Unlock()
+
+	if !d.running {
+		return nil
+	}
+
+	close(d.exit)
+	d.running = false
+	return nil
+}
+
+func (d *discordInput) String() string {
+	return "discord"
+}
+
+// CheckPrefixFactory Creates a prefix checking function and stuff.
+func CheckPrefixFactory(prefixes ...string) func(string) (string, bool) {
+	return func(content string) (string, bool) {
+		for _, prefix := range prefixes {
+			if strings.HasPrefix(content, prefix) {
+				return strings.TrimPrefix(content, prefix), true
+			}
+		}
+		return "", false
+	}
+}

agent/input/input.go

@@ -0,0 +1,55 @@
+// Package input is an interface for bot inputs
+package input
+
+import (
+	"github.com/micro/cli"
+)
+
+type EventType string
+
+const (
+	TextEvent EventType = "text"
+)
+
+var (
+	// Inputs keyed by name
+	// Example slack or hipchat
+	Inputs = map[string]Input{}
+)
+
+// Event is the unit sent and received
+type Event struct {
+	Type EventType
+	From string
+	To   string
+	Data []byte
+	Meta map[string]interface{}
+}
+
+// Input is an interface for sources which
+// provide a way to communicate with the bot.
+// Slack, HipChat, XMPP, etc.
+type Input interface {
+	// Provide cli flags
+	Flags() []cli.Flag
+	// Initialise input using cli context
+	Init(*cli.Context) error
+	// Stream events from the input
+	Stream() (Conn, error)
+	// Start the input
+	Start() error
+	// Stop the input
+	Stop() error
+	// name of the input
+	String() string
+}
+
+// Conn interface provides a way to
+// send and receive events. Send and
+// Recv both block until succeeding
+// or failing.
+type Conn interface {
+	Close() error
+	Recv(*Event) error
+	Send(*Event) error
+}

agent/input/slack/conn.go

@@ -0,0 +1,160 @@
+package slack
+
+import (
+	"errors"
+	"fmt"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/micro/go-micro/agent/input"
+	"github.com/nlopes/slack"
+)
+
+// Satisfies the input.Conn interface
+type slackConn struct {
+	auth *slack.AuthTestResponse
+	rtm  *slack.RTM
+	exit chan bool
+
+	sync.Mutex
+	names map[string]string
+}
+
+func (s *slackConn) run() {
+	// func retrieves user names and maps to IDs
+	setNames := func() {
+		names := make(map[string]string)
+		users, err := s.rtm.Client.GetUsers()
+		if err != nil {
+			return
+		}
+
+		for _, user := range users {
+			names[user.ID] = user.Name
+		}
+
+		s.Lock()
+		s.names = names
+		s.Unlock()
+	}
+
+	setNames()
+
+	t := time.NewTicker(time.Minute)
+	defer t.Stop()
+
+	for {
+		select {
+		case <-s.exit:
+			return
+		case <-t.C:
+			setNames()
+		}
+	}
+}
+
+func (s *slackConn) getName(id string) string {
+	s.Lock()
+	name := s.names[id]
+	s.Unlock()
+	return name
+}
+
+func (s *slackConn) Close() error {
+	select {
+	case <-s.exit:
+		return nil
+	default:
+		close(s.exit)
+	}
+	return nil
+}
+
+func (s *slackConn) Recv(event *input.Event) error {
+	if event == nil {
+		return errors.New("event cannot be nil")
+	}
+
+	for {
+		select {
+		case <-s.exit:
+			return errors.New("connection closed")
+		case e := <-s.rtm.IncomingEvents:
+			switch ev := e.Data.(type) {
+			case *slack.MessageEvent:
+				// only accept type message
+				if ev.Type != "message" {
+					continue
+				}
+
+				// only accept DMs or messages to me
+				switch {
+				case strings.HasPrefix(ev.Channel, "D"):
+				case strings.HasPrefix(ev.Text, s.auth.User):
+				case strings.HasPrefix(ev.Text, fmt.Sprintf("<@%s>", s.auth.UserID)):
+				default:
+					continue
+				}
+
+				// Strip username from text
+				switch {
+				case strings.HasPrefix(ev.Text, s.auth.User):
+					args := strings.Split(ev.Text, " ")[1:]
+					ev.Text = strings.Join(args, " ")
+					event.To = s.auth.User
+				case strings.HasPrefix(ev.Text, fmt.Sprintf("<@%s>", s.auth.UserID)):
+					args := strings.Split(ev.Text, " ")[1:]
+					ev.Text = strings.Join(args, " ")
+					event.To = s.auth.UserID
+				}
+
+				if event.Meta == nil {
+					event.Meta = make(map[string]interface{})
+				}
+
+				// fill in the blanks
+				event.From = ev.Channel + ":" + ev.User
+				event.Type = input.TextEvent
+				event.Data = []byte(ev.Text)
+				event.Meta["reply"] = ev
+				return nil
+			case *slack.InvalidAuthEvent:
+				return errors.New("invalid credentials")
+			}
+		}
+	}
+}
+
+func (s *slackConn) Send(event *input.Event) error {
+	var channel, message, name string
+
+	if len(event.To) == 0 {
+		return errors.New("require Event.To")
+	}
+
+	parts := strings.Split(event.To, ":")
+
+	if len(parts) == 2 {
+		channel = parts[0]
+		name = s.getName(parts[1])
+		// try using reply meta
+	} else if ev, ok := event.Meta["reply"]; ok {
+		channel = ev.(*slack.MessageEvent).Channel
+		name = s.getName(ev.(*slack.MessageEvent).User)
+	}
+
+	// don't know where to send the message
+	if len(channel) == 0 {
+		return errors.New("could not determine who message is to")
+	}
+
+	if len(name) == 0 || strings.HasPrefix(channel, "D") {
+		message = string(event.Data)
+	} else {
+		message = fmt.Sprintf("@%s: %s", name, string(event.Data))
+	}
+
+	s.rtm.SendMessage(s.rtm.NewOutgoingMessage(message, channel))
+	return nil
+}

agent/input/slack/slack.go

@@ -0,0 +1,147 @@
+package slack
+
+import (
+	"errors"
+	"sync"
+
+	"github.com/micro/cli"
+	"github.com/micro/go-micro/agent/input"
+	"github.com/nlopes/slack"
+)
+
+type slackInput struct {
+	debug bool
+	token string
+
+	sync.Mutex
+	running bool
+	exit    chan bool
+
+	api *slack.Client
+}
+
+func init() {
+	input.Inputs["slack"] = NewInput()
+}
+
+func (p *slackInput) Flags() []cli.Flag {
+	return []cli.Flag{
+		cli.BoolFlag{
+			Name:   "slack_debug",
+			Usage:  "Slack debug output",
+			EnvVar: "MICRO_SLACK_DEBUG",
+		},
+		cli.StringFlag{
+			Name:   "slack_token",
+			Usage:  "Slack token",
+			EnvVar: "MICRO_SLACK_TOKEN",
+		},
+	}
+}
+
+func (p *slackInput) Init(ctx *cli.Context) error {
+	debug := ctx.Bool("slack_debug")
+	token := ctx.String("slack_token")
+
+	if len(token) == 0 {
+		return errors.New("missing slack token")
+	}
+
+	p.debug = debug
+	p.token = token
+
+	return nil
+}
+
+func (p *slackInput) Stream() (input.Conn, error) {
+	p.Lock()
+	defer p.Unlock()
+
+	if !p.running {
+		return nil, errors.New("not running")
+	}
+
+	// test auth
+	auth, err := p.api.AuthTest()
+	if err != nil {
+		return nil, err
+	}
+
+	rtm := p.api.NewRTM()
+	exit := make(chan bool)
+
+	go rtm.ManageConnection()
+
+	go func() {
+		select {
+		case <-p.exit:
+			select {
+			case <-exit:
+				return
+			default:
+				close(exit)
+			}
+		case <-exit:
+		}
+
+		rtm.Disconnect()
+	}()
+
+	conn := &slackConn{
+		auth:  auth,
+		rtm:   rtm,
+		exit:  exit,
+		names: make(map[string]string),
+	}
+
+	go conn.run()
+
+	return conn, nil
+}
+
+func (p *slackInput) Start() error {
+	if len(p.token) == 0 {
+		return errors.New("missing slack token")
+	}
+
+	p.Lock()
+	defer p.Unlock()
+
+	if p.running {
+		return nil
+	}
+
+	api := slack.New(p.token, slack.OptionDebug(p.debug))
+
+	// test auth
+	_, err := api.AuthTest()
+	if err != nil {
+		return err
+	}
+
+	p.api = api
+	p.exit = make(chan bool)
+	p.running = true
+	return nil
+}
+
+func (p *slackInput) Stop() error {
+	p.Lock()
+	defer p.Unlock()
+
+	if !p.running {
+		return nil
+	}
+
+	close(p.exit)
+	p.running = false
+	return nil
+}
+
+func (p *slackInput) String() string {
+	return "slack"
+}
+
+func NewInput() input.Input {
+	return &slackInput{}
+}

agent/input/telegram/README.md

@@ -0,0 +1,18 @@
+# Telegram Messenger input for micro bot
+[Telegram](https://telegram.org) support for micro bot based on [telegram-bot-api](https://github.com/go-telegram-bot-api/telegram-bot-api).
+
+## Options
+### --telegram_token (required)
+
+Sets bot's token for interacting with API.
+
+Head over to Telegram's [API documentation](https://core.telegram.org/bots/api)
+to learn how to create bots and how the API works.
+
+### --telegram_debug
+
+Sets the debug flag to make the bot's output verbose.
+
+### --telegram_whitelist
+
+Sets a list of comma-separated nicknames (without @ symbol in the beginning) for interacting with bot. Only these users can use the bot.

agent/input/telegram/conn.go

@@ -0,0 +1,115 @@
+package telegram
+
+import (
+	"errors"
+	"strings"
+	"sync"
+
+	"github.com/forestgiant/sliceutil"
+	"github.com/micro/go-micro/agent/input"
+	"github.com/micro/go-micro/util/log"
+	"gopkg.in/telegram-bot-api.v4"
+)
+
+type telegramConn struct {
+	input *telegramInput
+
+	recv <-chan tgbotapi.Update
+	exit chan bool
+
+	syncCond *sync.Cond
+	mutex    sync.Mutex
+}
+
+func newConn(input *telegramInput) (*telegramConn, error) {
+	conn := &telegramConn{
+		input: input,
+	}
+
+	conn.syncCond = sync.NewCond(&conn.mutex)
+
+	go conn.run()
+
+	return conn, nil
+}
+
+func (tc *telegramConn) run() {
+	u := tgbotapi.NewUpdate(0)
+	u.Timeout = 60
+	updates, err := tc.input.api.GetUpdatesChan(u)
+	if err != nil {
+		return
+	}
+
+	tc.recv = updates
+	tc.syncCond.Signal()
+
+	for {
+		select {
+		case <-tc.exit:
+			return
+		}
+	}
+}
+
+func (tc *telegramConn) Close() error {
+	return nil
+}
+
+func (tc *telegramConn) Recv(event *input.Event) error {
+	if event == nil {
+		return errors.New("event cannot be nil")
+	}
+
+	for {
+		if tc.recv == nil {
+			tc.mutex.Lock()
+			tc.syncCond.Wait()
+		}
+
+		update := <-tc.recv
+
+		if update.Message == nil || (len(tc.input.whitelist) > 0 && !sliceutil.Contains(tc.input.whitelist, update.Message.From.UserName)) {
+			continue
+		}
+
+		if event.Meta == nil {
+			event.Meta = make(map[string]interface{})
+		}
+
+		event.Type = input.TextEvent
+		event.From = update.Message.From.UserName
+		event.To = tc.input.api.Self.UserName
+		event.Data = []byte(update.Message.Text)
+		event.Meta["chatId"] = update.Message.Chat.ID
+		event.Meta["chatType"] = update.Message.Chat.Type
+		event.Meta["messageId"] = update.Message.MessageID
+
+		return nil
+	}
+}
+
+func (tc *telegramConn) Send(event *input.Event) error {
+	messageText := strings.TrimSpace(string(event.Data))
+
+	chatId := event.Meta["chatId"].(int64)
+	chatType := ChatType(event.Meta["chatType"].(string))
+
+	msgConfig := tgbotapi.NewMessage(chatId, messageText)
+	msgConfig.ParseMode = tgbotapi.ModeHTML
+
+	if sliceutil.Contains([]ChatType{Group, Supergroup}, chatType) {
+		msgConfig.ReplyToMessageID = event.Meta["messageId"].(int)
+	}
+
+	_, err := tc.input.api.Send(msgConfig)
+
+	if err != nil {
+		// probably it could be because of nested HTML tags -- telegram doesn't allow nested tags
+		log.Log("[telegram][Send] error:", err)
+		msgConfig.Text = "This bot couldn't send the response (Internal error)"
+		tc.input.api.Send(msgConfig)
+	}
+
+	return nil
+}

agent/input/telegram/telegram.go

@@ -0,0 +1,101 @@
+package telegram
+
+import (
+	"errors"
+	"strings"
+	"sync"
+
+	"github.com/micro/cli"
+	"github.com/micro/go-micro/agent/input"
+	"gopkg.in/telegram-bot-api.v4"
+)
+
+type telegramInput struct {
+	sync.Mutex
+
+	debug     bool
+	token     string
+	whitelist []string
+
+	api *tgbotapi.BotAPI
+}
+
+type ChatType string
+
+const (
+	Private    ChatType = "private"
+	Group      ChatType = "group"
+	Supergroup ChatType = "supergroup"
+)
+
+func init() {
+	input.Inputs["telegram"] = &telegramInput{}
+}
+
+func (ti *telegramInput) Flags() []cli.Flag {
+	return []cli.Flag{
+		cli.BoolFlag{
+			Name:   "telegram_debug",
+			EnvVar: "MICRO_TELEGRAM_DEBUG",
+			Usage:  "Telegram debug output",
+		},
+		cli.StringFlag{
+			Name:   "telegram_token",
+			EnvVar: "MICRO_TELEGRAM_TOKEN",
+			Usage:  "Telegram token",
+		},
+		cli.StringFlag{
+			Name:   "telegram_whitelist",
+			EnvVar: "MICRO_TELEGRAM_WHITELIST",
+			Usage:  "Telegram bot's users (comma-separated values)",
+		},
+	}
+}
+
+func (ti *telegramInput) Init(ctx *cli.Context) error {
+	ti.debug = ctx.Bool("telegram_debug")
+	ti.token = ctx.String("telegram_token")
+
+	whitelist := ctx.String("telegram_whitelist")
+
+	if whitelist != "" {
+		ti.whitelist = strings.Split(whitelist, ",")
+	}
+
+	if len(ti.token) == 0 {
+		return errors.New("missing telegram token")
+	}
+
+	return nil
+}
+
+func (ti *telegramInput) Stream() (input.Conn, error) {
+	ti.Lock()
+	defer ti.Unlock()
+
+	return newConn(ti)
+}
+
+func (ti *telegramInput) Start() error {
+	ti.Lock()
+	defer ti.Unlock()
+
+	api, err := tgbotapi.NewBotAPI(ti.token)
+	if err != nil {
+		return err
+	}
+
+	ti.api = api
+
+	api.Debug = ti.debug
+
+	return nil
+}
+
+func (ti *telegramInput) Stop() error {
+	return nil
+}
+
+func (p *telegramInput) String() string {
+	return "telegram"
+}

agent/proto/bot.micro.go

@@ -0,0 +1,118 @@
+// Code generated by protoc-gen-micro. DO NOT EDIT.
+// source: github.com/micro/go-bot/proto/bot.proto
+
+/*
+Package go_micro_bot is a generated protocol buffer package.
+
+It is generated from these files:
+	github.com/micro/go-bot/proto/bot.proto
+
+It has these top-level messages:
+	HelpRequest
+	HelpResponse
+	ExecRequest
+	ExecResponse
+*/
+package go_micro_bot
+
+import proto "github.com/golang/protobuf/proto"
+import fmt "fmt"
+import math "math"
+
+import (
+	context "context"
+	client "github.com/micro/go-micro/client"
+	server "github.com/micro/go-micro/server"
+)
+
+// Reference imports to suppress errors if they are not otherwise used.
+var _ = proto.Marshal
+var _ = fmt.Errorf
+var _ = math.Inf
+
+// This is a compile-time assertion to ensure that this generated file
+// is compatible with the proto package it is being compiled against.
+// A compilation error at this line likely means your copy of the
+// proto package needs to be updated.
+const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package
+
+// Reference imports to suppress errors if they are not otherwise used.
+var _ context.Context
+var _ client.Option
+var _ server.Option
+
+// Client API for Command service
+
+type CommandService interface {
+	Help(ctx context.Context, in *HelpRequest, opts ...client.CallOption) (*HelpResponse, error)
+	Exec(ctx context.Context, in *ExecRequest, opts ...client.CallOption) (*ExecResponse, error)
+}
+
+type commandService struct {
+	c    client.Client
+	name string
+}
+
+func NewCommandService(name string, c client.Client) CommandService {
+	if c == nil {
+		c = client.NewClient()
+	}
+	if len(name) == 0 {
+		name = "go.micro.bot"
+	}
+	return &commandService{
+		c:    c,
+		name: name,
+	}
+}
+
+func (c *commandService) Help(ctx context.Context, in *HelpRequest, opts ...client.CallOption) (*HelpResponse, error) {
+	req := c.c.NewRequest(c.name, "Command.Help", in)
+	out := new(HelpResponse)
+	err := c.c.Call(ctx, req, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+func (c *commandService) Exec(ctx context.Context, in *ExecRequest, opts ...client.CallOption) (*ExecResponse, error) {
+	req := c.c.NewRequest(c.name, "Command.Exec", in)
+	out := new(ExecResponse)
+	err := c.c.Call(ctx, req, out, opts...)
+	if err != nil {
+		return nil, err
+	}
+	return out, nil
+}
+
+// Server API for Command service
+
+type CommandHandler interface {
+	Help(context.Context, *HelpRequest, *HelpResponse) error
+	Exec(context.Context, *ExecRequest, *ExecResponse) error
+}
+
+func RegisterCommandHandler(s server.Server, hdlr CommandHandler, opts ...server.HandlerOption) error {
+	type _command interface {
+		Help(ctx context.Context, in *HelpRequest, out *HelpResponse) error
+		Exec(ctx context.Context, in *ExecRequest, out *ExecResponse) error
+	}
+	type Command struct {
+		_command
+	}
+	h := &commandHandler{hdlr}
+	return s.Handle(s.NewHandler(&Command{h}, opts...))
+}
+
+type commandHandler struct {
+	CommandHandler
+}
+
+func (h *commandHandler) Help(ctx context.Context, in *HelpRequest, out *HelpResponse) error {
+	return h.CommandHandler.Help(ctx, in, out)
+}
+
+func (h *commandHandler) Exec(ctx context.Context, in *ExecRequest, out *ExecResponse) error {
+	return h.CommandHandler.Exec(ctx, in, out)
+}

agent/proto/bot.pb.go

@@ -0,0 +1,210 @@
+// Code generated by protoc-gen-go. DO NOT EDIT.
+// source: github.com/micro/go-bot/proto/bot.proto
+
+package go_micro_bot
+
+import proto "github.com/golang/protobuf/proto"
+import fmt "fmt"
+import math "math"
+
+// Reference imports to suppress errors if they are not otherwise used.
+var _ = proto.Marshal
+var _ = fmt.Errorf
+var _ = math.Inf
+
+// This is a compile-time assertion to ensure that this generated file
+// is compatible with the proto package it is being compiled against.
+// A compilation error at this line likely means your copy of the
+// proto package needs to be updated.
+const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package
+
+type HelpRequest struct {
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
+}
+
+func (m *HelpRequest) Reset()         { *m = HelpRequest{} }
+func (m *HelpRequest) String() string { return proto.CompactTextString(m) }
+func (*HelpRequest) ProtoMessage()    {}
+func (*HelpRequest) Descriptor() ([]byte, []int) {
+	return fileDescriptor_bot_654832eab83ed4b5, []int{0}
+}
+func (m *HelpRequest) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_HelpRequest.Unmarshal(m, b)
+}
+func (m *HelpRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_HelpRequest.Marshal(b, m, deterministic)
+}
+func (dst *HelpRequest) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_HelpRequest.Merge(dst, src)
+}
+func (m *HelpRequest) XXX_Size() int {
+	return xxx_messageInfo_HelpRequest.Size(m)
+}
+func (m *HelpRequest) XXX_DiscardUnknown() {
+	xxx_messageInfo_HelpRequest.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_HelpRequest proto.InternalMessageInfo
+
+type HelpResponse struct {
+	Usage                string   `protobuf:"bytes,1,opt,name=usage,proto3" json:"usage,omitempty"`
+	Description          string   `protobuf:"bytes,2,opt,name=description,proto3" json:"description,omitempty"`
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
+}
+
+func (m *HelpResponse) Reset()         { *m = HelpResponse{} }
+func (m *HelpResponse) String() string { return proto.CompactTextString(m) }
+func (*HelpResponse) ProtoMessage()    {}
+func (*HelpResponse) Descriptor() ([]byte, []int) {
+	return fileDescriptor_bot_654832eab83ed4b5, []int{1}
+}
+func (m *HelpResponse) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_HelpResponse.Unmarshal(m, b)
+}
+func (m *HelpResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_HelpResponse.Marshal(b, m, deterministic)
+}
+func (dst *HelpResponse) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_HelpResponse.Merge(dst, src)
+}
+func (m *HelpResponse) XXX_Size() int {
+	return xxx_messageInfo_HelpResponse.Size(m)
+}
+func (m *HelpResponse) XXX_DiscardUnknown() {
+	xxx_messageInfo_HelpResponse.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_HelpResponse proto.InternalMessageInfo
+
+func (m *HelpResponse) GetUsage() string {
+	if m != nil {
+		return m.Usage
+	}
+	return ""
+}
+
+func (m *HelpResponse) GetDescription() string {
+	if m != nil {
+		return m.Description
+	}
+	return ""
+}
+
+type ExecRequest struct {
+	Args                 []string `protobuf:"bytes,1,rep,name=args,proto3" json:"args,omitempty"`
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
+}
+
+func (m *ExecRequest) Reset()         { *m = ExecRequest{} }
+func (m *ExecRequest) String() string { return proto.CompactTextString(m) }
+func (*ExecRequest) ProtoMessage()    {}
+func (*ExecRequest) Descriptor() ([]byte, []int) {
+	return fileDescriptor_bot_654832eab83ed4b5, []int{2}
+}
+func (m *ExecRequest) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_ExecRequest.Unmarshal(m, b)
+}
+func (m *ExecRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_ExecRequest.Marshal(b, m, deterministic)
+}
+func (dst *ExecRequest) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_ExecRequest.Merge(dst, src)
+}
+func (m *ExecRequest) XXX_Size() int {
+	return xxx_messageInfo_ExecRequest.Size(m)
+}
+func (m *ExecRequest) XXX_DiscardUnknown() {
+	xxx_messageInfo_ExecRequest.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_ExecRequest proto.InternalMessageInfo
+
+func (m *ExecRequest) GetArgs() []string {
+	if m != nil {
+		return m.Args
+	}
+	return nil
+}
+
+type ExecResponse struct {
+	Result               []byte   `protobuf:"bytes,1,opt,name=result,proto3" json:"result,omitempty"`
+	Error                string   `protobuf:"bytes,2,opt,name=error,proto3" json:"error,omitempty"`
+	XXX_NoUnkeyedLiteral struct{} `json:"-"`
+	XXX_unrecognized     []byte   `json:"-"`
+	XXX_sizecache        int32    `json:"-"`
+}
+
+func (m *ExecResponse) Reset()         { *m = ExecResponse{} }
+func (m *ExecResponse) String() string { return proto.CompactTextString(m) }
+func (*ExecResponse) ProtoMessage()    {}
+func (*ExecResponse) Descriptor() ([]byte, []int) {
+	return fileDescriptor_bot_654832eab83ed4b5, []int{3}
+}
+func (m *ExecResponse) XXX_Unmarshal(b []byte) error {
+	return xxx_messageInfo_ExecResponse.Unmarshal(m, b)
+}
+func (m *ExecResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
+	return xxx_messageInfo_ExecResponse.Marshal(b, m, deterministic)
+}
+func (dst *ExecResponse) XXX_Merge(src proto.Message) {
+	xxx_messageInfo_ExecResponse.Merge(dst, src)
+}
+func (m *ExecResponse) XXX_Size() int {
+	return xxx_messageInfo_ExecResponse.Size(m)
+}
+func (m *ExecResponse) XXX_DiscardUnknown() {
+	xxx_messageInfo_ExecResponse.DiscardUnknown(m)
+}
+
+var xxx_messageInfo_ExecResponse proto.InternalMessageInfo
+
+func (m *ExecResponse) GetResult() []byte {
+	if m != nil {
+		return m.Result
+	}
+	return nil
+}
+
+func (m *ExecResponse) GetError() string {
+	if m != nil {
+		return m.Error
+	}
+	return ""
+}
+
+func init() {
+	proto.RegisterType((*HelpRequest)(nil), "go.micro.bot.HelpRequest")
+	proto.RegisterType((*HelpResponse)(nil), "go.micro.bot.HelpResponse")
+	proto.RegisterType((*ExecRequest)(nil), "go.micro.bot.ExecRequest")
+	proto.RegisterType((*ExecResponse)(nil), "go.micro.bot.ExecResponse")
+}
+
+func init() {
+	proto.RegisterFile("github.com/micro/go-bot/proto/bot.proto", fileDescriptor_bot_654832eab83ed4b5)
+}
+
+var fileDescriptor_bot_654832eab83ed4b5 = []byte{
+	// 246 bytes of a gzipped FileDescriptorProto
+	0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x6c, 0x90, 0x41, 0x4b, 0xc4, 0x30,
+	0x10, 0x85, 0xb7, 0xba, 0xae, 0xec, 0xb4, 0x5e, 0x82, 0x48, 0xdd, 0x53, 0xcd, 0xc5, 0xbd, 0x98,
+	0x82, 0x5e, 0x05, 0x0f, 0xa2, 0x78, 0xee, 0x3f, 0x68, 0xba, 0x43, 0x2c, 0x6c, 0x3b, 0x35, 0x99,
+	0x82, 0xff, 0xc1, 0x3f, 0x2d, 0x4d, 0x72, 0x08, 0xcb, 0xde, 0xe6, 0x65, 0x86, 0xf7, 0xbe, 0x17,
+	0x78, 0x34, 0x3d, 0x7f, 0xcf, 0x5a, 0x75, 0x34, 0xd4, 0x43, 0xdf, 0x59, 0xaa, 0x0d, 0x3d, 0x69,
+	0xe2, 0x7a, 0xb2, 0xc4, 0x54, 0x6b, 0x62, 0xe5, 0x27, 0x51, 0x18, 0x52, 0xfe, 0x40, 0x69, 0x62,
+	0x79, 0x03, 0xf9, 0x17, 0x1e, 0xa7, 0x06, 0x7f, 0x66, 0x74, 0x2c, 0x3f, 0xa1, 0x08, 0xd2, 0x4d,
+	0x34, 0x3a, 0x14, 0xb7, 0x70, 0x35, 0xbb, 0xd6, 0x60, 0x99, 0x55, 0xd9, 0x7e, 0xdb, 0x04, 0x21,
+	0x2a, 0xc8, 0x0f, 0xe8, 0x3a, 0xdb, 0x4f, 0xdc, 0xd3, 0x58, 0x5e, 0xf8, 0x5d, 0xfa, 0x24, 0x1f,
+	0x20, 0xff, 0xf8, 0xc5, 0x2e, 0xda, 0x0a, 0x01, 0xeb, 0xd6, 0x1a, 0x57, 0x66, 0xd5, 0xe5, 0x7e,
+	0xdb, 0xf8, 0x59, 0xbe, 0x42, 0x11, 0x4e, 0x62, 0xd4, 0x1d, 0x6c, 0x2c, 0xba, 0xf9, 0xc8, 0x3e,
+	0xab, 0x68, 0xa2, 0x5a, 0x10, 0xd0, 0x5a, 0xb2, 0x31, 0x26, 0x88, 0xe7, 0xbf, 0x0c, 0xae, 0xdf,
+	0x69, 0x18, 0xda, 0xf1, 0x20, 0xde, 0x60, 0xbd, 0x40, 0x8b, 0x7b, 0x95, 0x56, 0x53, 0x49, 0xaf,
+	0xdd, 0xee, 0xdc, 0x2a, 0x04, 0xcb, 0xd5, 0x62, 0xb0, 0xa0, 0x9c, 0x1a, 0x24, 0x0d, 0x4e, 0x0d,
+	0x52, 0x72, 0xb9, 0xd2, 0x1b, 0xff, 0xb5, 0x2f, 0xff, 0x01, 0x00, 0x00, 0xff, 0xff, 0xcb, 0x77,
+	0xdf, 0x28, 0x85, 0x01, 0x00, 0x00,
+}

agent/proto/bot.proto

@@ -0,0 +1,25 @@
+syntax = "proto3";
+
+package go.micro.bot;
+
+service Command {
+	rpc Help(HelpRequest) returns (HelpResponse) {};
+	rpc Exec(ExecRequest) returns (ExecResponse) {};
+}
+
+message HelpRequest {
+}
+
+message HelpResponse {
+	string usage = 1;
+	string description = 2;
+}
+
+message ExecRequest {
+	repeated string args = 1;
+}
+
+message ExecResponse {
+	bytes result = 1;
+	string error = 2;
+}

broker/codec/codec.go

@@ -1,10 +0,0 @@
-package codec
-
-// Codec is used for encoding where the broker doesn't natively support
-// headers in the message type. In this case the entire message is
-// encoded as the payload
-type Codec interface {
-	Marshal(interface{}) ([]byte, error)
-	Unmarshal([]byte, interface{}) error
-	String() string
-}

broker/codec/json/json.go

@@ -1,25 +0,0 @@
-package json
-
-import (
-	"encoding/json"
-
-	"github.com/micro/go-micro/broker/codec"
-)
-
-type jsonCodec struct{}
-
-func (j jsonCodec) Marshal(v interface{}) ([]byte, error) {
-	return json.Marshal(v)
-}
-
-func (j jsonCodec) Unmarshal(d []byte, v interface{}) error {
-	return json.Unmarshal(d, v)
-}
-
-func (j jsonCodec) String() string {
-	return "json"
-}
-
-func NewCodec() codec.Codec {
-	return jsonCodec{}
-}

broker/codec/noop/noop.go

@@ -1,35 +0,0 @@
-package noop
-
-import (
-	"errors"
-
-	"github.com/micro/go-micro/broker"
-	"github.com/micro/go-micro/broker/codec"
-)
-
-type noopCodec struct{}
-
-func (n noopCodec) Marshal(v interface{}) ([]byte, error) {
-	msg, ok := v.(*broker.Message)
-	if !ok {
-		return nil, errors.New("invalid message")
-	}
-	return msg.Body, nil
-}
-
-func (n noopCodec) Unmarshal(d []byte, v interface{}) error {
-	msg, ok := v.(*broker.Message)
-	if !ok {
-		return errors.New("invalid message")
-	}
-	msg.Body = d
-	return nil
-}
-
-func (n noopCodec) String() string {
-	return "noop"
-}
-
-func NewCodec() codec.Codec {
-	return noopCodec{}
-}

broker/http/http.go

@@ -1,9 +1,11 @@
+// Package http provides a http based message broker
 package http
 
 import (
 	"github.com/micro/go-micro/broker"
 )
 
+// NewBroker returns a new http broker
 func NewBroker(opts ...broker.Option) broker.Broker {
 	return broker.NewBroker(opts...)
 }

broker/http/options.go

@@ -0,0 +1,23 @@
+package http
+
+import (
+	"context"
+	"net/http"
+
+	"github.com/micro/go-micro/broker"
+)
+
+// Handle registers the handler for the given pattern.
+func Handle(pattern string, handler http.Handler) broker.Option {
+	return func(o *broker.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		handlers, ok := o.Context.Value("http_handlers").(map[string]http.Handler)
+		if !ok {
+			handlers = make(map[string]http.Handler)
+		}
+		handlers[pattern] = handler
+		o.Context = context.WithValue(o.Context, "http_handlers", handlers)
+	}
+}

broker/http_broker.go

@@ -2,6 +2,7 @@ package broker
 
 import (
 	"bytes"
+	"context"
 	"crypto/tls"
 	"errors"
 	"fmt"
@@ -17,17 +18,15 @@ import (
 	"sync"
 	"time"
 
-	"github.com/micro/go-log"
-	"github.com/micro/go-micro/broker/codec/json"
+	"github.com/google/uuid"
+	"github.com/micro/go-micro/codec/json"
 	merr "github.com/micro/go-micro/errors"
 	"github.com/micro/go-micro/registry"
-	"github.com/micro/go-rcache"
-	maddr "github.com/micro/misc/lib/addr"
-	mnet "github.com/micro/misc/lib/net"
-	mls "github.com/micro/misc/lib/tls"
-	"github.com/pborman/uuid"
-
-	"golang.org/x/net/context"
+	"github.com/micro/go-micro/registry/cache"
+	maddr "github.com/micro/go-micro/util/addr"
+	mnet "github.com/micro/go-micro/util/net"
+	mls "github.com/micro/go-micro/util/tls"
+	"golang.org/x/net/http2"
 )
 
 // HTTP Broker is a point to point async broker
@@ -45,6 +44,10 @@ type httpBroker struct {
 	subscribers map[string][]*httpSubscriber
 	running     bool
 	exit        chan chan error
+
+	// offline message inbox
+	mtx   sync.RWMutex
+	inbox map[string][][]byte
 }
 
 type httpSubscriber struct {
@@ -79,6 +82,10 @@ func newTransport(config *tls.Config) *http.Transport {
 		}
 	}
 
+	dialTLS := func(network string, addr string) (net.Conn, error) {
+		return tls.Dial(network, addr, config)
+	}
+
 	t := &http.Transport{
 		Proxy: http.ProxyFromEnvironment,
 		Dial: (&net.Dialer{
@@ -86,17 +93,21 @@ func newTransport(config *tls.Config) *http.Transport {
 			KeepAlive: 30 * time.Second,
 		}).Dial,
 		TLSHandshakeTimeout: 10 * time.Second,
-		TLSClientConfig:     config,
+		DialTLS:             dialTLS,
 	}
 	runtime.SetFinalizer(&t, func(tr **http.Transport) {
 		(*tr).CloseIdleConnections()
 	})
+
+	// setup http2
+	http2.ConfigureTransport(t)
+
 	return t
 }
 
 func newHttpBroker(opts ...Option) Broker {
 	options := Options{
-		Codec:   json.NewCodec(),
+		Codec:   json.Marshaler{},
 		Context: context.TODO(),
 	}
 
@@ -117,7 +128,7 @@ func newHttpBroker(opts ...Option) Broker {
 	}
 
 	h := &httpBroker{
-		id:          "broker-" + uuid.NewUUID().String(),
+		id:          "broker-" + uuid.New().String(),
 		address:     addr,
 		opts:        options,
 		r:           reg,
@@ -125,9 +136,22 @@ func newHttpBroker(opts ...Option) Broker {
 		subscribers: make(map[string][]*httpSubscriber),
 		exit:        make(chan chan error),
 		mux:         http.NewServeMux(),
+		inbox:       make(map[string][][]byte),
 	}
 
+	// specify the message handler
 	h.mux.Handle(DefaultSubPath, h)
+
+	// get optional handlers
+	if h.opts.Context != nil {
+		handlers, ok := h.opts.Context.Value("http_handlers").(map[string]http.Handler)
+		if ok {
+			for pattern, handler := range handlers {
+				h.mux.Handle(pattern, handler)
+			}
+		}
+	}
+
 	return h
 }
 
@@ -155,6 +179,49 @@ func (h *httpSubscriber) Unsubscribe() error {
 	return h.hb.unsubscribe(h)
 }
 
+func (h *httpBroker) saveMessage(topic string, msg []byte) {
+	h.mtx.Lock()
+	defer h.mtx.Unlock()
+
+	// get messages
+	c := h.inbox[topic]
+
+	// save message
+	c = append(c, msg)
+
+	// max length 64
+	if len(c) > 64 {
+		c = c[:64]
+	}
+
+	// save inbox
+	h.inbox[topic] = c
+}
+
+func (h *httpBroker) getMessage(topic string, num int) [][]byte {
+	h.mtx.Lock()
+	defer h.mtx.Unlock()
+
+	// get messages
+	c, ok := h.inbox[topic]
+	if !ok {
+		return nil
+	}
+
+	// more message than requests
+	if len(c) >= num {
+		msg := c[:num]
+		h.inbox[topic] = c[num:]
+		return msg
+	}
+
+	// reset inbox
+	h.inbox[topic] = nil
+
+	// return all messages
+	return c
+}
+
 func (h *httpBroker) subscribe(s *httpSubscriber) error {
 	h.Lock()
 	defer h.Unlock()
@@ -177,7 +244,7 @@ func (h *httpBroker) unsubscribe(s *httpSubscriber) error {
 	for _, sub := range h.subscribers[s.topic] {
 		// deregister and skip forward
 		if sub.id == s.id {
-			h.r.Deregister(sub.svc)
+			_ = h.r.Deregister(sub.svc)
 			continue
 		}
 		// keep subscriber
@@ -201,7 +268,7 @@ func (h *httpBroker) run(l net.Listener) {
 			h.RLock()
 			for _, subs := range h.subscribers {
 				for _, sub := range subs {
-					h.r.Register(sub.svc, registry.RegisterTTL(registerTTL))
+					_ = h.r.Register(sub.svc, registry.RegisterTTL(registerTTL))
 				}
 			}
 			h.RUnlock()
@@ -211,7 +278,7 @@ func (h *httpBroker) run(l net.Listener) {
 			h.RLock()
 			for _, subs := range h.subscribers {
 				for _, sub := range subs {
-					h.r.Deregister(sub.svc)
+					_ = h.r.Deregister(sub.svc)
 				}
 			}
 			h.RUnlock()
@@ -277,12 +344,15 @@ func (h *httpBroker) Address() string {
 }
 
 func (h *httpBroker) Connect() error {
-	h.Lock()
-	defer h.Unlock()
-
+	h.RLock()
 	if h.running {
+		h.RUnlock()
 		return nil
 	}
+	h.RUnlock()
+
+	h.Lock()
+	defer h.Unlock()
 
 	var l net.Listener
 	var err error
@@ -326,19 +396,24 @@ func (h *httpBroker) Connect() error {
 		return err
 	}
 
-	log.Logf("Broker Listening on %s", l.Addr().String())
+	addr := h.address
 	h.address = l.Addr().String()
 
 	go http.Serve(l, h.mux)
-	go h.run(l)
+	go func() {
+		h.run(l)
+		h.Lock()
+		h.address = addr
+		h.Unlock()
+	}()
 
 	// get registry
 	reg, ok := h.opts.Context.Value(registryKey).(registry.Registry)
 	if !ok {
 		reg = registry.DefaultRegistry
 	}
-	// set rcache
-	h.r = rcache.New(reg)
+	// set cache
+	h.r = cache.New(reg)
 
 	// set running
 	h.running = true
@@ -346,15 +421,19 @@ func (h *httpBroker) Connect() error {
 }
 
 func (h *httpBroker) Disconnect() error {
+
+	h.RLock()
+	if !h.running {
+		h.RUnlock()
+		return nil
+	}
+	h.RUnlock()
+
 	h.Lock()
 	defer h.Unlock()
 
-	if !h.running {
-		return nil
-	}
-
-	// stop rcache
-	rc, ok := h.r.(rcache.Cache)
+	// stop cache
+	rc, ok := h.r.(cache.Cache)
 	if ok {
 		rc.Stop()
 	}
@@ -370,19 +449,26 @@ func (h *httpBroker) Disconnect() error {
 }
 
 func (h *httpBroker) Init(opts ...Option) error {
-	h.Lock()
-	defer h.Unlock()
-
+	h.RLock()
 	if h.running {
+		h.RUnlock()
 		return errors.New("cannot init while connected")
 	}
+	h.RUnlock()
+
+	h.Lock()
+	defer h.Unlock()
 
 	for _, o := range opts {
 		o(&h.opts)
 	}
 
+	if len(h.opts.Addrs) > 0 && len(h.opts.Addrs[0]) > 0 {
+		h.address = h.opts.Addrs[0]
+	}
+
 	if len(h.id) == 0 {
-		h.id = "broker-" + uuid.NewUUID().String()
+		h.id = "broker-" + uuid.New().String()
 	}
 
 	// get registry
@@ -391,13 +477,20 @@ func (h *httpBroker) Init(opts ...Option) error {
 		reg = registry.DefaultRegistry
 	}
 
-	// get rcache
-	if rc, ok := h.r.(rcache.Cache); ok {
+	// get cache
+	if rc, ok := h.r.(cache.Cache); ok {
 		rc.Stop()
 	}
 
 	// set registry
-	h.r = rcache.New(reg)
+	h.r = cache.New(reg)
+
+	// reconfigure tls config
+	if c := h.opts.TLSConfig; c != nil {
+		h.c = &http.Client{
+			Transport: newTransport(c),
+		}
+	}
 
 	return nil
 }
@@ -407,14 +500,7 @@ func (h *httpBroker) Options() Options {
 }
 
 func (h *httpBroker) Publish(topic string, msg *Message, opts ...PublishOption) error {
-	h.RLock()
-	s, err := h.r.GetService("topic:" + topic)
-	if err != nil {
-		h.RUnlock()
-		return err
-	}
-	h.RUnlock()
-
+	// create the message first
 	m := &Message{
 		Header: make(map[string]string),
 		Body:   msg.Body,
@@ -426,12 +512,26 @@ func (h *httpBroker) Publish(topic string, msg *Message, opts ...PublishOption)
 
 	m.Header[":topic"] = topic
 
+	// encode the message
 	b, err := h.opts.Codec.Marshal(m)
 	if err != nil {
 		return err
 	}
 
-	pub := func(node *registry.Node, b []byte) {
+	// save the message
+	h.saveMessage(topic, b)
+
+	// now attempt to get the service
+	h.RLock()
+	s, err := h.r.GetService("topic:" + topic)
+	if err != nil {
+		h.RUnlock()
+		// ignore error
+		return nil
+	}
+	h.RUnlock()
+
+	pub := func(node *registry.Node, t string, b []byte) error {
 		scheme := "http"
 
 		// check if secure is added in metadata
@@ -444,39 +544,76 @@ func (h *httpBroker) Publish(topic string, msg *Message, opts ...PublishOption)
 
 		uri := fmt.Sprintf("%s://%s:%d%s?%s", scheme, node.Address, node.Port, DefaultSubPath, vals.Encode())
 		r, err := h.c.Post(uri, "application/json", bytes.NewReader(b))
-		if err == nil {
-			io.Copy(ioutil.Discard, r.Body)
-			r.Body.Close()
+		if err != nil {
+			return err
 		}
+
+		// discard response body
+		io.Copy(ioutil.Discard, r.Body)
+		r.Body.Close()
+		return nil
 	}
 
-	for _, service := range s {
-		// only process if we have nodes
-		if len(service.Nodes) == 0 {
-			continue
-		}
-
-		switch service.Version {
-		// broadcast version means broadcast to all nodes
-		case broadcastVersion:
-			for _, node := range service.Nodes {
-				// publish async
-				go pub(node, b)
+	srv := func(s []*registry.Service, b []byte) {
+		for _, service := range s {
+			// only process if we have nodes
+			if len(service.Nodes) == 0 {
+				continue
 			}
-		default:
-			// select node to publish to
-			node := service.Nodes[rand.Int()%len(service.Nodes)]
 
-			// publish async
-			go pub(node, b)
+			switch service.Version {
+			// broadcast version means broadcast to all nodes
+			case broadcastVersion:
+				var success bool
+
+				// publish to all nodes
+				for _, node := range service.Nodes {
+					// publish async
+					if err := pub(node, topic, b); err == nil {
+						success = true
+					}
+				}
+
+				// save if it failed to publish at least once
+				if !success {
+					h.saveMessage(topic, b)
+				}
+			default:
+				// select node to publish to
+				node := service.Nodes[rand.Int()%len(service.Nodes)]
+
+				// publish async to one node
+				if err := pub(node, topic, b); err != nil {
+					// if failed save it
+					h.saveMessage(topic, b)
+				}
+			}
 		}
 	}
 
+	// do the rest async
+	go func() {
+		// get a third of the backlog
+		messages := h.getMessage(topic, 8)
+		delay := (len(messages) > 1)
+
+		// publish all the messages
+		for _, msg := range messages {
+			// serialize here
+			srv(s, msg)
+
+			// sending a backlog of messages
+			if delay {
+				time.Sleep(time.Millisecond * 100)
+			}
+		}
+	}()
+
 	return nil
 }
 
 func (h *httpBroker) Subscribe(topic string, handler Handler, opts ...SubscribeOption) (Subscriber, error) {
-	options := newSubscribeOptions(opts...)
+	options := NewSubscribeOptions(opts...)
 
 	// parse address for host, port
 	parts := strings.Split(h.Address(), ":")
@@ -489,7 +626,7 @@ func (h *httpBroker) Subscribe(topic string, handler Handler, opts ...SubscribeO
 	}
 
 	// create unique id
-	id := h.id + "." + uuid.NewUUID().String()
+	id := h.id + "." + uuid.New().String()
 
 	var secure bool
 

broker/http_broker_test.go

@@ -5,15 +5,28 @@ import (
 	"testing"
 	"time"
 
-	"github.com/micro/go-micro/registry/mock"
-	"github.com/pborman/uuid"
+	glog "github.com/go-log/log"
+	"github.com/google/uuid"
+	"github.com/micro/go-micro/registry/memory"
+	"github.com/micro/go-micro/util/log"
 )
 
+func newTestRegistry() *memory.Registry {
+	r := memory.NewRegistry()
+	m := r.(*memory.Registry)
+	m.Setup()
+	return m
+}
+
 func sub(be *testing.B, c int) {
+	// set no op logger
+	log.SetLogger(glog.DefaultLogger)
+
 	be.StopTimer()
-	m := mock.NewRegistry()
+	m := newTestRegistry()
+
 	b := NewBroker(Registry(m))
-	topic := uuid.NewUUID().String()
+	topic := uuid.New().String()
 
 	if err := b.Init(); err != nil {
 		be.Fatalf("Unexpected init error: %v", err)
@@ -69,10 +82,13 @@ func sub(be *testing.B, c int) {
 }
 
 func pub(be *testing.B, c int) {
+	// set no op logger
+	log.SetLogger(glog.DefaultLogger)
+
 	be.StopTimer()
-	m := mock.NewRegistry()
+	m := newTestRegistry()
 	b := NewBroker(Registry(m))
-	topic := uuid.NewUUID().String()
+	topic := uuid.New().String()
 
 	if err := b.Init(); err != nil {
 		be.Fatalf("Unexpected init error: %v", err)
@@ -139,7 +155,7 @@ func pub(be *testing.B, c int) {
 }
 
 func TestBroker(t *testing.T) {
-	m := mock.NewRegistry()
+	m := newTestRegistry()
 	b := NewBroker(Registry(m))
 
 	if err := b.Init(); err != nil {
@@ -186,7 +202,7 @@ func TestBroker(t *testing.T) {
 }
 
 func TestConcurrentSubBroker(t *testing.T) {
-	m := mock.NewRegistry()
+	m := newTestRegistry()
 	b := NewBroker(Registry(m))
 
 	if err := b.Init(); err != nil {
@@ -243,7 +259,7 @@ func TestConcurrentSubBroker(t *testing.T) {
 }
 
 func TestConcurrentPubBroker(t *testing.T) {
-	m := mock.NewRegistry()
+	m := newTestRegistry()
 	b := NewBroker(Registry(m))
 
 	if err := b.Init(); err != nil {

broker/memory/memory.go

@@ -1,27 +1,28 @@
-package mock
+// Package memory provides a memory broker
+package memory
 
 import (
 	"errors"
 	"sync"
 
+	"github.com/google/uuid"
 	"github.com/micro/go-micro/broker"
-	"github.com/pborman/uuid"
 )
 
-type mockBroker struct {
+type memoryBroker struct {
 	opts broker.Options
 
 	sync.RWMutex
 	connected   bool
-	Subscribers map[string][]*mockSubscriber
+	Subscribers map[string][]*memorySubscriber
 }
 
-type mockPublication struct {
+type memoryPublication struct {
 	topic   string
 	message *broker.Message
 }
 
-type mockSubscriber struct {
+type memorySubscriber struct {
 	id      string
 	topic   string
 	exit    chan bool
@@ -29,15 +30,15 @@ type mockSubscriber struct {
 	opts    broker.SubscribeOptions
 }
 
-func (m *mockBroker) Options() broker.Options {
+func (m *memoryBroker) Options() broker.Options {
 	return m.opts
 }
 
-func (m *mockBroker) Address() string {
+func (m *memoryBroker) Address() string {
 	return ""
 }
 
-func (m *mockBroker) Connect() error {
+func (m *memoryBroker) Connect() error {
 	m.Lock()
 	defer m.Unlock()
 
@@ -50,7 +51,7 @@ func (m *mockBroker) Connect() error {
 	return nil
 }
 
-func (m *mockBroker) Disconnect() error {
+func (m *memoryBroker) Disconnect() error {
 	m.Lock()
 	defer m.Unlock()
 
@@ -63,14 +64,14 @@ func (m *mockBroker) Disconnect() error {
 	return nil
 }
 
-func (m *mockBroker) Init(opts ...broker.Option) error {
+func (m *memoryBroker) Init(opts ...broker.Option) error {
 	for _, o := range opts {
 		o(&m.opts)
 	}
 	return nil
 }
 
-func (m *mockBroker) Publish(topic string, message *broker.Message, opts ...broker.PublishOption) error {
+func (m *memoryBroker) Publish(topic string, message *broker.Message, opts ...broker.PublishOption) error {
 	m.Lock()
 	defer m.Unlock()
 
@@ -83,7 +84,7 @@ func (m *mockBroker) Publish(topic string, message *broker.Message, opts ...brok
 		return nil
 	}
 
-	p := &mockPublication{
+	p := &memoryPublication{
 		topic:   topic,
 		message: message,
 	}
@@ -97,7 +98,7 @@ func (m *mockBroker) Publish(topic string, message *broker.Message, opts ...brok
 	return nil
 }
 
-func (m *mockBroker) Subscribe(topic string, handler broker.Handler, opts ...broker.SubscribeOption) (broker.Subscriber, error) {
+func (m *memoryBroker) Subscribe(topic string, handler broker.Handler, opts ...broker.SubscribeOption) (broker.Subscriber, error) {
 	m.Lock()
 	defer m.Unlock()
 
@@ -110,9 +111,9 @@ func (m *mockBroker) Subscribe(topic string, handler broker.Handler, opts ...bro
 		o(&options)
 	}
 
-	sub := &mockSubscriber{
+	sub := &memorySubscriber{
 		exit:    make(chan bool, 1),
-		id:      uuid.NewUUID().String(),
+		id:      uuid.New().String(),
 		topic:   topic,
 		handler: handler,
 		opts:    options,
@@ -123,7 +124,7 @@ func (m *mockBroker) Subscribe(topic string, handler broker.Handler, opts ...bro
 	go func() {
 		<-sub.exit
 		m.Lock()
-		var newSubscribers []*mockSubscriber
+		var newSubscribers []*memorySubscriber
 		for _, sb := range m.Subscribers[topic] {
 			if sb.id == sub.id {
 				continue
@@ -137,31 +138,31 @@ func (m *mockBroker) Subscribe(topic string, handler broker.Handler, opts ...bro
 	return sub, nil
 }
 
-func (m *mockBroker) String() string {
-	return "mock"
+func (m *memoryBroker) String() string {
+	return "memory"
 }
 
-func (m *mockPublication) Topic() string {
+func (m *memoryPublication) Topic() string {
 	return m.topic
 }
 
-func (m *mockPublication) Message() *broker.Message {
+func (m *memoryPublication) Message() *broker.Message {
 	return m.message
 }
 
-func (m *mockPublication) Ack() error {
+func (m *memoryPublication) Ack() error {
 	return nil
 }
 
-func (m *mockSubscriber) Options() broker.SubscribeOptions {
+func (m *memorySubscriber) Options() broker.SubscribeOptions {
 	return m.opts
 }
 
-func (m *mockSubscriber) Topic() string {
+func (m *memorySubscriber) Topic() string {
 	return m.topic
 }
 
-func (m *mockSubscriber) Unsubscribe() error {
+func (m *memorySubscriber) Unsubscribe() error {
 	m.exit <- true
 	return nil
 }
@@ -172,8 +173,8 @@ func NewBroker(opts ...broker.Option) broker.Broker {
 		o(&options)
 	}
 
-	return &mockBroker{
+	return &memoryBroker{
 		opts:        options,
-		Subscribers: make(map[string][]*mockSubscriber),
+		Subscribers: make(map[string][]*memorySubscriber),
 	}
 }

broker/memory/memory_test.go

@@ -1,4 +1,4 @@
-package mock
+package memory
 
 import (
 	"fmt"
@@ -7,7 +7,7 @@ import (
 	"github.com/micro/go-micro/broker"
 )
 
-func TestBroker(t *testing.T) {
+func TestMemoryBroker(t *testing.T) {
 	b := NewBroker()
 
 	if err := b.Connect(); err != nil {

broker/nats/context.go

@@ -0,0 +1,37 @@
+package nats
+
+import (
+	"context"
+
+	"github.com/micro/go-micro/broker"
+)
+
+// setSubscribeOption returns a function to setup a context with given value
+func setSubscribeOption(k, v interface{}) broker.SubscribeOption {
+	return func(o *broker.SubscribeOptions) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, k, v)
+	}
+}
+
+// setBrokerOption returns a function to setup a context with given value
+func setBrokerOption(k, v interface{}) broker.Option {
+	return func(o *broker.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, k, v)
+	}
+}
+
+// setPublishOption returns a function to setup a context with given value
+func setPublishOption(k, v interface{}) broker.PublishOption {
+	return func(o *broker.PublishOptions) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, k, v)
+	}
+}

broker/nats/nats.go

@@ -0,0 +1,245 @@
+// Package nats provides a NATS broker
+package nats
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"sync"
+
+	"github.com/micro/go-micro/broker"
+	"github.com/micro/go-micro/codec/json"
+	nats "github.com/nats-io/nats.go"
+)
+
+type natsBroker struct {
+	sync.RWMutex
+	addrs []string
+	conn  *nats.Conn
+	opts  broker.Options
+	nopts nats.Options
+	drain bool
+}
+
+type subscriber struct {
+	s     *nats.Subscription
+	opts  broker.SubscribeOptions
+	drain bool
+}
+
+type publication struct {
+	t string
+	m *broker.Message
+}
+
+func (p *publication) Topic() string {
+	return p.t
+}
+
+func (p *publication) Message() *broker.Message {
+	return p.m
+}
+
+func (p *publication) Ack() error {
+	// nats does not support acking
+	return nil
+}
+
+func (s *subscriber) Options() broker.SubscribeOptions {
+	return s.opts
+}
+
+func (s *subscriber) Topic() string {
+	return s.s.Subject
+}
+
+func (s *subscriber) Unsubscribe() error {
+	if s.drain {
+		return s.s.Drain()
+	}
+	return s.s.Unsubscribe()
+}
+
+func (n *natsBroker) Address() string {
+	if n.conn != nil && n.conn.IsConnected() {
+		return n.conn.ConnectedUrl()
+	}
+	if len(n.addrs) > 0 {
+		return n.addrs[0]
+	}
+
+	return ""
+}
+
+func setAddrs(addrs []string) []string {
+	var cAddrs []string
+	for _, addr := range addrs {
+		if len(addr) == 0 {
+			continue
+		}
+		if !strings.HasPrefix(addr, "nats://") {
+			addr = "nats://" + addr
+		}
+		cAddrs = append(cAddrs, addr)
+	}
+	if len(cAddrs) == 0 {
+		cAddrs = []string{nats.DefaultURL}
+	}
+	return cAddrs
+}
+
+func (n *natsBroker) Connect() error {
+	n.Lock()
+	defer n.Unlock()
+
+	status := nats.CLOSED
+	if n.conn != nil {
+		status = n.conn.Status()
+	}
+
+	switch status {
+	case nats.CONNECTED, nats.RECONNECTING, nats.CONNECTING:
+		return nil
+	default: // DISCONNECTED or CLOSED or DRAINING
+		opts := n.nopts
+		opts.Servers = n.addrs
+		opts.Secure = n.opts.Secure
+		opts.TLSConfig = n.opts.TLSConfig
+
+		// secure might not be set
+		if n.opts.TLSConfig != nil {
+			opts.Secure = true
+		}
+
+		c, err := opts.Connect()
+		if err != nil {
+			return err
+		}
+		n.conn = c
+		return nil
+	}
+}
+
+func (n *natsBroker) Disconnect() error {
+	n.RLock()
+	if n.drain {
+		n.conn.Drain()
+	} else {
+		n.conn.Close()
+	}
+	n.RUnlock()
+	return nil
+}
+
+func (n *natsBroker) Init(opts ...broker.Option) error {
+	for _, o := range opts {
+		o(&n.opts)
+	}
+	n.addrs = setAddrs(n.opts.Addrs)
+	return nil
+}
+
+func (n *natsBroker) Options() broker.Options {
+	return n.opts
+}
+
+func (n *natsBroker) Publish(topic string, msg *broker.Message, opts ...broker.PublishOption) error {
+	b, err := n.opts.Codec.Marshal(msg)
+	if err != nil {
+		return err
+	}
+	n.RLock()
+	defer n.RUnlock()
+	return n.conn.Publish(topic, b)
+}
+
+func (n *natsBroker) Subscribe(topic string, handler broker.Handler, opts ...broker.SubscribeOption) (broker.Subscriber, error) {
+	if n.conn == nil {
+		return nil, errors.New("not connected")
+	}
+
+	opt := broker.SubscribeOptions{
+		AutoAck: true,
+		Context: context.Background(),
+	}
+
+	for _, o := range opts {
+		o(&opt)
+	}
+
+	var drain bool
+	if _, ok := opt.Context.Value(drainSubscriptionKey{}).(bool); ok {
+		drain = true
+	}
+
+	fn := func(msg *nats.Msg) {
+		var m broker.Message
+		if err := n.opts.Codec.Unmarshal(msg.Data, &m); err != nil {
+			return
+		}
+		handler(&publication{m: &m, t: msg.Subject})
+	}
+
+	var sub *nats.Subscription
+	var err error
+
+	n.RLock()
+	if len(opt.Queue) > 0 {
+		sub, err = n.conn.QueueSubscribe(topic, opt.Queue, fn)
+	} else {
+		sub, err = n.conn.Subscribe(topic, fn)
+	}
+	n.RUnlock()
+	if err != nil {
+		return nil, err
+	}
+	return &subscriber{s: sub, opts: opt, drain: drain}, nil
+}
+
+func (n *natsBroker) String() string {
+	return "nats"
+}
+
+func NewBroker(opts ...broker.Option) broker.Broker {
+	options := broker.Options{
+		// Default codec
+		Codec:   json.Marshaler{},
+		Context: context.Background(),
+	}
+
+	for _, o := range opts {
+		o(&options)
+	}
+
+	natsOpts := nats.GetDefaultOptions()
+	if n, ok := options.Context.Value(optionsKey{}).(nats.Options); ok {
+		natsOpts = n
+	}
+
+	var drain bool
+	if _, ok := options.Context.Value(drainSubscriptionKey{}).(bool); ok {
+		drain = true
+	}
+
+	// broker.Options have higher priority than nats.Options
+	// only if Addrs, Secure or TLSConfig were not set through a broker.Option
+	// we read them from nats.Option
+	if len(options.Addrs) == 0 {
+		options.Addrs = natsOpts.Servers
+	}
+
+	if !options.Secure {
+		options.Secure = natsOpts.Secure
+	}
+
+	if options.TLSConfig == nil {
+		options.TLSConfig = natsOpts.TLSConfig
+	}
+
+	return &natsBroker{
+		opts:  options,
+		nopts: natsOpts,
+		addrs: setAddrs(options.Addrs),
+		drain: drain,
+	}
+}

broker/nats/nats_test.go

@@ -0,0 +1,99 @@
+package nats
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/micro/go-micro/broker"
+	nats "github.com/nats-io/nats.go"
+)
+
+var addrTestCases = []struct {
+	name        string
+	description string
+	addrs       map[string]string // expected address : set address
+}{
+	{
+		"brokerOpts",
+		"set broker addresses through a broker.Option in constructor",
+		map[string]string{
+			"nats://192.168.10.1:5222": "192.168.10.1:5222",
+			"nats://10.20.10.0:4222":   "10.20.10.0:4222"},
+	},
+	{
+		"brokerInit",
+		"set broker addresses through a broker.Option in broker.Init()",
+		map[string]string{
+			"nats://192.168.10.1:5222": "192.168.10.1:5222",
+			"nats://10.20.10.0:4222":   "10.20.10.0:4222"},
+	},
+	{
+		"natsOpts",
+		"set broker addresses through the nats.Option in constructor",
+		map[string]string{
+			"nats://192.168.10.1:5222": "192.168.10.1:5222",
+			"nats://10.20.10.0:4222":   "10.20.10.0:4222"},
+	},
+	{
+		"default",
+		"check if default Address is set correctly",
+		map[string]string{
+			"nats://localhost:4222": "",
+		},
+	},
+}
+
+// TestInitAddrs tests issue #100. Ensures that if the addrs is set by an option in init it will be used.
+func TestInitAddrs(t *testing.T) {
+
+	for _, tc := range addrTestCases {
+		t.Run(fmt.Sprintf("%s: %s", tc.name, tc.description), func(t *testing.T) {
+
+			var br broker.Broker
+			var addrs []string
+
+			for _, addr := range tc.addrs {
+				addrs = append(addrs, addr)
+			}
+
+			switch tc.name {
+			case "brokerOpts":
+				// we know that there are just two addrs in the dict
+				br = NewBroker(broker.Addrs(addrs[0], addrs[1]))
+				br.Init()
+			case "brokerInit":
+				br = NewBroker()
+				// we know that there are just two addrs in the dict
+				br.Init(broker.Addrs(addrs[0], addrs[1]))
+			case "natsOpts":
+				nopts := nats.GetDefaultOptions()
+				nopts.Servers = addrs
+				br = NewBroker(Options(nopts))
+				br.Init()
+			case "default":
+				br = NewBroker()
+				br.Init()
+			}
+
+			natsBroker, ok := br.(*natsBroker)
+			if !ok {
+				t.Fatal("Expected broker to be of types *natsBroker")
+			}
+			// check if the same amount of addrs we set has actually been set, default
+			// have only 1 address nats://127.0.0.1:4222 (current nats code) or
+			// nats://localhost:4222 (older code version)
+			if len(natsBroker.addrs) != len(tc.addrs) && tc.name != "default" {
+				t.Errorf("Expected Addr count = %d, Actual Addr count = %d",
+					len(natsBroker.addrs), len(tc.addrs))
+			}
+
+			for _, addr := range natsBroker.addrs {
+				_, ok := tc.addrs[addr]
+				if !ok {
+					t.Errorf("Expected '%s' has not been set", addr)
+				}
+			}
+		})
+
+	}
+}

broker/nats/options.go

@@ -0,0 +1,25 @@
+package nats
+
+import (
+	"github.com/micro/go-micro/broker"
+	nats "github.com/nats-io/nats.go"
+)
+
+type optionsKey struct{}
+type drainConnectionKey struct{}
+type drainSubscriptionKey struct{}
+
+// Options accepts nats.Options
+func Options(opts nats.Options) broker.Option {
+	return setBrokerOption(optionsKey{}, opts)
+}
+
+// DrainConnection will drain subscription on close
+func DrainConnection() broker.Option {
+	return setBrokerOption(drainConnectionKey{}, true)
+}
+
+// DrainSubscription will drain pending messages when unsubscribe
+func DrainSubscription() broker.SubscribeOption {
+	return setSubscribeOption(drainSubscriptionKey{}, true)
+}

broker/options.go

@@ -1,17 +1,17 @@
 package broker
 
 import (
+	"context"
 	"crypto/tls"
 
-	"github.com/micro/go-micro/broker/codec"
+	"github.com/micro/go-micro/codec"
 	"github.com/micro/go-micro/registry"
-	"golang.org/x/net/context"
 )
 
 type Options struct {
 	Addrs     []string
 	Secure    bool
-	Codec     codec.Codec
+	Codec     codec.Marshaler
 	TLSConfig *tls.Config
 	// Other options for implementations of the interface
 	// can be stored in a context
@@ -44,13 +44,11 @@ type PublishOption func(*PublishOptions)
 
 type SubscribeOption func(*SubscribeOptions)
 
-type contextKeyT string
-
 var (
-	registryKey = contextKeyT("github.com/micro/go-micro/registry")
+	registryKey = "github.com/micro/go-micro/registry"
 )
 
-func newSubscribeOptions(opts ...SubscribeOption) SubscribeOptions {
+func NewSubscribeOptions(opts ...SubscribeOption) SubscribeOptions {
 	opt := SubscribeOptions{
 		AutoAck: true,
 	}
@@ -71,7 +69,7 @@ func Addrs(addrs ...string) Option {
 
 // Codec sets the codec used for encoding/decoding used where
 // a broker does not support headers
-func Codec(c codec.Codec) Option {
+func Codec(c codec.Marshaler) Option {
 	return func(o *Options) {
 		o.Codec = c
 	}
@@ -111,3 +109,10 @@ func TLSConfig(t *tls.Config) Option {
 		o.TLSConfig = t
 	}
 }
+
+// SubscribeContext set context
+func SubscribeContext(ctx context.Context) SubscribeOption {
+	return func(o *SubscribeOptions) {
+		o.Context = ctx
+	}
+}

client/backoff.go

@@ -1,10 +1,9 @@
 package client
 
 import (
+	"context"
 	"math"
 	"time"
-
-	"golang.org/x/net/context"
 )
 
 type BackoffFunc func(ctx context.Context, req Request, attempts int) (time.Duration, error)

client/backoff_test.go

@@ -1,18 +1,19 @@
 package client
 
 import (
+	"context"
 	"math"
 	"testing"
 	"time"
-
-	"golang.org/x/net/context"
 )
 
 func TestBackoff(t *testing.T) {
 	delta := time.Duration(0)
 
+	c := NewClient()
+
 	for i := 0; i < 5; i++ {
-		d, err := exponentialBackoff(context.TODO(), NewJsonRequest("test", "test", nil), i)
+		d, err := exponentialBackoff(context.TODO(), c.NewRequest("test", "test", nil), i)
 		if err != nil {
 			t.Fatal(err)
 		}

client/client.go

@@ -2,9 +2,10 @@
 package client
 
 import (
+	"context"
 	"time"
 
-	"golang.org/x/net/context"
+	"github.com/micro/go-micro/codec"
 )
 
 // Client is the interface used to make requests to services.
@@ -13,42 +14,69 @@ import (
 type Client interface {
 	Init(...Option) error
 	Options() Options
-	NewPublication(topic string, msg interface{}) Publication
-	NewRequest(service, method string, req interface{}, reqOpts ...RequestOption) Request
-	NewProtoRequest(service, method string, req interface{}, reqOpts ...RequestOption) Request
-	NewJsonRequest(service, method string, req interface{}, reqOpts ...RequestOption) Request
+	NewMessage(topic string, msg interface{}, opts ...MessageOption) Message
+	NewRequest(service, endpoint string, req interface{}, reqOpts ...RequestOption) Request
 	Call(ctx context.Context, req Request, rsp interface{}, opts ...CallOption) error
-	CallRemote(ctx context.Context, addr string, req Request, rsp interface{}, opts ...CallOption) error
-	Stream(ctx context.Context, req Request, opts ...CallOption) (Streamer, error)
-	StreamRemote(ctx context.Context, addr string, req Request, opts ...CallOption) (Streamer, error)
-	Publish(ctx context.Context, p Publication, opts ...PublishOption) error
+	Stream(ctx context.Context, req Request, opts ...CallOption) (Stream, error)
+	Publish(ctx context.Context, msg Message, opts ...PublishOption) error
 	String() string
 }
 
-// Publication is the interface for a message published asynchronously
-type Publication interface {
+// Router manages request routing
+type Router interface {
+	SendRequest(context.Context, Request) (Response, error)
+}
+
+// Message is the interface for publishing asynchronously
+type Message interface {
 	Topic() string
-	Message() interface{}
+	Payload() interface{}
 	ContentType() string
 }
 
 // Request is the interface for a synchronous request used by Call or Stream
 type Request interface {
+	// The service to call
 	Service() string
+	// The action to take
 	Method() string
+	// The endpoint to invoke
+	Endpoint() string
+	// The content type
 	ContentType() string
-	Request() interface{}
+	// The unencoded request body
+	Body() interface{}
+	// Write to the encoded request writer. This is nil before a call is made
+	Codec() codec.Writer
 	// indicates whether the request will be a streaming one rather than unary
 	Stream() bool
 }
 
-// Streamer is the inteface for a bidirectional synchronous stream
-type Streamer interface {
+// Response is the response received from a service
+type Response interface {
+	// Read the response
+	Codec() codec.Reader
+	// read the header
+	Header() map[string]string
+	// Read the undecoded response
+	Read() ([]byte, error)
+}
+
+// Stream is the inteface for a bidirectional synchronous stream
+type Stream interface {
+	// Context for the stream
 	Context() context.Context
+	// The request made
 	Request() Request
+	// The response read
+	Response() Response
+	// Send will encode and send a request
 	Send(interface{}) error
+	// Recv will decode and read a response
 	Recv(interface{}) error
+	// Error returns the stream error
 	Error() error
+	// Close closes the stream
 	Close() error
 }
 
@@ -61,6 +89,9 @@ type CallOption func(*CallOptions)
 // PublishOption used by Publish
 type PublishOption func(*PublishOptions)
 
+// MessageOption used by NewMessage
+type MessageOption func(*MessageOptions)
+
 // RequestOption used by NewRequest
 type RequestOption func(*RequestOptions)
 
@@ -70,13 +101,13 @@ var (
 	// DefaultBackoff is the default backoff function for retries
 	DefaultBackoff = exponentialBackoff
 	// DefaultRetry is the default check-for-retry function for retries
-	DefaultRetry = alwaysRetry
+	DefaultRetry = RetryOnError
 	// DefaultRetries is the default number of times a request is tried
 	DefaultRetries = 1
 	// DefaultRequestTimeout is the default request timeout
 	DefaultRequestTimeout = time.Second * 5
 	// DefaultPoolSize sets the connection pool size
-	DefaultPoolSize = 0
+	DefaultPoolSize = 100
 	// DefaultPoolTTL sets the connection pool ttl
 	DefaultPoolTTL = time.Minute
 )
@@ -86,26 +117,15 @@ func Call(ctx context.Context, request Request, response interface{}, opts ...Ca
 	return DefaultClient.Call(ctx, request, response, opts...)
 }
 
-// Makes a synchronous call to the specified address using the default client
-func CallRemote(ctx context.Context, address string, request Request, response interface{}, opts ...CallOption) error {
-	return DefaultClient.CallRemote(ctx, address, request, response, opts...)
-}
-
-// Creates a streaming connection with a service and returns responses on the
-// channel passed in. It's up to the user to close the streamer.
-func Stream(ctx context.Context, request Request, opts ...CallOption) (Streamer, error) {
-	return DefaultClient.Stream(ctx, request, opts...)
-}
-
-// Creates a streaming connection to the address specified.
-func StreamRemote(ctx context.Context, address string, request Request, opts ...CallOption) (Streamer, error) {
-	return DefaultClient.StreamRemote(ctx, address, request, opts...)
-}
-
 // Publishes a publication using the default client. Using the underlying broker
 // set within the options.
-func Publish(ctx context.Context, p Publication) error {
-	return DefaultClient.Publish(ctx, p)
+func Publish(ctx context.Context, msg Message, opts ...PublishOption) error {
+	return DefaultClient.Publish(ctx, msg, opts...)
+}
+
+// Creates a new message using the default client
+func NewMessage(topic string, payload interface{}, opts ...MessageOption) Message {
+	return DefaultClient.NewMessage(topic, payload, opts...)
 }
 
 // Creates a new client with the options passed in
@@ -113,25 +133,16 @@ func NewClient(opt ...Option) Client {
 	return newRpcClient(opt...)
 }
 
-// Creates a new publication using the default client
-func NewPublication(topic string, message interface{}) Publication {
-	return DefaultClient.NewPublication(topic, message)
-}
-
 // Creates a new request using the default client. Content Type will
 // be set to the default within options and use the appropriate codec
-func NewRequest(service, method string, request interface{}, reqOpts ...RequestOption) Request {
-	return DefaultClient.NewRequest(service, method, request, reqOpts...)
+func NewRequest(service, endpoint string, request interface{}, reqOpts ...RequestOption) Request {
+	return DefaultClient.NewRequest(service, endpoint, request, reqOpts...)
 }
 
-// Creates a new protobuf request using the default client
-func NewProtoRequest(service, method string, request interface{}, reqOpts ...RequestOption) Request {
-	return DefaultClient.NewProtoRequest(service, method, request, reqOpts...)
-}
-
-// Creates a new json request using the default client
-func NewJsonRequest(service, method string, request interface{}, reqOpts ...RequestOption) Request {
-	return DefaultClient.NewJsonRequest(service, method, request, reqOpts...)
+// Creates a streaming connection with a service and returns responses on the
+// channel passed in. It's up to the user to close the streamer.
+func NewStream(ctx context.Context, request Request, opts ...CallOption) (Stream, error) {
+	return DefaultClient.Stream(ctx, request, opts...)
 }
 
 func String() string {

client/client_wrapper.go

@@ -1,51 +0,0 @@
-package client
-
-/*
-Wrapper is a type of middleware for the go-micro client. It allows
-the client to be "wrapped" so that requests and responses can be intercepted
-to perform extra requirements such as auth, tracing, monitoring, logging, etc.
-
-Example usage:
-
-	import (
-		"log"
-		"github.com/micro/go-micro/client"
-
-	)
-
-	type LogWrapper struct {
-		client.Client
-	}
-
-	func (l *LogWrapper) Call(ctx context.Context, req client.Request, rsp interface{}, opts ...client.CallOption) error {
-		log.Println("Making request to service " + req.Service() + " method " + req.Method())
-		return w.Client.Call(ctx, req, rsp)
-	}
-
-	func Wrapper(c client.Client) client.Client {
-		return &LogWrapper{c}
-	}
-
-	func main() {
-		c := client.NewClient(client.Wrap(Wrapper))
-
-	}
-
-
-*/
-
-import (
-	"golang.org/x/net/context"
-)
-
-// CallFunc represents the individual call func
-type CallFunc func(ctx context.Context, address string, req Request, rsp interface{}, opts CallOptions) error
-
-// CallWrapper is a low level wrapper for the CallFunc
-type CallWrapper func(CallFunc) CallFunc
-
-// Wrapper wraps a client and returns a client
-type Wrapper func(Client) Client
-
-// StreamWrapper wraps a Stream and returns the equivalent
-type StreamWrapper func(Streamer) Streamer

client/context.go

@@ -1,7 +1,7 @@
 package client
 
 import (
-	"golang.org/x/net/context"
+	"context"
 )
 
 type clientKey struct{}

client/mock/context.go

@@ -1,7 +1,7 @@
 package mock
 
 import (
-	"golang.org/x/net/context"
+	"context"
 )
 
 type responseKey struct{}

client/mock/mock.go

@@ -1,14 +1,14 @@
+// Package mock provides a mock client for testing
 package mock
 
 import (
+	"context"
 	"fmt"
 	"reflect"
 	"sync"
 
 	"github.com/micro/go-micro/client"
 	"github.com/micro/go-micro/errors"
-
-	"golang.org/x/net/context"
 )
 
 var (
@@ -16,7 +16,7 @@ var (
 )
 
 type MockResponse struct {
-	Method   string
+	Endpoint string
 	Response interface{}
 	Error    error
 }
@@ -50,20 +50,12 @@ func (m *MockClient) Options() client.Options {
 	return m.Opts
 }
 
-func (m *MockClient) NewPublication(topic string, msg interface{}) client.Publication {
-	return m.Client.NewPublication(topic, msg)
+func (m *MockClient) NewMessage(topic string, msg interface{}, opts ...client.MessageOption) client.Message {
+	return m.Client.NewMessage(topic, msg, opts...)
 }
 
-func (m *MockClient) NewRequest(service, method string, req interface{}, reqOpts ...client.RequestOption) client.Request {
-	return m.Client.NewRequest(service, method, req, reqOpts...)
-}
-
-func (m *MockClient) NewProtoRequest(service, method string, req interface{}, reqOpts ...client.RequestOption) client.Request {
-	return m.Client.NewProtoRequest(service, method, req, reqOpts...)
-}
-
-func (m *MockClient) NewJsonRequest(service, method string, req interface{}, reqOpts ...client.RequestOption) client.Request {
-	return m.Client.NewJsonRequest(service, method, req, reqOpts...)
+func (m *MockClient) NewRequest(service, endpoint string, req interface{}, reqOpts ...client.RequestOption) client.Request {
+	return m.Client.NewRequest(service, endpoint, req, reqOpts...)
 }
 
 func (m *MockClient) Call(ctx context.Context, req client.Request, rsp interface{}, opts ...client.CallOption) error {
@@ -76,7 +68,7 @@ func (m *MockClient) Call(ctx context.Context, req client.Request, rsp interface
 	}
 
 	for _, r := range response {
-		if r.Method != req.Method() {
+		if r.Endpoint != req.Endpoint() {
 			continue
 		}
 
@@ -89,48 +81,24 @@ func (m *MockClient) Call(ctx context.Context, req client.Request, rsp interface
 		if t := reflect.TypeOf(rsp); t.Kind() == reflect.Ptr {
 			v = reflect.Indirect(v)
 		}
+		response := r.Response
+		if t := reflect.TypeOf(r.Response); t.Kind() == reflect.Func {
+			var request []reflect.Value
+			if t.NumIn() == 1 {
+				request = append(request, reflect.ValueOf(req.Body()))
+			}
+			response = reflect.ValueOf(r.Response).Call(request)[0].Interface()
+		}
 
-		v.Set(reflect.ValueOf(r.Response))
+		v.Set(reflect.ValueOf(response))
 
 		return nil
 	}
 
-	return fmt.Errorf("rpc: can't find service %s", req.Method())
+	return fmt.Errorf("rpc: can't find service %s", req.Endpoint())
 }
 
-func (m *MockClient) CallRemote(ctx context.Context, addr string, req client.Request, rsp interface{}, opts ...client.CallOption) error {
-	m.Lock()
-	defer m.Unlock()
-
-	response, ok := m.Response[req.Service()]
-	if !ok {
-		return errors.NotFound("go.micro.client.mock", "service not found")
-	}
-
-	for _, r := range response {
-		if r.Method != req.Method() {
-			continue
-		}
-
-		if r.Error != nil {
-			return r.Error
-		}
-
-		v := reflect.ValueOf(rsp)
-
-		if t := reflect.TypeOf(rsp); t.Kind() == reflect.Ptr {
-			v = reflect.Indirect(v)
-		}
-
-		v.Set(reflect.ValueOf(r.Response))
-
-		return nil
-	}
-
-	return fmt.Errorf("rpc: can't find service %s", req.Method())
-}
-
-func (m *MockClient) Stream(ctx context.Context, req client.Request, opts ...client.CallOption) (client.Streamer, error) {
+func (m *MockClient) Stream(ctx context.Context, req client.Request, opts ...client.CallOption) (client.Stream, error) {
 	m.Lock()
 	defer m.Unlock()
 
@@ -138,15 +106,7 @@ func (m *MockClient) Stream(ctx context.Context, req client.Request, opts ...cli
 	return nil, nil
 }
 
-func (m *MockClient) StreamRemote(ctx context.Context, addr string, req client.Request, opts ...client.CallOption) (client.Streamer, error) {
-	m.Lock()
-	defer m.Unlock()
-
-	// TODO: mock stream
-	return nil, nil
-}
-
-func (m *MockClient) Publish(ctx context.Context, p client.Publication, opts ...client.PublishOption) error {
+func (m *MockClient) Publish(ctx context.Context, p client.Message, opts ...client.PublishOption) error {
 	return nil
 }
 

client/mock/mock_test.go

@@ -1,11 +1,10 @@
 package mock
 
 import (
+	"context"
 	"testing"
 
 	"github.com/micro/go-micro/errors"
-
-	"golang.org/x/net/context"
 )
 
 func TestClient(t *testing.T) {
@@ -14,15 +13,23 @@ func TestClient(t *testing.T) {
 	}
 
 	response := []MockResponse{
-		{Method: "Foo.Bar", Response: map[string]interface{}{"foo": "bar"}},
-		{Method: "Foo.Struct", Response: &TestResponse{Param: "aparam"}},
-		{Method: "Foo.Fail", Error: errors.InternalServerError("go.mock", "failed")},
+		{Endpoint: "Foo.Bar", Response: map[string]interface{}{"foo": "bar"}},
+		{Endpoint: "Foo.Struct", Response: &TestResponse{Param: "aparam"}},
+		{Endpoint: "Foo.Fail", Error: errors.InternalServerError("go.mock", "failed")},
+		{Endpoint: "Foo.Func", Response: func() string { return "string" }},
+		{Endpoint: "Foo.FuncStruct", Response: func() *TestResponse { return &TestResponse{Param: "aparam"} }},
+		{Endpoint: "Foo.FuncWithReqBody", Response: func(req interface{}) string {
+			if req.(map[string]string)["foo"] == "bar" {
+				return "string"
+			}
+			return "wrong"
+		}},
 	}
 
 	c := NewClient(Response("go.mock", response))
 
 	for _, r := range response {
-		req := c.NewJsonRequest("go.mock", r.Method, map[string]interface{}{"foo": "bar"})
+		req := c.NewRequest("go.mock", r.Endpoint, map[string]string{"foo": "bar"})
 		var rsp interface{}
 
 		err := c.Call(context.TODO(), req, &rsp)
@@ -32,6 +39,20 @@ func TestClient(t *testing.T) {
 		}
 
 		t.Log(rsp)
+		if r.Endpoint == "Foo.FuncWithReqBody" {
+			req := c.NewRequest("go.mock", r.Endpoint, map[string]string{"foo": "wrong"})
+			var rsp interface{}
+
+			err := c.Call(context.TODO(), req, &rsp)
+
+			if err != r.Error {
+				t.Fatalf("Expecter error %v got %v", r.Error, err)
+			}
+			if rsp.(string) != "wrong" {
+				t.Fatalf("Expecter response 'wrong' got %v", rsp)
+			}
+			t.Log(rsp)
+		}
 	}
 
 }

client/options.go

@@ -1,6 +1,7 @@
 package client
 
 import (
+	"context"
 	"time"
 
 	"github.com/micro/go-micro/broker"
@@ -8,8 +9,6 @@ import (
 	"github.com/micro/go-micro/registry"
 	"github.com/micro/go-micro/selector"
 	"github.com/micro/go-micro/transport"
-
-	"golang.org/x/net/context"
 )
 
 type Options struct {
@@ -23,6 +22,9 @@ type Options struct {
 	Selector  selector.Selector
 	Transport transport.Transport
 
+	// Router sets the router
+	Router Router
+
 	// Connection Pool
 	PoolSize int
 	PoolTTL  time.Duration
@@ -41,6 +43,8 @@ type Options struct {
 type CallOptions struct {
 	SelectOptions []selector.SelectOption
 
+	// Address of remote host
+	Address string
 	// Backoff func
 	Backoff BackoffFunc
 	// Check if retriable func
@@ -61,13 +65,20 @@ type CallOptions struct {
 }
 
 type PublishOptions struct {
+	// Exchange is the routing exchange for the message
+	Exchange string
 	// Other options for implementations of the interface
 	// can be stored in a context
 	Context context.Context
 }
 
+type MessageOptions struct {
+	ContentType string
+}
+
 type RequestOptions struct {
-	Stream bool
+	ContentType string
+	Stream      bool
 
 	// Other options for implementations of the interface
 	// can be stored in a context
@@ -93,7 +104,7 @@ func newOptions(options ...Option) Options {
 	}
 
 	if len(opts.ContentType) == 0 {
-		opts.ContentType = defaultContentType
+		opts.ContentType = DefaultContentType
 	}
 
 	if opts.Broker == nil {
@@ -227,6 +238,20 @@ func DialTimeout(d time.Duration) Option {
 
 // Call Options
 
+// WithExchange sets the exchange to route a message through
+func WithExchange(e string) PublishOption {
+	return func(o *PublishOptions) {
+		o.Exchange = e
+	}
+}
+
+// WithAddress sets the remote address to use rather than using service discovery
+func WithAddress(a string) CallOption {
+	return func(o *CallOptions) {
+		o.Address = a
+	}
+}
+
 func WithSelectOption(so ...selector.SelectOption) CallOption {
 	return func(o *CallOptions) {
 		o.SelectOptions = append(o.SelectOptions, so...)
@@ -282,8 +307,21 @@ func WithDialTimeout(d time.Duration) CallOption {
 
 // Request Options
 
+func WithContentType(ct string) RequestOption {
+	return func(o *RequestOptions) {
+		o.ContentType = ct
+	}
+}
+
 func StreamingRequest() RequestOption {
 	return func(o *RequestOptions) {
 		o.Stream = true
 	}
 }
+
+// WithRouter sets the client router
+func WithRouter(r Router) Option {
+	return func(o *Options) {
+		o.Router = r
+	}
+}

client/retry.go

@@ -1,11 +1,35 @@
 package client
 
-import "golang.org/x/net/context"
+import (
+	"context"
+
+	"github.com/micro/go-micro/errors"
+)
 
 // note that returning either false or a non-nil error will result in the call not being retried
 type RetryFunc func(ctx context.Context, req Request, retryCount int, err error) (bool, error)
 
-// always retry on error
-func alwaysRetry(ctx context.Context, req Request, retryCount int, err error) (bool, error) {
+// RetryAlways always retry on error
+func RetryAlways(ctx context.Context, req Request, retryCount int, err error) (bool, error) {
 	return true, nil
 }
+
+// RetryOnError retries a request on a 500 or timeout error
+func RetryOnError(ctx context.Context, req Request, retryCount int, err error) (bool, error) {
+	if err == nil {
+		return false, nil
+	}
+
+	e := errors.Parse(err.Error())
+	if e == nil {
+		return false, nil
+	}
+
+	switch e.Code {
+	// retry on timeout or internal server error
+	case 408, 500:
+		return true, nil
+	default:
+		return false, nil
+	}
+}

client/rpc/rpc.go

@@ -1,9 +1,11 @@
+// Package rpc provides an rpc client
 package rpc
 
 import (
 	"github.com/micro/go-micro/client"
 )
 
+// NewClient returns a new micro client interface
 func NewClient(opts ...client.Option) client.Client {
 	return client.NewClient(opts...)
 }

client/rpc_client.go

@@ -2,24 +2,30 @@ package client
 
 import (
 	"bytes"
+	"context"
 	"fmt"
+	"net"
+	"os"
+	"strconv"
 	"sync"
+	"sync/atomic"
 	"time"
 
+	"github.com/google/uuid"
 	"github.com/micro/go-micro/broker"
 	"github.com/micro/go-micro/codec"
 	"github.com/micro/go-micro/errors"
 	"github.com/micro/go-micro/metadata"
+	"github.com/micro/go-micro/registry"
 	"github.com/micro/go-micro/selector"
 	"github.com/micro/go-micro/transport"
-
-	"golang.org/x/net/context"
 )
 
 type rpcClient struct {
 	once sync.Once
 	opts Options
 	pool *pool
+	seq  uint64
 }
 
 func newRpcClient(opt ...Option) Client {
@@ -29,6 +35,7 @@ func newRpcClient(opt ...Option) Client {
 		once: sync.Once{},
 		opts: opts,
 		pool: newPool(opts.PoolSize, opts.PoolTTL),
+		seq:  0,
 	}
 
 	c := Client(rc)
@@ -45,13 +52,18 @@ func (r *rpcClient) newCodec(contentType string) (codec.NewCodec, error) {
 	if c, ok := r.opts.Codecs[contentType]; ok {
 		return c, nil
 	}
-	if cf, ok := defaultCodecs[contentType]; ok {
+	if cf, ok := DefaultCodecs[contentType]; ok {
 		return cf, nil
 	}
 	return nil, fmt.Errorf("Unsupported Content-Type: %s", contentType)
 }
 
-func (r *rpcClient) call(ctx context.Context, address string, req Request, resp interface{}, opts CallOptions) error {
+func (r *rpcClient) call(ctx context.Context, node *registry.Node, req Request, resp interface{}, opts CallOptions) error {
+	address := node.Address
+	if node.Port > 0 {
+		address = fmt.Sprintf("%s:%d", address, node.Port)
+	}
+
 	msg := &transport.Message{
 		Header: make(map[string]string),
 	}
@@ -70,9 +82,16 @@ func (r *rpcClient) call(ctx context.Context, address string, req Request, resp
 	// set the accept header
 	msg.Header["Accept"] = req.ContentType()
 
-	cf, err := r.newCodec(req.ContentType())
-	if err != nil {
-		return errors.InternalServerError("go.micro.client", err.Error())
+	// setup old protocol
+	cf := setupProtocol(msg, node)
+
+	// no codec specified
+	if cf == nil {
+		var err error
+		cf, err = r.newCodec(req.ContentType())
+		if err != nil {
+			return errors.InternalServerError("go.micro.client", err.Error())
+		}
 	}
 
 	var grr error
@@ -85,11 +104,22 @@ func (r *rpcClient) call(ctx context.Context, address string, req Request, resp
 		r.pool.release(address, c, grr)
 	}()
 
+	seq := atomic.LoadUint64(&r.seq)
+	atomic.AddUint64(&r.seq, 1)
+	codec := newRpcCodec(msg, c, cf)
+
+	rsp := &rpcResponse{
+		socket: c,
+		codec:  codec,
+	}
+
 	stream := &rpcStream{
-		context: ctx,
-		request: req,
-		closed:  make(chan bool),
-		codec:   newRpcPlusCodec(msg, c, cf),
+		context:  ctx,
+		request:  req,
+		response: rsp,
+		codec:    codec,
+		closed:   make(chan bool),
+		id:       fmt.Sprintf("%v", seq),
 	}
 	defer stream.Close()
 
@@ -103,7 +133,7 @@ func (r *rpcClient) call(ctx context.Context, address string, req Request, resp
 		}()
 
 		// send request
-		if err := stream.Send(req.Request()); err != nil {
+		if err := stream.Send(req.Body()); err != nil {
 			ch <- err
 			return
 		}
@@ -124,11 +154,16 @@ func (r *rpcClient) call(ctx context.Context, address string, req Request, resp
 		return err
 	case <-ctx.Done():
 		grr = ctx.Err()
-		return errors.New("go.micro.client", fmt.Sprintf("request timeout: %v", ctx.Err()), 408)
+		return errors.Timeout("go.micro.client", fmt.Sprintf("%v", ctx.Err()))
 	}
 }
 
-func (r *rpcClient) stream(ctx context.Context, address string, req Request, opts CallOptions) (Streamer, error) {
+func (r *rpcClient) stream(ctx context.Context, node *registry.Node, req Request, opts CallOptions) (Stream, error) {
+	address := node.Address
+	if node.Port > 0 {
+		address = fmt.Sprintf("%s:%d", address, node.Port)
+	}
+
 	msg := &transport.Message{
 		Header: make(map[string]string),
 	}
@@ -147,27 +182,55 @@ func (r *rpcClient) stream(ctx context.Context, address string, req Request, opt
 	// set the accept header
 	msg.Header["Accept"] = req.ContentType()
 
-	cf, err := r.newCodec(req.ContentType())
-	if err != nil {
-		return nil, errors.InternalServerError("go.micro.client", err.Error())
+	// set old codecs
+	cf := setupProtocol(msg, node)
+
+	// no codec specified
+	if cf == nil {
+		var err error
+		cf, err = r.newCodec(req.ContentType())
+		if err != nil {
+			return nil, errors.InternalServerError("go.micro.client", err.Error())
+		}
 	}
 
-	c, err := r.opts.Transport.Dial(address, transport.WithStream(), transport.WithTimeout(opts.DialTimeout))
+	dOpts := []transport.DialOption{
+		transport.WithStream(),
+	}
+
+	if opts.DialTimeout >= 0 {
+		dOpts = append(dOpts, transport.WithTimeout(opts.DialTimeout))
+	}
+
+	c, err := r.opts.Transport.Dial(address, dOpts...)
 	if err != nil {
 		return nil, errors.InternalServerError("go.micro.client", "connection error: %v", err)
 	}
 
+	codec := newRpcCodec(msg, c, cf)
+
+	rsp := &rpcResponse{
+		socket: c,
+		codec:  codec,
+	}
+
+	// set request codec
+	if r, ok := req.(*rpcRequest); ok {
+		r.codec = codec
+	}
+
 	stream := &rpcStream{
-		context: ctx,
-		request: req,
-		closed:  make(chan bool),
-		codec:   newRpcPlusCodec(msg, c, cf),
+		context:  ctx,
+		request:  req,
+		response: rsp,
+		closed:   make(chan bool),
+		codec:    codec,
 	}
 
 	ch := make(chan error, 1)
 
 	go func() {
-		ch <- stream.Send(req.Request())
+		ch <- stream.Send(req.Body())
 	}()
 
 	var grr error
@@ -176,7 +239,7 @@ func (r *rpcClient) stream(ctx context.Context, address string, req Request, opt
 	case err := <-ch:
 		grr = err
 	case <-ctx.Done():
-		grr = errors.New("go.micro.client", fmt.Sprintf("request timeout: %v", ctx.Err()), 408)
+		grr = errors.Timeout("go.micro.client", fmt.Sprintf("%v", ctx.Err()))
 	}
 
 	if grr != nil {
@@ -195,9 +258,12 @@ func (r *rpcClient) Init(opts ...Option) error {
 		o(&r.opts)
 	}
 
-	// recreate the pool if the options changed
+	// update pool configuration if the options changed
 	if size != r.opts.PoolSize || ttl != r.opts.PoolTTL {
-		r.pool = newPool(r.opts.PoolSize, r.opts.PoolTTL)
+		r.pool.Lock()
+		r.pool.size = r.opts.PoolSize
+		r.pool.ttl = int64(r.opts.PoolTTL.Seconds())
+		r.pool.Unlock()
 	}
 
 	return nil
@@ -207,13 +273,47 @@ func (r *rpcClient) Options() Options {
 	return r.opts
 }
 
-func (r *rpcClient) CallRemote(ctx context.Context, address string, request Request, response interface{}, opts ...CallOption) error {
-	// make a copy of call opts
-	callOpts := r.opts.CallOptions
-	for _, opt := range opts {
-		opt(&callOpts)
+func (r *rpcClient) next(request Request, opts CallOptions) (selector.Next, error) {
+	service := request.Service()
+
+	// get proxy
+	if prx := os.Getenv("MICRO_PROXY"); len(prx) > 0 {
+		service = prx
 	}
-	return r.call(ctx, address, request, response, callOpts)
+
+	// get proxy address
+	if prx := os.Getenv("MICRO_PROXY_ADDRESS"); len(prx) > 0 {
+		opts.Address = prx
+	}
+
+	// return remote address
+	if len(opts.Address) > 0 {
+		address := opts.Address
+		port := 0
+
+		host, sport, err := net.SplitHostPort(opts.Address)
+		if err == nil {
+			address = host
+			port, _ = strconv.Atoi(sport)
+		}
+
+		return func() (*registry.Node, error) {
+			return &registry.Node{
+				Address: address,
+				Port:    port,
+			}, nil
+		}, nil
+	}
+
+	// get next nodes from the selector
+	next, err := r.opts.Selector.Select(service, opts.SelectOptions...)
+	if err != nil && err == selector.ErrNotFound {
+		return nil, errors.NotFound("go.micro.client", "service %s: %v", service, err.Error())
+	} else if err != nil {
+		return nil, errors.InternalServerError("go.micro.client", "error selecting %s node: %v", service, err.Error())
+	}
+
+	return next, nil
 }
 
 func (r *rpcClient) Call(ctx context.Context, request Request, response interface{}, opts ...CallOption) error {
@@ -223,12 +323,9 @@ func (r *rpcClient) Call(ctx context.Context, request Request, response interfac
 		opt(&callOpts)
 	}
 
-	// get next nodes from the selector
-	next, err := r.opts.Selector.Select(request.Service(), callOpts.SelectOptions...)
-	if err != nil && err == selector.ErrNotFound {
-		return errors.NotFound("go.micro.client", err.Error())
-	} else if err != nil {
-		return errors.InternalServerError("go.micro.client", err.Error())
+	next, err := r.next(request, callOpts)
+	if err != nil {
+		return err
 	}
 
 	// check if we already have a deadline
@@ -246,7 +343,7 @@ func (r *rpcClient) Call(ctx context.Context, request Request, response interfac
 	// should we noop right here?
 	select {
 	case <-ctx.Done():
-		return errors.New("go.micro.client", fmt.Sprintf("%v", ctx.Err()), 408)
+		return errors.Timeout("go.micro.client", fmt.Sprintf("%v", ctx.Err()))
 	default:
 	}
 
@@ -263,7 +360,7 @@ func (r *rpcClient) Call(ctx context.Context, request Request, response interfac
 		// call backoff first. Someone may want an initial start delay
 		t, err := callOpts.Backoff(ctx, request, i)
 		if err != nil {
-			return errors.InternalServerError("go.micro.client", err.Error())
+			return errors.InternalServerError("go.micro.client", "backoff error: %v", err.Error())
 		}
 
 		// only sleep if greater than 0
@@ -274,34 +371,28 @@ func (r *rpcClient) Call(ctx context.Context, request Request, response interfac
 		// select next node
 		node, err := next()
 		if err != nil && err == selector.ErrNotFound {
-			return errors.NotFound("go.micro.client", err.Error())
+			return errors.NotFound("go.micro.client", "service %s: %v", request.Service(), err.Error())
 		} else if err != nil {
-			return errors.InternalServerError("go.micro.client", err.Error())
-		}
-
-		// set the address
-		address := node.Address
-		if node.Port > 0 {
-			address = fmt.Sprintf("%s:%d", address, node.Port)
+			return errors.InternalServerError("go.micro.client", "error getting next %s node: %v", request.Service(), err.Error())
 		}
 
 		// make the call
-		err = rcall(ctx, address, request, response, callOpts)
+		err = rcall(ctx, node, request, response, callOpts)
 		r.opts.Selector.Mark(request.Service(), node, err)
 		return err
 	}
 
-	ch := make(chan error, callOpts.Retries)
+	ch := make(chan error, callOpts.Retries+1)
 	var gerr error
 
-	for i := 0; i < callOpts.Retries; i++ {
-		go func() {
+	for i := 0; i <= callOpts.Retries; i++ {
+		go func(i int) {
 			ch <- call(i)
-		}()
+		}(i)
 
 		select {
 		case <-ctx.Done():
-			return errors.New("go.micro.client", fmt.Sprintf("call timeout: %v", ctx.Err()), 408)
+			return errors.Timeout("go.micro.client", fmt.Sprintf("call timeout: %v", ctx.Err()))
 		case err := <-ch:
 			// if the call succeeded lets bail early
 			if err == nil {
@@ -324,54 +415,30 @@ func (r *rpcClient) Call(ctx context.Context, request Request, response interfac
 	return gerr
 }
 
-func (r *rpcClient) StreamRemote(ctx context.Context, address string, request Request, opts ...CallOption) (Streamer, error) {
-	// make a copy of call opts
-	callOpts := r.opts.CallOptions
-	for _, opt := range opts {
-		opt(&callOpts)
-	}
-	return r.stream(ctx, address, request, callOpts)
-}
-
-func (r *rpcClient) Stream(ctx context.Context, request Request, opts ...CallOption) (Streamer, error) {
+func (r *rpcClient) Stream(ctx context.Context, request Request, opts ...CallOption) (Stream, error) {
 	// make a copy of call opts
 	callOpts := r.opts.CallOptions
 	for _, opt := range opts {
 		opt(&callOpts)
 	}
 
-	// get next nodes from the selector
-	next, err := r.opts.Selector.Select(request.Service(), callOpts.SelectOptions...)
-	if err != nil && err == selector.ErrNotFound {
-		return nil, errors.NotFound("go.micro.client", err.Error())
-	} else if err != nil {
-		return nil, errors.InternalServerError("go.micro.client", err.Error())
-	}
-
-	// check if we already have a deadline
-	d, ok := ctx.Deadline()
-	if !ok {
-		// no deadline so we create a new one
-		ctx, _ = context.WithTimeout(ctx, callOpts.RequestTimeout)
-	} else {
-		// got a deadline so no need to setup context
-		// but we need to set the timeout we pass along
-		opt := WithRequestTimeout(d.Sub(time.Now()))
-		opt(&callOpts)
+	next, err := r.next(request, callOpts)
+	if err != nil {
+		return nil, err
 	}
 
 	// should we noop right here?
 	select {
 	case <-ctx.Done():
-		return nil, errors.New("go.micro.client", fmt.Sprintf("%v", ctx.Err()), 408)
+		return nil, errors.Timeout("go.micro.client", fmt.Sprintf("%v", ctx.Err()))
 	default:
 	}
 
-	call := func(i int) (Streamer, error) {
+	call := func(i int) (Stream, error) {
 		// call backoff first. Someone may want an initial start delay
 		t, err := callOpts.Backoff(ctx, request, i)
 		if err != nil {
-			return nil, errors.InternalServerError("go.micro.client", err.Error())
+			return nil, errors.InternalServerError("go.micro.client", "backoff error: %v", err.Error())
 		}
 
 		// only sleep if greater than 0
@@ -381,38 +448,33 @@ func (r *rpcClient) Stream(ctx context.Context, request Request, opts ...CallOpt
 
 		node, err := next()
 		if err != nil && err == selector.ErrNotFound {
-			return nil, errors.NotFound("go.micro.client", err.Error())
+			return nil, errors.NotFound("go.micro.client", "service %s: %v", request.Service(), err.Error())
 		} else if err != nil {
-			return nil, errors.InternalServerError("go.micro.client", err.Error())
+			return nil, errors.InternalServerError("go.micro.client", "error getting next %s node: %v", request.Service(), err.Error())
 		}
 
-		address := node.Address
-		if node.Port > 0 {
-			address = fmt.Sprintf("%s:%d", address, node.Port)
-		}
-
-		stream, err := r.stream(ctx, address, request, callOpts)
+		stream, err := r.stream(ctx, node, request, callOpts)
 		r.opts.Selector.Mark(request.Service(), node, err)
 		return stream, err
 	}
 
 	type response struct {
-		stream Streamer
+		stream Stream
 		err    error
 	}
 
-	ch := make(chan response, callOpts.Retries)
+	ch := make(chan response, callOpts.Retries+1)
 	var grr error
 
-	for i := 0; i < callOpts.Retries; i++ {
-		go func() {
+	for i := 0; i <= callOpts.Retries; i++ {
+		go func(i int) {
 			s, err := call(i)
 			ch <- response{s, err}
-		}()
+		}(i)
 
 		select {
 		case <-ctx.Done():
-			return nil, errors.New("go.micro.client", fmt.Sprintf("call timeout: %v", ctx.Err()), 408)
+			return nil, errors.Timeout("go.micro.client", fmt.Sprintf("call timeout: %v", ctx.Err()))
 		case rsp := <-ch:
 			// if the call succeeded lets bail early
 			if rsp.err == nil {
@@ -435,49 +497,69 @@ func (r *rpcClient) Stream(ctx context.Context, request Request, opts ...CallOpt
 	return nil, grr
 }
 
-func (r *rpcClient) Publish(ctx context.Context, p Publication, opts ...PublishOption) error {
+func (r *rpcClient) Publish(ctx context.Context, msg Message, opts ...PublishOption) error {
+	options := PublishOptions{
+		Context: context.Background(),
+	}
+	for _, o := range opts {
+		o(&options)
+	}
+
 	md, ok := metadata.FromContext(ctx)
 	if !ok {
 		md = make(map[string]string)
 	}
-	md["Content-Type"] = p.ContentType()
+
+	id := uuid.New().String()
+	md["Content-Type"] = msg.ContentType()
+	md["Micro-Topic"] = msg.Topic()
+	md["Micro-Id"] = id
+
+	// set the topic
+	topic := msg.Topic()
+
+	// get proxy
+	if prx := os.Getenv("MICRO_PROXY"); len(prx) > 0 {
+		options.Exchange = prx
+	}
+
+	// get the exchange
+	if len(options.Exchange) > 0 {
+		topic = options.Exchange
+	}
 
 	// encode message body
-	cf, err := r.newCodec(p.ContentType())
+	cf, err := r.newCodec(msg.ContentType())
 	if err != nil {
 		return errors.InternalServerError("go.micro.client", err.Error())
 	}
 	b := &buffer{bytes.NewBuffer(nil)}
-	if err := cf(b).Write(&codec.Message{Type: codec.Publication}, p.Message()); err != nil {
+	if err := cf(b).Write(&codec.Message{
+		Target: topic,
+		Type:   codec.Publication,
+		Header: map[string]string{
+			"Micro-Id":    id,
+			"Micro-Topic": msg.Topic(),
+		},
+	}, msg.Payload()); err != nil {
 		return errors.InternalServerError("go.micro.client", err.Error())
 	}
 	r.once.Do(func() {
 		r.opts.Broker.Connect()
 	})
 
-	return r.opts.Broker.Publish(p.Topic(), &broker.Message{
+	return r.opts.Broker.Publish(topic, &broker.Message{
 		Header: md,
 		Body:   b.Bytes(),
 	})
 }
 
-func (r *rpcClient) NewPublication(topic string, message interface{}) Publication {
-	return newRpcPublication(topic, message, r.opts.ContentType)
+func (r *rpcClient) NewMessage(topic string, message interface{}, opts ...MessageOption) Message {
+	return newMessage(topic, message, r.opts.ContentType, opts...)
 }
 
-func (r *rpcClient) NewProtoPublication(topic string, message interface{}) Publication {
-	return newRpcPublication(topic, message, "application/octet-stream")
-}
 func (r *rpcClient) NewRequest(service, method string, request interface{}, reqOpts ...RequestOption) Request {
-	return newRpcRequest(service, method, request, r.opts.ContentType, reqOpts...)
-}
-
-func (r *rpcClient) NewProtoRequest(service, method string, request interface{}, reqOpts ...RequestOption) Request {
-	return newRpcRequest(service, method, request, "application/octet-stream", reqOpts...)
-}
-
-func (r *rpcClient) NewJsonRequest(service, method string, request interface{}, reqOpts ...RequestOption) Request {
-	return newRpcRequest(service, method, request, "application/json", reqOpts...)
+	return newRequest(service, method, request, r.opts.ContentType, reqOpts...)
 }
 
 func (r *rpcClient) String() string {

client/rpc_client_test.go

@@ -1,39 +1,47 @@
 package client
 
 import (
+	"context"
 	"fmt"
 	"testing"
 
+	"github.com/micro/go-micro/errors"
 	"github.com/micro/go-micro/registry"
-	"github.com/micro/go-micro/registry/mock"
+	"github.com/micro/go-micro/registry/memory"
 	"github.com/micro/go-micro/selector"
-
-	"golang.org/x/net/context"
 )
 
-func TestCallWrapper(t *testing.T) {
+func newTestRegistry() registry.Registry {
+	r := memory.NewRegistry()
+	r.(*memory.Registry).Setup()
+	return r
+}
+
+func TestCallAddress(t *testing.T) {
 	var called bool
-	id := "test.1"
 	service := "test.service"
-	method := "Test.Method"
-	host := "10.1.10.1"
+	endpoint := "Test.Endpoint"
+	address := "10.1.10.1"
 	port := 8080
-	address := "10.1.10.1:8080"
 
 	wrap := func(cf CallFunc) CallFunc {
-		return func(ctx context.Context, addr string, req Request, rsp interface{}, opts CallOptions) error {
+		return func(ctx context.Context, node *registry.Node, req Request, rsp interface{}, opts CallOptions) error {
 			called = true
 
 			if req.Service() != service {
 				return fmt.Errorf("expected service: %s got %s", service, req.Service())
 			}
 
-			if req.Method() != method {
-				return fmt.Errorf("expected service: %s got %s", method, req.Method())
+			if req.Endpoint() != endpoint {
+				return fmt.Errorf("expected service: %s got %s", endpoint, req.Endpoint())
 			}
 
-			if addr != address {
-				return fmt.Errorf("expected address: %s got %s", address, addr)
+			if node.Address != address {
+				return fmt.Errorf("expected address: %s got %s", address, node.Address)
+			}
+
+			if node.Port != port {
+				return fmt.Errorf("expected address: %d got %d", port, node.Port)
 			}
 
 			// don't do the call
@@ -41,7 +49,95 @@ func TestCallWrapper(t *testing.T) {
 		}
 	}
 
-	r := mock.NewRegistry()
+	r := newTestRegistry()
+	c := NewClient(
+		Registry(r),
+		WrapCall(wrap),
+	)
+	c.Options().Selector.Init(selector.Registry(r))
+
+	req := c.NewRequest(service, endpoint, nil)
+
+	// test calling remote address
+	if err := c.Call(context.Background(), req, nil, WithAddress(fmt.Sprintf("%s:%d", address, port))); err != nil {
+		t.Fatal("call with address error", err)
+	}
+
+	if !called {
+		t.Fatal("wrapper not called")
+	}
+
+}
+
+func TestCallRetry(t *testing.T) {
+	service := "test.service"
+	endpoint := "Test.Endpoint"
+	address := "10.1.10.1"
+
+	var called int
+
+	wrap := func(cf CallFunc) CallFunc {
+		return func(ctx context.Context, node *registry.Node, req Request, rsp interface{}, opts CallOptions) error {
+			called++
+			if called == 1 {
+				return errors.InternalServerError("test.error", "retry request")
+			}
+
+			// don't do the call
+			return nil
+		}
+	}
+
+	r := newTestRegistry()
+	c := NewClient(
+		Registry(r),
+		WrapCall(wrap),
+	)
+	c.Options().Selector.Init(selector.Registry(r))
+
+	req := c.NewRequest(service, endpoint, nil)
+
+	// test calling remote address
+	if err := c.Call(context.Background(), req, nil, WithAddress(address)); err != nil {
+		t.Fatal("call with address error", err)
+	}
+
+	// num calls
+	if called < c.Options().CallOptions.Retries+1 {
+		t.Fatal("request not retried")
+	}
+}
+
+func TestCallWrapper(t *testing.T) {
+	var called bool
+	id := "test.1"
+	service := "test.service"
+	endpoint := "Test.Endpoint"
+	address := "10.1.10.1"
+	port := 8080
+
+	wrap := func(cf CallFunc) CallFunc {
+		return func(ctx context.Context, node *registry.Node, req Request, rsp interface{}, opts CallOptions) error {
+			called = true
+
+			if req.Service() != service {
+				return fmt.Errorf("expected service: %s got %s", service, req.Service())
+			}
+
+			if req.Endpoint() != endpoint {
+				return fmt.Errorf("expected service: %s got %s", endpoint, req.Endpoint())
+			}
+
+			if node.Address != address {
+				return fmt.Errorf("expected address: %s got %s", address, node.Address)
+			}
+
+			// don't do the call
+			return nil
+		}
+	}
+
+	r := newTestRegistry()
 	c := NewClient(
 		Registry(r),
 		WrapCall(wrap),
@@ -54,13 +150,13 @@ func TestCallWrapper(t *testing.T) {
 		Nodes: []*registry.Node{
 			&registry.Node{
 				Id:      id,
-				Address: host,
+				Address: address,
 				Port:    port,
 			},
 		},
 	})
 
-	req := c.NewRequest(service, method, nil)
+	req := c.NewRequest(service, endpoint, nil)
 	if err := c.Call(context.Background(), req, nil); err != nil {
 		t.Fatal("call wrapper error", err)
 	}

client/rpc_codec.go

@@ -5,9 +5,14 @@ import (
 	errs "errors"
 
 	"github.com/micro/go-micro/codec"
+	raw "github.com/micro/go-micro/codec/bytes"
+	"github.com/micro/go-micro/codec/grpc"
+	"github.com/micro/go-micro/codec/json"
 	"github.com/micro/go-micro/codec/jsonrpc"
+	"github.com/micro/go-micro/codec/proto"
 	"github.com/micro/go-micro/codec/protorpc"
 	"github.com/micro/go-micro/errors"
+	"github.com/micro/go-micro/registry"
 	"github.com/micro/go-micro/transport"
 )
 
@@ -28,7 +33,7 @@ var (
 	errShutdown = errs.New("connection is shut down")
 )
 
-type rpcPlusCodec struct {
+type rpcCodec struct {
 	client transport.Client
 	codec  codec.Codec
 
@@ -41,31 +46,21 @@ type readWriteCloser struct {
 	rbuf *bytes.Buffer
 }
 
-type clientCodec interface {
-	WriteRequest(*request, interface{}) error
-	ReadResponseHeader(*response) error
-	ReadResponseBody(interface{}) error
-
-	Close() error
-}
-
-type request struct {
-	Service       string
-	ServiceMethod string   // format: "Service.Method"
-	Seq           uint64   // sequence number chosen by client
-	next          *request // for free list in Server
-}
-
-type response struct {
-	ServiceMethod string    // echoes that of the Request
-	Seq           uint64    // echoes that of the request
-	Error         string    // error, if any.
-	next          *response // for free list in Server
-}
-
 var (
-	defaultContentType = "application/octet-stream"
+	DefaultContentType = "application/protobuf"
 
+	DefaultCodecs = map[string]codec.NewCodec{
+		"application/grpc":         grpc.NewCodec,
+		"application/grpc+json":    grpc.NewCodec,
+		"application/grpc+proto":   grpc.NewCodec,
+		"application/protobuf":     proto.NewCodec,
+		"application/json":         json.NewCodec,
+		"application/json-rpc":     jsonrpc.NewCodec,
+		"application/proto-rpc":    protorpc.NewCodec,
+		"application/octet-stream": raw.NewCodec,
+	}
+
+	// TODO: remove legacy codec list
 	defaultCodecs = map[string]codec.NewCodec{
 		"application/json":         jsonrpc.NewCodec,
 		"application/json-rpc":     jsonrpc.NewCodec,
@@ -89,12 +84,77 @@ func (rwc *readWriteCloser) Close() error {
 	return nil
 }
 
-func newRpcPlusCodec(req *transport.Message, client transport.Client, c codec.NewCodec) *rpcPlusCodec {
+func getHeaders(m *codec.Message) {
+	get := func(hdr string) string {
+		if hd := m.Header[hdr]; len(hd) > 0 {
+			return hd
+		}
+		// old
+		return m.Header["X-"+hdr]
+	}
+
+	// check error in header
+	if len(m.Error) == 0 {
+		m.Error = get("Micro-Error")
+	}
+
+	// check endpoint in header
+	if len(m.Endpoint) == 0 {
+		m.Endpoint = get("Micro-Endpoint")
+	}
+
+	// check method in header
+	if len(m.Method) == 0 {
+		m.Method = get("Micro-Method")
+	}
+
+	if len(m.Id) == 0 {
+		m.Id = get("Micro-Id")
+	}
+}
+
+func setHeaders(m *codec.Message) {
+	set := func(hdr, v string) {
+		if len(v) == 0 {
+			return
+		}
+		m.Header[hdr] = v
+		m.Header["X-"+hdr] = v
+	}
+
+	set("Micro-Id", m.Id)
+	set("Micro-Service", m.Target)
+	set("Micro-Method", m.Method)
+	set("Micro-Endpoint", m.Endpoint)
+}
+
+// setupProtocol sets up the old protocol
+func setupProtocol(msg *transport.Message, node *registry.Node) codec.NewCodec {
+	protocol := node.Metadata["protocol"]
+
+	// got protocol
+	if len(protocol) > 0 {
+		return nil
+	}
+
+	// no protocol use old codecs
+	switch msg.Header["Content-Type"] {
+	case "application/json":
+		msg.Header["Content-Type"] = "application/json-rpc"
+	case "application/protobuf":
+		msg.Header["Content-Type"] = "application/proto-rpc"
+	}
+
+	// now return codec
+	return defaultCodecs[msg.Header["Content-Type"]]
+}
+
+func newRpcCodec(req *transport.Message, client transport.Client, c codec.NewCodec) codec.Codec {
 	rwc := &readWriteCloser{
 		wbuf: bytes.NewBuffer(nil),
 		rbuf: bytes.NewBuffer(nil),
 	}
-	r := &rpcPlusCodec{
+	r := &rpcCodec{
 		buf:    rwc,
 		client: client,
 		codec:  c(rwc),
@@ -103,54 +163,90 @@ func newRpcPlusCodec(req *transport.Message, client transport.Client, c codec.Ne
 	return r
 }
 
-func (c *rpcPlusCodec) WriteRequest(req *request, body interface{}) error {
+func (c *rpcCodec) Write(m *codec.Message, body interface{}) error {
 	c.buf.wbuf.Reset()
-	m := &codec.Message{
-		Id:     req.Seq,
-		Target: req.Service,
-		Method: req.ServiceMethod,
-		Type:   codec.Request,
-		Header: map[string]string{},
+
+	// create header
+	if m.Header == nil {
+		m.Header = map[string]string{}
 	}
-	if err := c.codec.Write(m, body); err != nil {
-		return errors.InternalServerError("go.micro.client.codec", err.Error())
+
+	// copy original header
+	for k, v := range c.req.Header {
+		m.Header[k] = v
 	}
-	c.req.Body = c.buf.wbuf.Bytes()
-	for k, v := range m.Header {
-		c.req.Header[k] = v
+
+	// set the mucp headers
+	setHeaders(m)
+
+	// if body is bytes Frame don't encode
+	if body != nil {
+		b, ok := body.(*raw.Frame)
+		if ok {
+			// set body
+			m.Body = b.Data
+			body = nil
+		}
 	}
-	if err := c.client.Send(c.req); err != nil {
+
+	if len(m.Body) == 0 {
+		// write to codec
+		if err := c.codec.Write(m, body); err != nil {
+			return errors.InternalServerError("go.micro.client.codec", err.Error())
+		}
+		// set body
+		m.Body = c.buf.wbuf.Bytes()
+	}
+
+	// create new transport message
+	msg := transport.Message{
+		Header: m.Header,
+		Body:   m.Body,
+	}
+	// send the request
+	if err := c.client.Send(&msg); err != nil {
 		return errors.InternalServerError("go.micro.client.transport", err.Error())
 	}
 	return nil
 }
 
-func (c *rpcPlusCodec) ReadResponseHeader(r *response) error {
-	var m transport.Message
-	if err := c.client.Recv(&m); err != nil {
+func (c *rpcCodec) ReadHeader(m *codec.Message, r codec.MessageType) error {
+	var tm transport.Message
+
+	// read message from transport
+	if err := c.client.Recv(&tm); err != nil {
 		return errors.InternalServerError("go.micro.client.transport", err.Error())
 	}
+
 	c.buf.rbuf.Reset()
-	c.buf.rbuf.Write(m.Body)
-	var me codec.Message
-	err := c.codec.ReadHeader(&me, codec.Response)
-	r.ServiceMethod = me.Method
-	r.Seq = me.Id
-	r.Error = me.Error
+	c.buf.rbuf.Write(tm.Body)
+
+	// set headers from transport
+	m.Header = tm.Header
+
+	// read header
+	err := c.codec.ReadHeader(m, r)
+
+	// get headers
+	getHeaders(m)
+
+	// return header error
 	if err != nil {
 		return errors.InternalServerError("go.micro.client.codec", err.Error())
 	}
+
 	return nil
 }
 
-func (c *rpcPlusCodec) ReadResponseBody(b interface{}) error {
+func (c *rpcCodec) ReadBody(b interface{}) error {
+	// read body
 	if err := c.codec.ReadBody(b); err != nil {
 		return errors.InternalServerError("go.micro.client.codec", err.Error())
 	}
 	return nil
 }
 
-func (c *rpcPlusCodec) Close() error {
+func (c *rpcCodec) Close() error {
 	c.buf.Close()
 	c.codec.Close()
 	if err := c.client.Close(); err != nil {
@@ -158,3 +254,7 @@ func (c *rpcPlusCodec) Close() error {
 	}
 	return nil
 }
+
+func (c *rpcCodec) String() string {
+	return "rpc"
+}

client/rpc_message.go

@@ -0,0 +1,36 @@
+package client
+
+type message struct {
+	topic       string
+	contentType string
+	payload     interface{}
+}
+
+func newMessage(topic string, payload interface{}, contentType string, opts ...MessageOption) Message {
+	var options MessageOptions
+	for _, o := range opts {
+		o(&options)
+	}
+
+	if len(options.ContentType) > 0 {
+		contentType = options.ContentType
+	}
+
+	return &message{
+		payload:     payload,
+		topic:       topic,
+		contentType: contentType,
+	}
+}
+
+func (m *message) ContentType() string {
+	return m.contentType
+}
+
+func (m *message) Topic() string {
+	return m.topic
+}
+
+func (m *message) Payload() interface{} {
+	return m.payload
+}

client/rpc_pool_test.go

@@ -5,7 +5,7 @@ import (
 	"time"
 
 	"github.com/micro/go-micro/transport"
-	"github.com/micro/go-micro/transport/mock"
+	"github.com/micro/go-micro/transport/memory"
 )
 
 func testPool(t *testing.T, size int, ttl time.Duration) {
@@ -13,7 +13,7 @@ func testPool(t *testing.T, size int, ttl time.Duration) {
 	p := newPool(size, ttl)
 
 	// mock transport
-	tr := mock.NewTransport()
+	tr := memory.NewTransport()
 
 	// listen
 	l, err := tr.Listen(":0")

client/rpc_publication.go

@@ -1,27 +0,0 @@
-package client
-
-type rpcPublication struct {
-	topic       string
-	contentType string
-	message     interface{}
-}
-
-func newRpcPublication(topic string, message interface{}, contentType string) Publication {
-	return &rpcPublication{
-		message:     message,
-		topic:       topic,
-		contentType: contentType,
-	}
-}
-
-func (r *rpcPublication) ContentType() string {
-	return r.contentType
-}
-
-func (r *rpcPublication) Topic() string {
-	return r.topic
-}
-
-func (r *rpcPublication) Message() interface{} {
-	return r.message
-}

client/rpc_request.go

@@ -1,24 +1,36 @@
 package client
 
+import (
+	"github.com/micro/go-micro/codec"
+)
+
 type rpcRequest struct {
 	service     string
 	method      string
+	endpoint    string
 	contentType string
-	request     interface{}
+	codec       codec.Codec
+	body        interface{}
 	opts        RequestOptions
 }
 
-func newRpcRequest(service, method string, request interface{}, contentType string, reqOpts ...RequestOption) Request {
+func newRequest(service, endpoint string, request interface{}, contentType string, reqOpts ...RequestOption) Request {
 	var opts RequestOptions
 
 	for _, o := range reqOpts {
 		o(&opts)
 	}
 
+	// set the content-type specified
+	if len(opts.ContentType) > 0 {
+		contentType = opts.ContentType
+	}
+
 	return &rpcRequest{
 		service:     service,
-		method:      method,
-		request:     request,
+		method:      endpoint,
+		endpoint:    endpoint,
+		body:        request,
 		contentType: contentType,
 		opts:        opts,
 	}
@@ -36,8 +48,16 @@ func (r *rpcRequest) Method() string {
 	return r.method
 }
 
-func (r *rpcRequest) Request() interface{} {
-	return r.request
+func (r *rpcRequest) Endpoint() string {
+	return r.endpoint
+}
+
+func (r *rpcRequest) Body() interface{} {
+	return r.body
+}
+
+func (r *rpcRequest) Codec() codec.Writer {
+	return r.codec
 }
 
 func (r *rpcRequest) Stream() bool {

client/rpc_request_test.go

@@ -0,0 +1,23 @@
+package client
+
+import (
+	"testing"
+)
+
+func TestRequestOptions(t *testing.T) {
+	r := newRequest("service", "endpoint", nil, "application/json")
+	if r.Service() != "service" {
+		t.Fatalf("expected 'service' got %s", r.Service())
+	}
+	if r.Endpoint() != "endpoint" {
+		t.Fatalf("expected 'endpoint' got %s", r.Endpoint())
+	}
+	if r.ContentType() != "application/json" {
+		t.Fatalf("expected 'endpoint' got %s", r.ContentType())
+	}
+
+	r2 := newRequest("service", "endpoint", nil, "application/json", WithContentType("application/protobuf"))
+	if r2.ContentType() != "application/protobuf" {
+		t.Fatalf("expected 'endpoint' got %s", r2.ContentType())
+	}
+}

client/rpc_response.go

@@ -0,0 +1,35 @@
+package client
+
+import (
+	"github.com/micro/go-micro/codec"
+	"github.com/micro/go-micro/transport"
+)
+
+type rpcResponse struct {
+	header map[string]string
+	body   []byte
+	socket transport.Socket
+	codec  codec.Codec
+}
+
+func (r *rpcResponse) Codec() codec.Reader {
+	return r.codec
+}
+
+func (r *rpcResponse) Header() map[string]string {
+	return r.header
+}
+
+func (r *rpcResponse) Read() ([]byte, error) {
+	var msg transport.Message
+
+	if err := r.socket.Recv(&msg); err != nil {
+		return nil, err
+	}
+
+	// set internals
+	r.header = msg.Header
+	r.body = msg.Body
+
+	return msg.Body, nil
+}

client/rpc_stream.go

@@ -1,21 +1,23 @@
 package client
 
 import (
+	"context"
 	"io"
 	"sync"
 
-	"golang.org/x/net/context"
+	"github.com/micro/go-micro/codec"
 )
 
 // Implements the streamer interface
 type rpcStream struct {
 	sync.RWMutex
-	seq     uint64
-	closed  chan bool
-	err     error
-	request Request
-	codec   clientCodec
-	context context.Context
+	id       string
+	closed   chan bool
+	err      error
+	request  Request
+	response Response
+	codec    codec.Codec
+	context  context.Context
 }
 
 func (r *rpcStream) isClosed() bool {
@@ -35,6 +37,10 @@ func (r *rpcStream) Request() Request {
 	return r.request
 }
 
+func (r *rpcStream) Response() Response {
+	return r.response
+}
+
 func (r *rpcStream) Send(msg interface{}) error {
 	r.Lock()
 	defer r.Unlock()
@@ -44,19 +50,19 @@ func (r *rpcStream) Send(msg interface{}) error {
 		return errShutdown
 	}
 
-	seq := r.seq
-	r.seq++
-
-	req := request{
-		Service:       r.request.Service(),
-		Seq:           seq,
-		ServiceMethod: r.request.Method(),
+	req := codec.Message{
+		Id:       r.id,
+		Target:   r.request.Service(),
+		Method:   r.request.Method(),
+		Endpoint: r.request.Endpoint(),
+		Type:     codec.Request,
 	}
 
-	if err := r.codec.WriteRequest(&req, msg); err != nil {
+	if err := r.codec.Write(&req, msg); err != nil {
 		r.err = err
 		return err
 	}
+
 	return nil
 }
 
@@ -69,8 +75,9 @@ func (r *rpcStream) Recv(msg interface{}) error {
 		return errShutdown
 	}
 
-	var resp response
-	if err := r.codec.ReadResponseHeader(&resp); err != nil {
+	var resp codec.Message
+
+	if err := r.codec.ReadHeader(&resp, codec.Response); err != nil {
 		if err == io.EOF && !r.isClosed() {
 			r.err = io.ErrUnexpectedEOF
 			return io.ErrUnexpectedEOF
@@ -89,11 +96,11 @@ func (r *rpcStream) Recv(msg interface{}) error {
 		} else {
 			r.err = io.EOF
 		}
-		if err := r.codec.ReadResponseBody(nil); err != nil {
+		if err := r.codec.ReadBody(nil); err != nil {
 			r.err = err
 		}
 	default:
-		if err := r.codec.ReadResponseBody(msg); err != nil {
+		if err := r.codec.ReadBody(msg); err != nil {
 			r.err = err
 		}
 	}

client/wrapper.go

@@ -0,0 +1,19 @@
+package client
+
+import (
+	"context"
+
+	"github.com/micro/go-micro/registry"
+)
+
+// CallFunc represents the individual call func
+type CallFunc func(ctx context.Context, node *registry.Node, req Request, rsp interface{}, opts CallOptions) error
+
+// CallWrapper is a low level wrapper for the CallFunc
+type CallWrapper func(CallFunc) CallFunc
+
+// Wrapper wraps a client and returns a client
+type Wrapper func(Client) Client
+
+// StreamWrapper wraps a Stream and returns the equivalent
+type StreamWrapper func(Stream) Stream

cmd/cmd.go

@@ -12,23 +12,30 @@ import (
 	"github.com/micro/cli"
 	"github.com/micro/go-micro/client"
 	"github.com/micro/go-micro/server"
+	"github.com/micro/go-micro/util/log"
 
 	// brokers
 	"github.com/micro/go-micro/broker"
 	"github.com/micro/go-micro/broker/http"
+	"github.com/micro/go-micro/broker/memory"
+	"github.com/micro/go-micro/broker/nats"
 
 	// registries
 	"github.com/micro/go-micro/registry"
 	"github.com/micro/go-micro/registry/consul"
+	"github.com/micro/go-micro/registry/gossip"
 	"github.com/micro/go-micro/registry/mdns"
+	rmem "github.com/micro/go-micro/registry/memory"
 
 	// selectors
 	"github.com/micro/go-micro/selector"
-	"github.com/micro/go-micro/selector/cache"
+	"github.com/micro/go-micro/selector/dns"
+	"github.com/micro/go-micro/selector/static"
 
 	// transports
 	"github.com/micro/go-micro/transport"
 	thttp "github.com/micro/go-micro/transport/http"
+	tmem "github.com/micro/go-micro/transport/memory"
 )
 
 type Cmd interface {
@@ -65,19 +72,34 @@ var (
 		cli.IntFlag{
 			Name:   "client_retries",
 			EnvVar: "MICRO_CLIENT_RETRIES",
+			Value:  client.DefaultRetries,
 			Usage:  "Sets the client retries. Default: 1",
 		},
 		cli.IntFlag{
 			Name:   "client_pool_size",
 			EnvVar: "MICRO_CLIENT_POOL_SIZE",
 			Usage:  "Sets the client connection pool size. Default: 1",
-			Value:  1,
 		},
 		cli.StringFlag{
 			Name:   "client_pool_ttl",
 			EnvVar: "MICRO_CLIENT_POOL_TTL",
 			Usage:  "Sets the client connection pool ttl. e.g 500ms, 5s, 1m. Default: 1m",
 		},
+		cli.IntFlag{
+			Name:   "register_ttl",
+			EnvVar: "MICRO_REGISTER_TTL",
+			Usage:  "Register TTL in seconds",
+		},
+		cli.IntFlag{
+			Name:   "register_interval",
+			EnvVar: "MICRO_REGISTER_INTERVAL",
+			Usage:  "Register interval in seconds",
+		},
+		cli.StringFlag{
+			Name:   "server",
+			EnvVar: "MICRO_SERVER",
+			Usage:  "Server for go-micro; rpc",
+		},
 		cli.StringFlag{
 			Name:   "server_name",
 			EnvVar: "MICRO_SERVER_NAME",
@@ -133,12 +155,6 @@ var (
 			Name:   "selector",
 			EnvVar: "MICRO_SELECTOR",
 			Usage:  "Selector used to pick nodes for querying",
-			Value:  "cache",
-		},
-		cli.StringFlag{
-			Name:   "server",
-			EnvVar: "MICRO_SERVER",
-			Usage:  "Server for go-micro; rpc",
 		},
 		cli.StringFlag{
 			Name:   "transport",
@@ -153,7 +169,9 @@ var (
 	}
 
 	DefaultBrokers = map[string]func(...broker.Option) broker.Broker{
-		"http": http.NewBroker,
+		"http":   http.NewBroker,
+		"memory": memory.NewBroker,
+		"nats":   nats.NewBroker,
 	}
 
 	DefaultClients = map[string]func(...client.Option) client.Client{
@@ -162,12 +180,16 @@ var (
 
 	DefaultRegistries = map[string]func(...registry.Option) registry.Registry{
 		"consul": consul.NewRegistry,
+		"gossip": gossip.NewRegistry,
 		"mdns":   mdns.NewRegistry,
+		"memory": rmem.NewRegistry,
 	}
 
 	DefaultSelectors = map[string]func(...selector.Option) selector.Selector{
 		"default": selector.NewSelector,
-		"cache":   cache.NewSelector,
+		"dns":     dns.NewSelector,
+		"cache":   selector.NewSelector,
+		"static":  static.NewSelector,
 	}
 
 	DefaultServers = map[string]func(...server.Option) server.Server{
@@ -175,15 +197,16 @@ var (
 	}
 
 	DefaultTransports = map[string]func(...transport.Option) transport.Transport{
-		"http": thttp.NewTransport,
+		"memory": tmem.NewTransport,
+		"http":   thttp.NewTransport,
 	}
 
 	// used for default selection as the fall back
 	defaultClient    = "rpc"
 	defaultServer    = "rpc"
 	defaultBroker    = "http"
-	defaultRegistry  = "consul"
-	defaultSelector  = "cache"
+	defaultRegistry  = "mdns"
+	defaultSelector  = "registry"
 	defaultTransport = "http"
 )
 
@@ -253,83 +276,75 @@ func (c *cmd) Before(ctx *cli.Context) error {
 
 	// Set the client
 	if name := ctx.String("client"); len(name) > 0 {
-		if cl, ok := c.opts.Clients[name]; ok {
+		// only change if we have the client and type differs
+		if cl, ok := c.opts.Clients[name]; ok && (*c.opts.Client).String() != name {
 			*c.opts.Client = cl()
 		}
 	}
 
 	// Set the server
 	if name := ctx.String("server"); len(name) > 0 {
-		if s, ok := c.opts.Servers[name]; ok {
+		// only change if we have the server and type differs
+		if s, ok := c.opts.Servers[name]; ok && (*c.opts.Server).String() != name {
 			*c.opts.Server = s()
 		}
 	}
 
 	// Set the broker
-	if name := ctx.String("broker"); len(name) > 0 || len(ctx.String("broker_address")) > 0 {
-		if len(name) == 0 {
-			name = defaultBroker
-		}
-
-		if b, ok := c.opts.Brokers[name]; ok {
-			n := b(broker.Addrs(strings.Split(ctx.String("broker_address"), ",")...))
-			*c.opts.Broker = n
-		} else {
+	if name := ctx.String("broker"); len(name) > 0 && (*c.opts.Broker).String() != name {
+		b, ok := c.opts.Brokers[name]
+		if !ok {
 			return fmt.Errorf("Broker %s not found", name)
 		}
 
+		*c.opts.Broker = b()
 		serverOpts = append(serverOpts, server.Broker(*c.opts.Broker))
 		clientOpts = append(clientOpts, client.Broker(*c.opts.Broker))
 	}
 
 	// Set the registry
-	if name := ctx.String("registry"); len(name) > 0 || len(ctx.String("registry_address")) > 0 {
-		if len(name) == 0 {
-			name = defaultRegistry
-		}
-
-		if r, ok := c.opts.Registries[name]; ok {
-			n := r(registry.Addrs(strings.Split(ctx.String("registry_address"), ",")...))
-			*c.opts.Registry = n
-		} else {
+	if name := ctx.String("registry"); len(name) > 0 && (*c.opts.Registry).String() != name {
+		r, ok := c.opts.Registries[name]
+		if !ok {
 			return fmt.Errorf("Registry %s not found", name)
 		}
 
+		*c.opts.Registry = r()
 		serverOpts = append(serverOpts, server.Registry(*c.opts.Registry))
 		clientOpts = append(clientOpts, client.Registry(*c.opts.Registry))
 
-		(*c.opts.Selector).Init(selector.Registry(*c.opts.Registry))
+		if err := (*c.opts.Selector).Init(selector.Registry(*c.opts.Registry)); err != nil {
+			log.Fatalf("Error configuring registry: %v", err)
+		}
+
 		clientOpts = append(clientOpts, client.Selector(*c.opts.Selector))
 
-		(*c.opts.Broker).Init(broker.Registry(*c.opts.Registry))
+		if err := (*c.opts.Broker).Init(broker.Registry(*c.opts.Registry)); err != nil {
+			log.Fatalf("Error configuring broker: %v", err)
+		}
 	}
 
 	// Set the selector
-	if name := ctx.String("selector"); len(name) > 0 {
-		if s, ok := c.opts.Selectors[name]; ok {
-			n := s(selector.Registry(*c.opts.Registry))
-			*c.opts.Selector = n
-		} else {
+	if name := ctx.String("selector"); len(name) > 0 && (*c.opts.Selector).String() != name {
+		s, ok := c.opts.Selectors[name]
+		if !ok {
 			return fmt.Errorf("Selector %s not found", name)
 		}
 
+		*c.opts.Selector = s(selector.Registry(*c.opts.Registry))
+
 		// No server option here. Should there be?
 		clientOpts = append(clientOpts, client.Selector(*c.opts.Selector))
 	}
 
 	// Set the transport
-	if name := ctx.String("transport"); len(name) > 0 || len(ctx.String("transport_address")) > 0 {
-		if len(name) == 0 {
-			name = defaultTransport
-		}
-
-		if t, ok := c.opts.Transports[name]; ok {
-			n := t(transport.Addrs(strings.Split(ctx.String("transport_address"), ",")...))
-			*c.opts.Transport = n
-		} else {
+	if name := ctx.String("transport"); len(name) > 0 && (*c.opts.Transport).String() != name {
+		t, ok := c.opts.Transports[name]
+		if !ok {
 			return fmt.Errorf("Transport %s not found", name)
 		}
 
+		*c.opts.Transport = t()
 		serverOpts = append(serverOpts, server.Transport(*c.opts.Transport))
 		clientOpts = append(clientOpts, client.Transport(*c.opts.Transport))
 	}
@@ -350,6 +365,24 @@ func (c *cmd) Before(ctx *cli.Context) error {
 		serverOpts = append(serverOpts, server.Metadata(metadata))
 	}
 
+	if len(ctx.String("broker_address")) > 0 {
+		if err := (*c.opts.Broker).Init(broker.Addrs(strings.Split(ctx.String("broker_address"), ",")...)); err != nil {
+			log.Fatalf("Error configuring broker: %v", err)
+		}
+	}
+
+	if len(ctx.String("registry_address")) > 0 {
+		if err := (*c.opts.Registry).Init(registry.Addrs(strings.Split(ctx.String("registry_address"), ",")...)); err != nil {
+			log.Fatalf("Error configuring registry: %v", err)
+		}
+	}
+
+	if len(ctx.String("transport_address")) > 0 {
+		if err := (*c.opts.Transport).Init(transport.Addrs(strings.Split(ctx.String("transport_address"), ",")...)); err != nil {
+			log.Fatalf("Error configuring transport: %v", err)
+		}
+	}
+
 	if len(ctx.String("server_name")) > 0 {
 		serverOpts = append(serverOpts, server.Name(ctx.String("server_name")))
 	}
@@ -370,8 +403,16 @@ func (c *cmd) Before(ctx *cli.Context) error {
 		serverOpts = append(serverOpts, server.Advertise(ctx.String("server_advertise")))
 	}
 
+	if ttl := time.Duration(ctx.GlobalInt("register_ttl")); ttl > 0 {
+		serverOpts = append(serverOpts, server.RegisterTTL(ttl*time.Second))
+	}
+
+	if val := time.Duration(ctx.GlobalInt("register_interval")); val > 0 {
+		serverOpts = append(serverOpts, server.RegisterInterval(val*time.Second))
+	}
+
 	// client opts
-	if r := ctx.Int("client_retries"); r > 0 {
+	if r := ctx.Int("client_retries"); r >= 0 {
 		clientOpts = append(clientOpts, client.Retries(r))
 	}
 
@@ -398,12 +439,16 @@ func (c *cmd) Before(ctx *cli.Context) error {
 	// We have some command line opts for the server.
 	// Lets set it up
 	if len(serverOpts) > 0 {
-		(*c.opts.Server).Init(serverOpts...)
+		if err := (*c.opts.Server).Init(serverOpts...); err != nil {
+			log.Fatalf("Error configuring server: %v", err)
+		}
 	}
 
 	// Use an init option?
 	if len(clientOpts) > 0 {
-		(*c.opts.Client).Init(clientOpts...)
+		if err := (*c.opts.Client).Init(clientOpts...); err != nil {
+			log.Fatalf("Error configuring client: %v", err)
+		}
 	}
 
 	return nil

cmd/options.go

@@ -1,14 +1,14 @@
 package cmd
 
 import (
+	"context"
+
 	"github.com/micro/go-micro/broker"
 	"github.com/micro/go-micro/client"
 	"github.com/micro/go-micro/registry"
 	"github.com/micro/go-micro/selector"
 	"github.com/micro/go-micro/server"
 	"github.com/micro/go-micro/transport"
-
-	"golang.org/x/net/context"
 )
 
 type Options struct {

codec/bytes/bytes.go

@@ -0,0 +1,75 @@
+// Package bytes provides a bytes codec which does not encode or decode anything
+package bytes
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+
+	"github.com/micro/go-micro/codec"
+)
+
+type Codec struct {
+	Conn io.ReadWriteCloser
+}
+
+// Frame gives us the ability to define raw data to send over the pipes
+type Frame struct {
+	Data []byte
+}
+
+func (c *Codec) ReadHeader(m *codec.Message, t codec.MessageType) error {
+	return nil
+}
+
+func (c *Codec) ReadBody(b interface{}) error {
+	// read bytes
+	buf, err := ioutil.ReadAll(c.Conn)
+	if err != nil {
+		return err
+	}
+
+	switch b.(type) {
+	case *[]byte:
+		v := b.(*[]byte)
+		*v = buf
+	case *Frame:
+		v := b.(*Frame)
+		v.Data = buf
+	default:
+		return fmt.Errorf("failed to read body: %v is not type of *[]byte", b)
+	}
+
+	return nil
+}
+
+func (c *Codec) Write(m *codec.Message, b interface{}) error {
+	var v []byte
+	switch b.(type) {
+	case *Frame:
+		v = b.(*Frame).Data
+	case *[]byte:
+		ve := b.(*[]byte)
+		v = *ve
+	case []byte:
+		v = b.([]byte)
+	default:
+		return fmt.Errorf("failed to write: %v is not type of *[]byte or []byte", b)
+	}
+	_, err := c.Conn.Write(v)
+	return err
+}
+
+func (c *Codec) Close() error {
+	return c.Conn.Close()
+}
+
+func (c *Codec) String() string {
+	return "bytes"
+}
+
+func NewCodec(c io.ReadWriteCloser) codec.Codec {
+	return &Codec{
+		Conn: c,
+	}
+}

codec/bytes/marshaler.go

@@ -0,0 +1,41 @@
+package bytes
+
+import (
+	"errors"
+)
+
+type Marshaler struct{}
+
+type Message struct {
+	Header map[string]string
+	Body   []byte
+}
+
+func (n Marshaler) Marshal(v interface{}) ([]byte, error) {
+	switch v.(type) {
+	case *[]byte:
+		ve := v.(*[]byte)
+		return *ve, nil
+	case []byte:
+		return v.([]byte), nil
+	case *Message:
+		return v.(*Message).Body, nil
+	}
+	return nil, errors.New("invalid message")
+}
+
+func (n Marshaler) Unmarshal(d []byte, v interface{}) error {
+	switch v.(type) {
+	case *[]byte:
+		ve := v.(*[]byte)
+		*ve = d
+	case *Message:
+		ve := v.(*Message)
+		ve.Body = d
+	}
+	return errors.New("invalid message")
+}
+
+func (n Marshaler) String() string {
+	return "bytes"
+}

codec/codec.go

@@ -23,10 +23,26 @@ type NewCodec func(io.ReadWriteCloser) Codec
 // connection. ReadBody may be called with a nil argument to force the
 // body to be read and discarded.
 type Codec interface {
+	Reader
+	Writer
+	Close() error
+	String() string
+}
+
+type Reader interface {
 	ReadHeader(*Message, MessageType) error
 	ReadBody(interface{}) error
+}
+
+type Writer interface {
 	Write(*Message, interface{}) error
-	Close() error
+}
+
+// Marshaler is a simple encoding interface used for the broker/transport
+// where headers are not supported by the underlying implementation.
+type Marshaler interface {
+	Marshal(interface{}) ([]byte, error)
+	Unmarshal([]byte, interface{}) error
 	String() string
 }
 
@@ -34,10 +50,14 @@ type Codec interface {
 // the communication, likely followed by the body.
 // In the case of an error, body may be nil.
 type Message struct {
-	Id     uint64
-	Type   MessageType
-	Target string
-	Method string
-	Error  string
+	Id       string
+	Type     MessageType
+	Target   string
+	Method   string
+	Endpoint string
+	Error    string
+
+	// The values read from the socket
 	Header map[string]string
+	Body   []byte
 }

codec/grpc/grpc.go

@@ -0,0 +1,132 @@
+// Package grpc provides a grpc codec
+package grpc
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"strings"
+
+	"github.com/golang/protobuf/proto"
+	"github.com/micro/go-micro/codec"
+)
+
+type Codec struct {
+	Conn        io.ReadWriteCloser
+	ContentType string
+}
+
+func (c *Codec) ReadHeader(m *codec.Message, t codec.MessageType) error {
+	if ct := m.Header["Content-Type"]; len(ct) > 0 {
+		c.ContentType = ct
+	}
+
+	if ct := m.Header["content-type"]; len(ct) > 0 {
+		c.ContentType = ct
+	}
+
+	// service method
+	path := m.Header[":path"]
+	if len(path) == 0 || path[0] != '/' {
+		m.Target = m.Header["Micro-Service"]
+		m.Endpoint = m.Header["Micro-Endpoint"]
+	} else {
+		// [ , a.package.Foo, Bar]
+		parts := strings.Split(path, "/")
+		if len(parts) != 3 {
+			return errors.New("Unknown request path")
+		}
+		service := strings.Split(parts[1], ".")
+		m.Endpoint = strings.Join([]string{service[len(service)-1], parts[2]}, ".")
+		m.Target = strings.Join(service[:len(service)-1], ".")
+	}
+
+	return nil
+}
+
+func (c *Codec) ReadBody(b interface{}) error {
+	// no body
+	if b == nil {
+		return nil
+	}
+
+	_, buf, err := decode(c.Conn)
+	if err != nil {
+		return err
+	}
+
+	switch c.ContentType {
+	case "application/grpc+json":
+		return json.Unmarshal(buf, b)
+	case "application/grpc+proto", "application/grpc":
+		return proto.Unmarshal(buf, b.(proto.Message))
+	}
+
+	return errors.New("Unsupported Content-Type")
+}
+
+func (c *Codec) Write(m *codec.Message, b interface{}) error {
+	var buf []byte
+	var err error
+
+	if ct := m.Header["Content-Type"]; len(ct) > 0 {
+		c.ContentType = ct
+	}
+
+	if ct := m.Header["content-type"]; len(ct) > 0 {
+		c.ContentType = ct
+	}
+
+	switch m.Type {
+	case codec.Request:
+		parts := strings.Split(m.Endpoint, ".")
+		m.Header[":method"] = "POST"
+		m.Header[":path"] = fmt.Sprintf("/%s.%s/%s", m.Target, parts[0], parts[1])
+		m.Header[":proto"] = "HTTP/2.0"
+		m.Header["te"] = "trailers"
+		m.Header["user-agent"] = "grpc-go/1.0.0"
+		m.Header[":authority"] = m.Target
+		m.Header["content-type"] = c.ContentType
+	case codec.Response:
+		m.Header["Trailer"] = "grpc-status, grpc-message"
+		m.Header["grpc-status"] = "0"
+		m.Header["grpc-message"] = ""
+	}
+
+	// marshal content
+	switch c.ContentType {
+	case "application/grpc+json":
+		buf, err = json.Marshal(b)
+	case "application/grpc+proto", "application/grpc":
+		pb, ok := b.(proto.Message)
+		if ok {
+			buf, err = proto.Marshal(pb)
+		}
+	default:
+		err = errors.New("Unsupported Content-Type")
+	}
+	// check error
+	if err != nil {
+		m.Header["grpc-status"] = "8"
+		m.Header["grpc-message"] = err.Error()
+		return err
+	}
+
+	return encode(0, buf, c.Conn)
+}
+
+func (c *Codec) Close() error {
+	return c.Conn.Close()
+}
+
+func (c *Codec) String() string {
+	return "grpc"
+}
+
+func NewCodec(c io.ReadWriteCloser) codec.Codec {
+	return &Codec{
+		Conn:        c,
+		ContentType: "application/grpc",
+	}
+}

codec/grpc/util.go

@@ -0,0 +1,70 @@
+package grpc
+
+import (
+	"encoding/binary"
+	"fmt"
+	"io"
+)
+
+var (
+	maxMessageSize = 1024 * 1024 * 4
+	maxInt         = int(^uint(0) >> 1)
+)
+
+func decode(r io.Reader) (uint8, []byte, error) {
+	header := make([]byte, 5)
+
+	// read the header
+	if _, err := r.Read(header[:]); err != nil {
+		return uint8(0), nil, err
+	}
+
+	// get encoding format e.g compressed
+	cf := uint8(header[0])
+
+	// get message length
+	length := binary.BigEndian.Uint32(header[1:])
+
+	// no encoding format
+	if length == 0 {
+		return cf, nil, nil
+	}
+
+	//
+	if int64(length) > int64(maxInt) {
+		return cf, nil, fmt.Errorf("grpc: received message larger than max length allowed on current machine (%d vs. %d)", length, maxInt)
+	}
+	if int(length) > maxMessageSize {
+		return cf, nil, fmt.Errorf("grpc: received message larger than max (%d vs. %d)", length, maxMessageSize)
+	}
+
+	msg := make([]byte, int(length))
+
+	if _, err := r.Read(msg); err != nil {
+		if err == io.EOF {
+			err = io.ErrUnexpectedEOF
+		}
+		return cf, nil, err
+	}
+
+	return cf, msg, nil
+}
+
+func encode(cf uint8, buf []byte, w io.Writer) error {
+	header := make([]byte, 5)
+
+	// set compression
+	header[0] = byte(cf)
+
+	// write length as header
+	binary.BigEndian.PutUint32(header[1:], uint32(len(buf)))
+
+	// read the header
+	if _, err := w.Write(header[:]); err != nil {
+		return err
+	}
+
+	// write the buffer
+	_, err := w.Write(buf)
+	return err
+}

codec/json/json.go

@@ -0,0 +1,49 @@
+// Package json provides a json codec
+package json
+
+import (
+	"encoding/json"
+	"io"
+
+	"github.com/micro/go-micro/codec"
+)
+
+type Codec struct {
+	Conn    io.ReadWriteCloser
+	Encoder *json.Encoder
+	Decoder *json.Decoder
+}
+
+func (c *Codec) ReadHeader(m *codec.Message, t codec.MessageType) error {
+	return nil
+}
+
+func (c *Codec) ReadBody(b interface{}) error {
+	if b == nil {
+		return nil
+	}
+	return c.Decoder.Decode(b)
+}
+
+func (c *Codec) Write(m *codec.Message, b interface{}) error {
+	if b == nil {
+		return nil
+	}
+	return c.Encoder.Encode(b)
+}
+
+func (c *Codec) Close() error {
+	return c.Conn.Close()
+}
+
+func (c *Codec) String() string {
+	return "json"
+}
+
+func NewCodec(c io.ReadWriteCloser) codec.Codec {
+	return &Codec{
+		Conn:    c,
+		Decoder: json.NewDecoder(c),
+		Encoder: json.NewEncoder(c),
+	}
+}

codec/json/marshaler.go

@@ -0,0 +1,19 @@
+package json
+
+import (
+	"encoding/json"
+)
+
+type Marshaler struct{}
+
+func (j Marshaler) Marshal(v interface{}) ([]byte, error) {
+	return json.Marshal(v)
+}
+
+func (j Marshaler) Unmarshal(d []byte, v interface{}) error {
+	return json.Unmarshal(d, v)
+}
+
+func (j Marshaler) String() string {
+	return "json"
+}

codec/jsonrpc/client.go

@@ -19,17 +19,17 @@ type clientCodec struct {
 	resp clientResponse
 
 	sync.Mutex
-	pending map[uint64]string
+	pending map[interface{}]string
 }
 
 type clientRequest struct {
 	Method string         `json:"method"`
 	Params [1]interface{} `json:"params"`
-	ID     uint64         `json:"id"`
+	ID     interface{}    `json:"id"`
 }
 
 type clientResponse struct {
-	ID     uint64           `json:"id"`
+	ID     interface{}      `json:"id"`
 	Result *json.RawMessage `json:"result"`
 	Error  interface{}      `json:"error"`
 }
@@ -39,7 +39,7 @@ func newClientCodec(conn io.ReadWriteCloser) *clientCodec {
 		dec:     json.NewDecoder(conn),
 		enc:     json.NewEncoder(conn),
 		c:       conn,
-		pending: make(map[uint64]string),
+		pending: make(map[interface{}]string),
 	}
 }
 
@@ -71,7 +71,7 @@ func (c *clientCodec) ReadHeader(m *codec.Message) error {
 	c.Unlock()
 
 	m.Error = ""
-	m.Id = c.resp.ID
+	m.Id = fmt.Sprintf("%v", c.resp.ID)
 	if c.resp.Error != nil {
 		x, ok := c.resp.Error.(string)
 		if !ok {

codec/jsonrpc/jsonrpc.go

@@ -1,3 +1,4 @@
+// Package jsonrpc provides a json-rpc 1.0 codec
 package jsonrpc
 
 import (
@@ -30,7 +31,7 @@ func (j *jsonCodec) Write(m *codec.Message, b interface{}) error {
 	switch m.Type {
 	case codec.Request:
 		return j.c.Write(m, b)
-	case codec.Response:
+	case codec.Response, codec.Error:
 		return j.s.Write(m, b)
 	case codec.Publication:
 		data, err := json.Marshal(b)
@@ -54,7 +55,8 @@ func (j *jsonCodec) ReadHeader(m *codec.Message, mt codec.MessageType) error {
 	case codec.Response:
 		return j.c.ReadHeader(m)
 	case codec.Publication:
-		io.Copy(j.buf, j.rwc)
+		_, err := io.Copy(j.buf, j.rwc)
+		return err
 	default:
 		return fmt.Errorf("Unrecognised message type: %v", mt)
 	}

codec/jsonrpc/server.go

@@ -2,9 +2,8 @@ package jsonrpc
 
 import (
 	"encoding/json"
-	"errors"
+	"fmt"
 	"io"
-	"sync"
 
 	"github.com/micro/go-micro/codec"
 )
@@ -17,30 +16,25 @@ type serverCodec struct {
 	// temporary work space
 	req  serverRequest
 	resp serverResponse
-
-	sync.Mutex
-	seq     uint64
-	pending map[uint64]*json.RawMessage
 }
 
 type serverRequest struct {
 	Method string           `json:"method"`
 	Params *json.RawMessage `json:"params"`
-	ID     *json.RawMessage `json:"id"`
+	ID     interface{}      `json:"id"`
 }
 
 type serverResponse struct {
-	ID     *json.RawMessage `json:"id"`
-	Result interface{}      `json:"result"`
-	Error  interface{}      `json:"error"`
+	ID     interface{} `json:"id"`
+	Result interface{} `json:"result"`
+	Error  interface{} `json:"error"`
 }
 
 func newServerCodec(conn io.ReadWriteCloser) *serverCodec {
 	return &serverCodec{
-		dec:     json.NewDecoder(conn),
-		enc:     json.NewEncoder(conn),
-		c:       conn,
-		pending: make(map[uint64]*json.RawMessage),
+		dec: json.NewDecoder(conn),
+		enc: json.NewEncoder(conn),
+		c:   conn,
 	}
 }
 
@@ -50,7 +44,7 @@ func (r *serverRequest) reset() {
 		*r.Params = (*r.Params)[0:0]
 	}
 	if r.ID != nil {
-		*r.ID = (*r.ID)[0:0]
+		r.ID = nil
 	}
 }
 
@@ -60,14 +54,8 @@ func (c *serverCodec) ReadHeader(m *codec.Message) error {
 		return err
 	}
 	m.Method = c.req.Method
-
-	c.Lock()
-	c.seq++
-	c.pending[c.seq] = c.req.ID
+	m.Id = fmt.Sprintf("%v", c.req.ID)
 	c.req.ID = nil
-	m.Id = c.seq
-	c.Unlock()
-
 	return nil
 }
 
@@ -84,19 +72,7 @@ var null = json.RawMessage([]byte("null"))
 
 func (c *serverCodec) Write(m *codec.Message, x interface{}) error {
 	var resp serverResponse
-	c.Lock()
-	b, ok := c.pending[m.Id]
-	if !ok {
-		c.Unlock()
-		return errors.New("invalid sequence number in response")
-	}
-	c.Unlock()
-
-	if b == nil {
-		// Invalid request so no id.  Use JSON null.
-		b = &null
-	}
-	resp.ID = b
+	resp.ID = m.Id
 	resp.Result = x
 	if m.Error == "" {
 		resp.Error = nil

codec/proto/marshaler.go

@@ -0,0 +1,19 @@
+package proto
+
+import (
+	"github.com/golang/protobuf/proto"
+)
+
+type Marshaler struct{}
+
+func (Marshaler) Marshal(v interface{}) ([]byte, error) {
+	return proto.Marshal(v.(proto.Message))
+}
+
+func (Marshaler) Unmarshal(data []byte, v interface{}) error {
+	return proto.Unmarshal(data, v.(proto.Message))
+}
+
+func (Marshaler) String() string {
+	return "proto"
+}

codec/proto/proto.go

@@ -0,0 +1,56 @@
+// Package proto provides a proto codec
+package proto
+
+import (
+	"io"
+	"io/ioutil"
+
+	"github.com/golang/protobuf/proto"
+	"github.com/micro/go-micro/codec"
+)
+
+type Codec struct {
+	Conn io.ReadWriteCloser
+}
+
+func (c *Codec) ReadHeader(m *codec.Message, t codec.MessageType) error {
+	return nil
+}
+
+func (c *Codec) ReadBody(b interface{}) error {
+	if b == nil {
+		return nil
+	}
+	buf, err := ioutil.ReadAll(c.Conn)
+	if err != nil {
+		return err
+	}
+	return proto.Unmarshal(buf, b.(proto.Message))
+}
+
+func (c *Codec) Write(m *codec.Message, b interface{}) error {
+	p, ok := b.(proto.Message)
+	if !ok {
+		return nil
+	}
+	buf, err := proto.Marshal(p)
+	if err != nil {
+		return err
+	}
+	_, err = c.Conn.Write(buf)
+	return err
+}
+
+func (c *Codec) Close() error {
+	return c.Conn.Close()
+}
+
+func (c *Codec) String() string {
+	return "proto"
+}
+
+func NewCodec(c io.ReadWriteCloser) codec.Codec {
+	return &Codec{
+		Conn: c,
+	}
+}

codec/protorpc/protorpc.go

@@ -1,9 +1,11 @@
+// Protorpc provides a net/rpc proto-rpc codec. See envelope.proto for the format.
 package protorpc
 
 import (
 	"bytes"
 	"fmt"
 	"io"
+	"strconv"
 	"sync"
 
 	"github.com/golang/protobuf/proto"
@@ -30,13 +32,22 @@ func (c *protoCodec) String() string {
 	return "proto-rpc"
 }
 
+func id(id string) *uint64 {
+	p, err := strconv.ParseInt(id, 10, 64)
+	if err != nil {
+		p = 0
+	}
+	i := uint64(p)
+	return &i
+}
+
 func (c *protoCodec) Write(m *codec.Message, b interface{}) error {
 	switch m.Type {
 	case codec.Request:
 		c.Lock()
 		defer c.Unlock()
 		// This is protobuf, of course we copy it.
-		pbr := &Request{ServiceMethod: &m.Method, Seq: &m.Id}
+		pbr := &Request{ServiceMethod: &m.Method, Seq: id(m.Id)}
 		data, err := proto.Marshal(pbr)
 		if err != nil {
 			return err
@@ -55,12 +66,14 @@ func (c *protoCodec) Write(m *codec.Message, b interface{}) error {
 			return err
 		}
 		if flusher, ok := c.rwc.(flusher); ok {
-			err = flusher.Flush()
+			if err = flusher.Flush(); err != nil {
+				return err
+			}
 		}
-	case codec.Response:
+	case codec.Response, codec.Error:
 		c.Lock()
 		defer c.Unlock()
-		rtmp := &Response{ServiceMethod: &m.Method, Seq: &m.Id, Error: &m.Error}
+		rtmp := &Response{ServiceMethod: &m.Method, Seq: id(m.Id), Error: &m.Error}
 		data, err := proto.Marshal(rtmp)
 		if err != nil {
 			return err
@@ -82,7 +95,9 @@ func (c *protoCodec) Write(m *codec.Message, b interface{}) error {
 			return err
 		}
 		if flusher, ok := c.rwc.(flusher); ok {
-			err = flusher.Flush()
+			if err = flusher.Flush(); err != nil {
+				return err
+			}
 		}
 	case codec.Publication:
 		data, err := proto.Marshal(b.(proto.Message))
@@ -112,7 +127,7 @@ func (c *protoCodec) ReadHeader(m *codec.Message, mt codec.MessageType) error {
 			return err
 		}
 		m.Method = rtmp.GetServiceMethod()
-		m.Id = rtmp.GetSeq()
+		m.Id = fmt.Sprintf("%d", rtmp.GetSeq())
 	case codec.Response:
 		data, err := ReadNetString(c.rwc)
 		if err != nil {
@@ -124,10 +139,11 @@ func (c *protoCodec) ReadHeader(m *codec.Message, mt codec.MessageType) error {
 			return err
 		}
 		m.Method = rtmp.GetServiceMethod()
-		m.Id = rtmp.GetSeq()
+		m.Id = fmt.Sprintf("%d", rtmp.GetSeq())
 		m.Error = rtmp.GetError()
 	case codec.Publication:
-		io.Copy(c.buf, c.rwc)
+		_, err := io.Copy(c.buf, c.rwc)
+		return err
 	default:
 		return fmt.Errorf("Unrecognised message type: %v", mt)
 	}

codec/text/text.go

@@ -0,0 +1,75 @@
+// Package text reads any text/* content-type
+package text
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+
+	"github.com/micro/go-micro/codec"
+)
+
+type Codec struct {
+	Conn io.ReadWriteCloser
+}
+
+// Frame gives us the ability to define raw data to send over the pipes
+type Frame struct {
+	Data []byte
+}
+
+func (c *Codec) ReadHeader(m *codec.Message, t codec.MessageType) error {
+	return nil
+}
+
+func (c *Codec) ReadBody(b interface{}) error {
+	// read bytes
+	buf, err := ioutil.ReadAll(c.Conn)
+	if err != nil {
+		return err
+	}
+
+	switch b.(type) {
+	case *[]byte:
+		v := b.(*[]byte)
+		*v = buf
+	case *Frame:
+		v := b.(*Frame)
+		v.Data = buf
+	default:
+		return fmt.Errorf("failed to read body: %v is not type of *[]byte", b)
+	}
+
+	return nil
+}
+
+func (c *Codec) Write(m *codec.Message, b interface{}) error {
+	var v []byte
+	switch b.(type) {
+	case *Frame:
+		v = b.(*Frame).Data
+	case *[]byte:
+		ve := b.(*[]byte)
+		v = *ve
+	case []byte:
+		v = b.([]byte)
+	default:
+		return fmt.Errorf("failed to write: %v is not type of *[]byte or []byte", b)
+	}
+	_, err := c.Conn.Write(v)
+	return err
+}
+
+func (c *Codec) Close() error {
+	return c.Conn.Close()
+}
+
+func (c *Codec) String() string {
+	return "bytes"
+}
+
+func NewCodec(c io.ReadWriteCloser) codec.Codec {
+	return &Codec{
+		Conn: c,
+	}
+}

config/README.md

@@ -0,0 +1,29 @@
+# Config [![GoDoc](https://godoc.org/github.com/micro/go-micro/config?status.svg)](https://godoc.org/github.com/micro/go-micro/config)
+
+Go Config is a pluggable dynamic config library.
+
+Most config in applications are statically configured or include complex logic to load from multiple sources. 
+Go Config makes this easy, pluggable and mergeable. You'll never have to deal with config in the same way again.
+
+## Features
+
+- **Dynamic Loading** - Load configuration from multiple source as and when needed. Go Config manages watching config sources 
+in the background and automatically merges and updates an in memory view. 
+
+- **Pluggable Sources** - Choose from any number of sources to load and merge config. The backend source is abstracted away into 
+a standard format consumed internally and decoded via encoders. Sources can be env vars, flags, file, etcd, k8s configmap, etc.
+
+- **Mergeable Config** - If you specify multiple sources of config, regardless of format, they will be merged and presented in 
+a single view. This massively simplifies priority order loading and changes based on environment.
+
+- **Observe Changes** - Optionally watch the config for changes to specific values. Hot reload your app using Go Config's watcher. 
+You don't have to handle ad-hoc hup reloading or whatever else, just keep reading the config and watch for changes if you need 
+to be notified.
+
+- **Sane Defaults** - In case config loads badly or is completely wiped away for some unknown reason, you can specify fallback 
+values when accessing any config values directly. This ensures you'll always be reading some sane default in the event of a problem.
+
+## Getting Started
+
+For detailed information or architecture, installation and general usage see the [docs](https://micro.mu/docs/go-config.html)
+

config/config.go

@@ -0,0 +1,94 @@
+// Package config is an interface for dynamic configuration.
+package config
+
+import (
+	"context"
+
+	"github.com/micro/go-micro/config/loader"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/source"
+	"github.com/micro/go-micro/config/source/file"
+)
+
+// Config is an interface abstraction for dynamic configuration
+type Config interface {
+	// provide the reader.Values interface
+	reader.Values
+	// Stop the config loader/watcher
+	Close() error
+	// Load config sources
+	Load(source ...source.Source) error
+	// Force a source changeset sync
+	Sync() error
+	// Watch a value for changes
+	Watch(path ...string) (Watcher, error)
+}
+
+// Watcher is the config watcher
+type Watcher interface {
+	Next() (reader.Value, error)
+	Stop() error
+}
+
+type Options struct {
+	Loader loader.Loader
+	Reader reader.Reader
+	Source []source.Source
+
+	// for alternative data
+	Context context.Context
+}
+
+type Option func(o *Options)
+
+var (
+	// Default Config Manager
+	DefaultConfig = NewConfig()
+)
+
+// NewConfig returns new config
+func NewConfig(opts ...Option) Config {
+	return newConfig(opts...)
+}
+
+// Return config as raw json
+func Bytes() []byte {
+	return DefaultConfig.Bytes()
+}
+
+// Return config as a map
+func Map() map[string]interface{} {
+	return DefaultConfig.Map()
+}
+
+// Scan values to a go type
+func Scan(v interface{}) error {
+	return DefaultConfig.Scan(v)
+}
+
+// Force a source changeset sync
+func Sync() error {
+	return DefaultConfig.Sync()
+}
+
+// Get a value from the config
+func Get(path ...string) reader.Value {
+	return DefaultConfig.Get(path...)
+}
+
+// Load config sources
+func Load(source ...source.Source) error {
+	return DefaultConfig.Load(source...)
+}
+
+// Watch a value for changes
+func Watch(path ...string) (Watcher, error) {
+	return DefaultConfig.Watch(path...)
+}
+
+// LoadFile is short hand for creating a file source and loading it
+func LoadFile(path string) error {
+	return Load(file.NewSource(
+		file.WithPath(path),
+	))
+}

config/default.go

@@ -0,0 +1,253 @@
+package config
+
+import (
+	"bytes"
+	"sync"
+	"time"
+
+	"github.com/micro/go-micro/config/loader"
+	"github.com/micro/go-micro/config/loader/memory"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/reader/json"
+	"github.com/micro/go-micro/config/source"
+)
+
+type config struct {
+	exit chan bool
+	opts Options
+
+	sync.RWMutex
+	// the current snapshot
+	snap *loader.Snapshot
+	// the current values
+	vals reader.Values
+}
+
+type watcher struct {
+	lw    loader.Watcher
+	rd    reader.Reader
+	path  []string
+	value reader.Value
+}
+
+func newConfig(opts ...Option) Config {
+	options := Options{
+		Loader: memory.NewLoader(),
+		Reader: json.NewReader(),
+	}
+
+	for _, o := range opts {
+		o(&options)
+	}
+
+	options.Loader.Load(options.Source...)
+	snap, _ := options.Loader.Snapshot()
+	vals, _ := options.Reader.Values(snap.ChangeSet)
+
+	c := &config{
+		exit: make(chan bool),
+		opts: options,
+		snap: snap,
+		vals: vals,
+	}
+
+	go c.run()
+
+	return c
+}
+
+func (c *config) run() {
+	watch := func(w loader.Watcher) error {
+		for {
+			// get changeset
+			snap, err := w.Next()
+			if err != nil {
+				return err
+			}
+
+			c.Lock()
+
+			// save
+			c.snap = snap
+
+			// set values
+			c.vals, _ = c.opts.Reader.Values(snap.ChangeSet)
+
+			c.Unlock()
+		}
+	}
+
+	for {
+		w, err := c.opts.Loader.Watch()
+		if err != nil {
+			time.Sleep(time.Second)
+			continue
+		}
+
+		done := make(chan bool)
+
+		// the stop watch func
+		go func() {
+			select {
+			case <-done:
+			case <-c.exit:
+			}
+			w.Stop()
+		}()
+
+		// block watch
+		if err := watch(w); err != nil {
+			// do something better
+			time.Sleep(time.Second)
+		}
+
+		// close done chan
+		close(done)
+
+		// if the config is closed exit
+		select {
+		case <-c.exit:
+			return
+		default:
+		}
+	}
+}
+
+func (c *config) Map() map[string]interface{} {
+	c.RLock()
+	defer c.RUnlock()
+	return c.vals.Map()
+}
+
+func (c *config) Scan(v interface{}) error {
+	c.RLock()
+	defer c.RUnlock()
+	return c.vals.Scan(v)
+}
+
+// sync loads all the sources, calls the parser and updates the config
+func (c *config) Sync() error {
+	if err := c.opts.Loader.Sync(); err != nil {
+		return err
+	}
+
+	snap, err := c.opts.Loader.Snapshot()
+	if err != nil {
+		return err
+	}
+
+	c.Lock()
+	defer c.Unlock()
+
+	c.snap = snap
+	vals, err := c.opts.Reader.Values(snap.ChangeSet)
+	if err != nil {
+		return err
+	}
+	c.vals = vals
+
+	return nil
+}
+
+func (c *config) Close() error {
+	select {
+	case <-c.exit:
+		return nil
+	default:
+		close(c.exit)
+	}
+	return nil
+}
+
+func (c *config) Get(path ...string) reader.Value {
+	c.RLock()
+	defer c.RUnlock()
+
+	// did sync actually work?
+	if c.vals != nil {
+		return c.vals.Get(path...)
+	}
+
+	// no value
+	return newValue()
+}
+
+func (c *config) Bytes() []byte {
+	c.RLock()
+	defer c.RUnlock()
+
+	if c.vals == nil {
+		return []byte{}
+	}
+
+	return c.vals.Bytes()
+}
+
+func (c *config) Load(sources ...source.Source) error {
+	if err := c.opts.Loader.Load(sources...); err != nil {
+		return err
+	}
+
+	snap, err := c.opts.Loader.Snapshot()
+	if err != nil {
+		return err
+	}
+
+	c.Lock()
+	defer c.Unlock()
+
+	c.snap = snap
+	vals, err := c.opts.Reader.Values(snap.ChangeSet)
+	if err != nil {
+		return err
+	}
+	c.vals = vals
+
+	return nil
+}
+
+func (c *config) Watch(path ...string) (Watcher, error) {
+	value := c.Get(path...)
+
+	w, err := c.opts.Loader.Watch(path...)
+	if err != nil {
+		return nil, err
+	}
+
+	return &watcher{
+		lw:    w,
+		rd:    c.opts.Reader,
+		path:  path,
+		value: value,
+	}, nil
+}
+
+func (c *config) String() string {
+	return "config"
+}
+
+func (w *watcher) Next() (reader.Value, error) {
+	for {
+		s, err := w.lw.Next()
+		if err != nil {
+			return nil, err
+		}
+
+		// only process changes
+		if bytes.Equal(w.value.Bytes(), s.ChangeSet.Data) {
+			continue
+		}
+
+		v, err := w.rd.Values(s.ChangeSet)
+		if err != nil {
+			return nil, err
+		}
+
+		w.value = v.Get()
+		return w.value, nil
+	}
+}
+
+func (w *watcher) Stop() error {
+	return w.lw.Stop()
+}

config/default_test.go

@@ -0,0 +1,101 @@
+package config
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/micro/go-micro/config/source/file"
+)
+
+func createFileForTest(t *testing.T) *os.File {
+	data := []byte(`{"foo": "bar"}`)
+	path := filepath.Join(os.TempDir(), fmt.Sprintf("file.%d", time.Now().UnixNano()))
+	fh, err := os.Create(path)
+	if err != nil {
+		t.Error(err)
+	}
+	_, err = fh.Write(data)
+	if err != nil {
+		t.Error(err)
+	}
+
+	return fh
+}
+
+func TestLoadWithGoodFile(t *testing.T) {
+	fh := createFileForTest(t)
+	path := fh.Name()
+	defer func() {
+		fh.Close()
+		os.Remove(path)
+	}()
+
+	// Create new config
+	conf := NewConfig()
+	// Load file source
+	if err := conf.Load(file.NewSource(
+		file.WithPath(path),
+	)); err != nil {
+		t.Fatalf("Expected no error but got %v", err)
+	}
+}
+
+func TestLoadWithInvalidFile(t *testing.T) {
+	fh := createFileForTest(t)
+	path := fh.Name()
+	defer func() {
+		fh.Close()
+		os.Remove(path)
+	}()
+
+	// Create new config
+	conf := NewConfig()
+	// Load file source
+	err := conf.Load(file.NewSource(
+		file.WithPath(path),
+		file.WithPath("/i/do/not/exists.json"),
+	))
+
+	if err == nil {
+		t.Fatal("Expected error but none !")
+	}
+	if !strings.Contains(fmt.Sprintf("%v", err), "/i/do/not/exists.json") {
+		t.Fatalf("Expected error to contain the unexisting file but got %v", err)
+	}
+}
+
+func TestConsul(t *testing.T) {
+	/*consulSource := consul.NewSource(
+		// optionally specify consul address; default to localhost:8500
+		consul.WithAddress("131.150.38.111:8500"),
+		// optionally specify prefix; defaults to /micro/config
+		consul.WithPrefix("/project"),
+		// optionally strip the provided prefix from the keys, defaults to false
+		consul.StripPrefix(true),
+		consul.WithDatacenter("dc1"),
+		consul.WithToken("xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"),
+	)
+
+	// Create new config
+	conf := NewConfig()
+
+	// Load file source
+	err := conf.Load(consulSource)
+	if err != nil {
+		t.Error(err)
+		return
+	}
+
+	m := conf.Map()
+	t.Log("m: ", m)
+
+	v := conf.Get("project", "dc111", "port")
+
+	t.Log("v: ", v.Int(13))*/
+
+	t.Log("OK")
+}

config/encoder/encoder.go

@@ -0,0 +1,8 @@
+// Package encoder handles source encoding formats
+package encoder
+
+type Encoder interface {
+	Encode(interface{}) ([]byte, error)
+	Decode([]byte, interface{}) error
+	String() string
+}

config/encoder/hcl/hcl.go

@@ -0,0 +1,26 @@
+package hcl
+
+import (
+	"encoding/json"
+
+	"github.com/hashicorp/hcl"
+	"github.com/micro/go-micro/config/encoder"
+)
+
+type hclEncoder struct{}
+
+func (h hclEncoder) Encode(v interface{}) ([]byte, error) {
+	return json.Marshal(v)
+}
+
+func (h hclEncoder) Decode(d []byte, v interface{}) error {
+	return hcl.Unmarshal(d, v)
+}
+
+func (h hclEncoder) String() string {
+	return "hcl"
+}
+
+func NewEncoder() encoder.Encoder {
+	return hclEncoder{}
+}

config/encoder/json/json.go

@@ -0,0 +1,25 @@
+package json
+
+import (
+	"encoding/json"
+
+	"github.com/micro/go-micro/config/encoder"
+)
+
+type jsonEncoder struct{}
+
+func (j jsonEncoder) Encode(v interface{}) ([]byte, error) {
+	return json.Marshal(v)
+}
+
+func (j jsonEncoder) Decode(d []byte, v interface{}) error {
+	return json.Unmarshal(d, v)
+}
+
+func (j jsonEncoder) String() string {
+	return "json"
+}
+
+func NewEncoder() encoder.Encoder {
+	return jsonEncoder{}
+}

config/encoder/toml/toml.go

@@ -0,0 +1,32 @@
+package toml
+
+import (
+	"bytes"
+
+	"github.com/BurntSushi/toml"
+	"github.com/micro/go-micro/config/encoder"
+)
+
+type tomlEncoder struct{}
+
+func (t tomlEncoder) Encode(v interface{}) ([]byte, error) {
+	b := bytes.NewBuffer(nil)
+	defer b.Reset()
+	err := toml.NewEncoder(b).Encode(v)
+	if err != nil {
+		return nil, err
+	}
+	return b.Bytes(), nil
+}
+
+func (t tomlEncoder) Decode(d []byte, v interface{}) error {
+	return toml.Unmarshal(d, v)
+}
+
+func (t tomlEncoder) String() string {
+	return "toml"
+}
+
+func NewEncoder() encoder.Encoder {
+	return tomlEncoder{}
+}

config/encoder/xml/xml.go

@@ -0,0 +1,25 @@
+package xml
+
+import (
+	"encoding/xml"
+
+	"github.com/micro/go-micro/config/encoder"
+)
+
+type xmlEncoder struct{}
+
+func (x xmlEncoder) Encode(v interface{}) ([]byte, error) {
+	return xml.Marshal(v)
+}
+
+func (x xmlEncoder) Decode(d []byte, v interface{}) error {
+	return xml.Unmarshal(d, v)
+}
+
+func (x xmlEncoder) String() string {
+	return "xml"
+}
+
+func NewEncoder() encoder.Encoder {
+	return xmlEncoder{}
+}

config/encoder/yaml/yaml.go

@@ -0,0 +1,24 @@
+package yaml
+
+import (
+	"github.com/ghodss/yaml"
+	"github.com/micro/go-micro/config/encoder"
+)
+
+type yamlEncoder struct{}
+
+func (y yamlEncoder) Encode(v interface{}) ([]byte, error) {
+	return yaml.Marshal(v)
+}
+
+func (y yamlEncoder) Decode(d []byte, v interface{}) error {
+	return yaml.Unmarshal(d, v)
+}
+
+func (y yamlEncoder) String() string {
+	return "yaml"
+}
+
+func NewEncoder() encoder.Encoder {
+	return yamlEncoder{}
+}

config/issue18_test.go

@@ -0,0 +1,60 @@
+package config
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"github.com/micro/go-micro/config/source/env"
+	"github.com/micro/go-micro/config/source/file"
+)
+
+func createFileForIssue18(t *testing.T, content string) *os.File {
+	data := []byte(content)
+	path := filepath.Join(os.TempDir(), fmt.Sprintf("file.%d", time.Now().UnixNano()))
+	fh, err := os.Create(path)
+	if err != nil {
+		t.Error(err)
+	}
+	_, err = fh.Write(data)
+	if err != nil {
+		t.Error(err)
+	}
+
+	return fh
+}
+
+func TestIssue18(t *testing.T) {
+	fh := createFileForIssue18(t, `{
+  "amqp": {
+    "host": "rabbit.platform",
+    "port": 80
+  },
+  "handler": {
+    "exchange": "springCloudBus"
+  }
+}`)
+	path := fh.Name()
+	defer func() {
+		fh.Close()
+		os.Remove(path)
+	}()
+	os.Setenv("AMQP_HOST", "rabbit.testing.com")
+
+	conf := NewConfig()
+	conf.Load(
+		file.NewSource(
+			file.WithPath(path),
+		),
+		env.NewSource(),
+	)
+
+	actualHost := conf.Get("amqp", "host").String("backup")
+	if actualHost != "rabbit.testing.com" {
+		t.Fatalf("Expected %v but got %v",
+			"rabbit.testing.com",
+			actualHost)
+	}
+}

config/loader/loader.go

@@ -0,0 +1,63 @@
+// package loader manages loading from multiple sources
+package loader
+
+import (
+	"context"
+
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/source"
+)
+
+// Loader manages loading sources
+type Loader interface {
+	// Stop the loader
+	Close() error
+	// Load the sources
+	Load(...source.Source) error
+	// A Snapshot of loaded config
+	Snapshot() (*Snapshot, error)
+	// Force sync of sources
+	Sync() error
+	// Watch for changes
+	Watch(...string) (Watcher, error)
+	// Name of loader
+	String() string
+}
+
+// Watcher lets you watch sources and returns a merged ChangeSet
+type Watcher interface {
+	// First call to next may return the current Snapshot
+	// If you are watching a path then only the data from
+	// that path is returned.
+	Next() (*Snapshot, error)
+	// Stop watching for changes
+	Stop() error
+}
+
+// Snapshot is a merged ChangeSet
+type Snapshot struct {
+	// The merged ChangeSet
+	ChangeSet *source.ChangeSet
+	// Deterministic and comparable version of the snapshot
+	Version string
+}
+
+type Options struct {
+	Reader reader.Reader
+	Source []source.Source
+
+	// for alternative data
+	Context context.Context
+}
+
+type Option func(o *Options)
+
+// Copy snapshot
+func Copy(s *Snapshot) *Snapshot {
+	cs := *(s.ChangeSet)
+
+	return &Snapshot{
+		ChangeSet: &cs,
+		Version:   s.Version,
+	}
+}

config/loader/memory/memory.go

@@ -0,0 +1,415 @@
+package memory
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/micro/go-micro/config/loader"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/reader/json"
+	"github.com/micro/go-micro/config/source"
+)
+
+type memory struct {
+	exit chan bool
+	opts loader.Options
+
+	sync.RWMutex
+	// the current snapshot
+	snap *loader.Snapshot
+	// the current values
+	vals reader.Values
+	// all the sets
+	sets []*source.ChangeSet
+	// all the sources
+	sources []source.Source
+
+	idx      int
+	watchers map[int]*watcher
+}
+
+type watcher struct {
+	exit    chan bool
+	path    []string
+	value   reader.Value
+	reader  reader.Reader
+	updates chan reader.Value
+}
+
+func (m *memory) watch(idx int, s source.Source) {
+	m.Lock()
+	m.sets = append(m.sets, &source.ChangeSet{Source: s.String()})
+	m.Unlock()
+
+	// watches a source for changes
+	watch := func(idx int, s source.Watcher) error {
+		for {
+			// get changeset
+			cs, err := s.Next()
+			if err != nil {
+				return err
+			}
+
+			m.Lock()
+
+			// save
+			m.sets[idx] = cs
+
+			// merge sets
+			set, err := m.opts.Reader.Merge(m.sets...)
+			if err != nil {
+				m.Unlock()
+				return err
+			}
+
+			// set values
+			m.vals, _ = m.opts.Reader.Values(set)
+			m.snap = &loader.Snapshot{
+				ChangeSet: set,
+				Version:   fmt.Sprintf("%d", time.Now().Unix()),
+			}
+			m.Unlock()
+
+			// send watch updates
+			m.update()
+		}
+	}
+
+	for {
+		// watch the source
+		w, err := s.Watch()
+		if err != nil {
+			time.Sleep(time.Second)
+			continue
+		}
+
+		done := make(chan bool)
+
+		// the stop watch func
+		go func() {
+			select {
+			case <-done:
+			case <-m.exit:
+			}
+			w.Stop()
+		}()
+
+		// block watch
+		if err := watch(idx, w); err != nil {
+			// do something better
+			time.Sleep(time.Second)
+		}
+
+		// close done chan
+		close(done)
+
+		// if the config is closed exit
+		select {
+		case <-m.exit:
+			return
+		default:
+		}
+	}
+}
+
+func (m *memory) loaded() bool {
+	var loaded bool
+	m.RLock()
+	if m.vals != nil {
+		loaded = true
+	}
+	m.RUnlock()
+	return loaded
+}
+
+// reload reads the sets and creates new values
+func (m *memory) reload() error {
+	m.Lock()
+
+	// merge sets
+	set, err := m.opts.Reader.Merge(m.sets...)
+	if err != nil {
+		m.Unlock()
+		return err
+	}
+
+	// set values
+	m.vals, _ = m.opts.Reader.Values(set)
+	m.snap = &loader.Snapshot{
+		ChangeSet: set,
+		Version:   fmt.Sprintf("%d", time.Now().Unix()),
+	}
+
+	m.Unlock()
+
+	// update watchers
+	m.update()
+
+	return nil
+}
+
+func (m *memory) update() {
+	var watchers []*watcher
+
+	m.RLock()
+	for _, w := range m.watchers {
+		watchers = append(watchers, w)
+	}
+	m.RUnlock()
+
+	for _, w := range watchers {
+		select {
+		case w.updates <- m.vals.Get(w.path...):
+		default:
+		}
+	}
+}
+
+// Snapshot returns a snapshot of the current loaded config
+func (m *memory) Snapshot() (*loader.Snapshot, error) {
+	if m.loaded() {
+		m.RLock()
+		snap := loader.Copy(m.snap)
+		m.RUnlock()
+		return snap, nil
+	}
+
+	// not loaded, sync
+	if err := m.Sync(); err != nil {
+		return nil, err
+	}
+
+	// make copy
+	m.RLock()
+	snap := loader.Copy(m.snap)
+	m.RUnlock()
+
+	return snap, nil
+}
+
+// Sync loads all the sources, calls the parser and updates the config
+func (m *memory) Sync() error {
+	var sets []*source.ChangeSet
+
+	m.Lock()
+
+	// read the source
+	var gerr []string
+
+	for _, source := range m.sources {
+		ch, err := source.Read()
+		if err != nil {
+			gerr = append(gerr, err.Error())
+			continue
+		}
+		sets = append(sets, ch)
+	}
+
+	// merge sets
+	set, err := m.opts.Reader.Merge(sets...)
+	if err != nil {
+		m.Unlock()
+		return err
+	}
+
+	// set values
+	vals, err := m.opts.Reader.Values(set)
+	if err != nil {
+		m.Unlock()
+		return err
+	}
+	m.vals = vals
+	m.snap = &loader.Snapshot{
+		ChangeSet: set,
+		Version:   fmt.Sprintf("%d", time.Now().Unix()),
+	}
+
+	m.Unlock()
+
+	// update watchers
+	m.update()
+
+	if len(gerr) > 0 {
+		return fmt.Errorf("source loading errors: %s", strings.Join(gerr, "\n"))
+	}
+
+	return nil
+}
+
+func (m *memory) Close() error {
+	select {
+	case <-m.exit:
+		return nil
+	default:
+		close(m.exit)
+	}
+	return nil
+}
+
+func (m *memory) Get(path ...string) (reader.Value, error) {
+	if !m.loaded() {
+		if err := m.Sync(); err != nil {
+			return nil, err
+		}
+	}
+
+	m.Lock()
+	defer m.Unlock()
+
+	// did sync actually work?
+	if m.vals != nil {
+		return m.vals.Get(path...), nil
+	}
+
+	// assuming vals is nil
+	// create new vals
+
+	ch := m.snap.ChangeSet
+
+	// we are truly screwed, trying to load in a hacked way
+	v, err := m.opts.Reader.Values(ch)
+	if err != nil {
+		return nil, err
+	}
+
+	// lets set it just because
+	m.vals = v
+
+	if m.vals != nil {
+		return m.vals.Get(path...), nil
+	}
+
+	// ok we're going hardcore now
+	return nil, errors.New("no values")
+}
+
+func (m *memory) Load(sources ...source.Source) error {
+	var gerrors []string
+
+	for _, source := range sources {
+		set, err := source.Read()
+		if err != nil {
+			gerrors = append(gerrors,
+				fmt.Sprintf("error loading source %s: %v",
+					source,
+					err))
+			// continue processing
+			continue
+		}
+		m.Lock()
+		m.sources = append(m.sources, source)
+		m.sets = append(m.sets, set)
+		idx := len(m.sets) - 1
+		m.Unlock()
+		go m.watch(idx, source)
+	}
+
+	if err := m.reload(); err != nil {
+		gerrors = append(gerrors, err.Error())
+	}
+
+	// Return errors
+	if len(gerrors) != 0 {
+		return errors.New(strings.Join(gerrors, "\n"))
+	}
+	return nil
+}
+
+func (m *memory) Watch(path ...string) (loader.Watcher, error) {
+	value, err := m.Get(path...)
+	if err != nil {
+		return nil, err
+	}
+
+	m.Lock()
+
+	w := &watcher{
+		exit:    make(chan bool),
+		path:    path,
+		value:   value,
+		reader:  m.opts.Reader,
+		updates: make(chan reader.Value, 1),
+	}
+
+	id := m.idx
+	m.watchers[id] = w
+	m.idx++
+
+	m.Unlock()
+
+	go func() {
+		<-w.exit
+		m.Lock()
+		delete(m.watchers, id)
+		m.Unlock()
+	}()
+
+	return w, nil
+}
+
+func (m *memory) String() string {
+	return "memory"
+}
+
+func (w *watcher) Next() (*loader.Snapshot, error) {
+	for {
+		select {
+		case <-w.exit:
+			return nil, errors.New("watcher stopped")
+		case v := <-w.updates:
+			if bytes.Equal(w.value.Bytes(), v.Bytes()) {
+				continue
+			}
+			w.value = v
+
+			cs := &source.ChangeSet{
+				Data:      v.Bytes(),
+				Format:    w.reader.String(),
+				Source:    "memory",
+				Timestamp: time.Now(),
+			}
+			cs.Sum()
+
+			return &loader.Snapshot{
+				ChangeSet: cs,
+				Version:   fmt.Sprintf("%d", time.Now().Unix()),
+			}, nil
+		}
+	}
+}
+
+func (w *watcher) Stop() error {
+	select {
+	case <-w.exit:
+	default:
+		close(w.exit)
+	}
+	return nil
+}
+
+func NewLoader(opts ...loader.Option) loader.Loader {
+	options := loader.Options{
+		Reader: json.NewReader(),
+	}
+
+	for _, o := range opts {
+		o(&options)
+	}
+
+	m := &memory{
+		exit:     make(chan bool),
+		opts:     options,
+		watchers: make(map[int]*watcher),
+		sources:  options.Source,
+	}
+
+	for i, s := range options.Source {
+		go m.watch(i, s)
+	}
+
+	return m
+}

config/loader/memory/options.go

@@ -0,0 +1,21 @@
+package memory
+
+import (
+	"github.com/micro/go-micro/config/loader"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/source"
+)
+
+// WithSource appends a source to list of sources
+func WithSource(s source.Source) loader.Option {
+	return func(o *loader.Options) {
+		o.Source = append(o.Source, s)
+	}
+}
+
+// WithReader sets the config reader
+func WithReader(r reader.Reader) loader.Option {
+	return func(o *loader.Options) {
+		o.Reader = r
+	}
+}

config/options.go

@@ -0,0 +1,28 @@
+package config
+
+import (
+	"github.com/micro/go-micro/config/loader"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/source"
+)
+
+// WithLoader sets the loader for manager config
+func WithLoader(l loader.Loader) Option {
+	return func(o *Options) {
+		o.Loader = l
+	}
+}
+
+// WithSource appends a source to list of sources
+func WithSource(s source.Source) Option {
+	return func(o *Options) {
+		o.Source = append(o.Source, s)
+	}
+}
+
+// WithReader sets the config reader
+func WithReader(r reader.Reader) Option {
+	return func(o *Options) {
+		o.Reader = r
+	}
+}

config/reader/json/json.go

@@ -0,0 +1,83 @@
+package json
+
+import (
+	"errors"
+	"time"
+
+	"github.com/imdario/mergo"
+	"github.com/micro/go-micro/config/encoder"
+	"github.com/micro/go-micro/config/encoder/json"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/source"
+)
+
+type jsonReader struct {
+	opts reader.Options
+	json encoder.Encoder
+}
+
+func (j *jsonReader) Merge(changes ...*source.ChangeSet) (*source.ChangeSet, error) {
+	var merged map[string]interface{}
+
+	for _, m := range changes {
+		if m == nil {
+			continue
+		}
+
+		if len(m.Data) == 0 {
+			continue
+		}
+
+		codec, ok := j.opts.Encoding[m.Format]
+		if !ok {
+			// fallback
+			codec = j.json
+		}
+
+		var data map[string]interface{}
+		if err := codec.Decode(m.Data, &data); err != nil {
+			return nil, err
+		}
+		if err := mergo.Map(&merged, data, mergo.WithOverride); err != nil {
+			return nil, err
+		}
+	}
+
+	b, err := j.json.Encode(merged)
+	if err != nil {
+		return nil, err
+	}
+
+	cs := &source.ChangeSet{
+		Timestamp: time.Now(),
+		Data:      b,
+		Source:    "json",
+		Format:    j.json.String(),
+	}
+	cs.Checksum = cs.Sum()
+
+	return cs, nil
+}
+
+func (j *jsonReader) Values(ch *source.ChangeSet) (reader.Values, error) {
+	if ch == nil {
+		return nil, errors.New("changeset is nil")
+	}
+	if ch.Format != "json" {
+		return nil, errors.New("unsupported format")
+	}
+	return newValues(ch)
+}
+
+func (j *jsonReader) String() string {
+	return "json"
+}
+
+// NewReader creates a json reader
+func NewReader(opts ...reader.Option) reader.Reader {
+	options := reader.NewOptions(opts...)
+	return &jsonReader{
+		json: json.NewEncoder(),
+		opts: options,
+	}
+}

config/reader/json/json_test.go

@@ -0,0 +1,43 @@
+package json
+
+import (
+	"testing"
+
+	"github.com/micro/go-micro/config/source"
+)
+
+func TestReader(t *testing.T) {
+	data := []byte(`{"foo": "bar", "baz": {"bar": "cat"}}`)
+
+	testData := []struct {
+		path  []string
+		value string
+	}{
+		{
+			[]string{"foo"},
+			"bar",
+		},
+		{
+			[]string{"baz", "bar"},
+			"cat",
+		},
+	}
+
+	r := NewReader()
+
+	c, err := r.Merge(&source.ChangeSet{Data: data}, &source.ChangeSet{})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	values, err := r.Values(c)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for _, test := range testData {
+		if v := values.Get(test.path...).String(""); v != test.value {
+			t.Fatalf("Expected %s got %s for path %v", test.value, v, test.path)
+		}
+	}
+}

config/reader/json/values.go

@@ -0,0 +1,208 @@
+package json
+
+import (
+	"encoding/json"
+	"fmt"
+	"strconv"
+	"strings"
+	"time"
+
+	simple "github.com/bitly/go-simplejson"
+	"github.com/micro/go-micro/config/reader"
+	"github.com/micro/go-micro/config/source"
+)
+
+type jsonValues struct {
+	ch *source.ChangeSet
+	sj *simple.Json
+}
+
+type jsonValue struct {
+	*simple.Json
+}
+
+func newValues(ch *source.ChangeSet) (reader.Values, error) {
+	sj := simple.New()
+	data, _ := reader.ReplaceEnvVars(ch.Data)
+	if err := sj.UnmarshalJSON(data); err != nil {
+		sj.SetPath(nil, string(ch.Data))
+	}
+	return &jsonValues{ch, sj}, nil
+}
+
+func newValue(s *simple.Json) reader.Value {
+	if s == nil {
+		s = simple.New()
+	}
+	return &jsonValue{s}
+}
+
+func (j *jsonValues) Get(path ...string) reader.Value {
+	return &jsonValue{j.sj.GetPath(path...)}
+}
+
+func (j *jsonValues) Del(path ...string) {
+	// delete the tree?
+	if len(path) == 0 {
+		j.sj = simple.New()
+		return
+	}
+
+	if len(path) == 1 {
+		j.sj.Del(path[0])
+		return
+	}
+
+	vals := j.sj.GetPath(path[:len(path)-1]...)
+	vals.Del(path[len(path)-1])
+	j.sj.SetPath(path[:len(path)-1], vals.Interface())
+	return
+}
+
+func (j *jsonValues) Set(val interface{}, path ...string) {
+	j.sj.SetPath(path, val)
+}
+
+func (j *jsonValues) Bytes() []byte {
+	b, _ := j.sj.MarshalJSON()
+	return b
+}
+
+func (j *jsonValues) Map() map[string]interface{} {
+	m, _ := j.sj.Map()
+	return m
+}
+
+func (j *jsonValues) Scan(v interface{}) error {
+	b, err := j.sj.MarshalJSON()
+	if err != nil {
+		return err
+	}
+	return json.Unmarshal(b, v)
+}
+
+func (j *jsonValues) String() string {
+	return "json"
+}
+
+func (j *jsonValue) Bool(def bool) bool {
+	b, err := j.Json.Bool()
+	if err == nil {
+		return b
+	}
+
+	str, ok := j.Interface().(string)
+	if !ok {
+		return def
+	}
+
+	b, err = strconv.ParseBool(str)
+	if err != nil {
+		return def
+	}
+
+	return b
+}
+
+func (j *jsonValue) Int(def int) int {
+	i, err := j.Json.Int()
+	if err == nil {
+		return i
+	}
+
+	str, ok := j.Interface().(string)
+	if !ok {
+		return def
+	}
+
+	i, err = strconv.Atoi(str)
+	if err != nil {
+		return def
+	}
+
+	return i
+}
+
+func (j *jsonValue) String(def string) string {
+	return j.Json.MustString(def)
+}
+
+func (j *jsonValue) Float64(def float64) float64 {
+	f, err := j.Json.Float64()
+	if err == nil {
+		return f
+	}
+
+	str, ok := j.Interface().(string)
+	if !ok {
+		return def
+	}
+
+	f, err = strconv.ParseFloat(str, 64)
+	if err != nil {
+		return def
+	}
+
+	return f
+}
+
+func (j *jsonValue) Duration(def time.Duration) time.Duration {
+	v, err := j.Json.String()
+	if err != nil {
+		return def
+	}
+
+	value, err := time.ParseDuration(v)
+	if err != nil {
+		return def
+	}
+
+	return value
+}
+
+func (j *jsonValue) StringSlice(def []string) []string {
+	v, err := j.Json.String()
+	if err == nil {
+		sl := strings.Split(v, ",")
+		if len(sl) > 1 {
+			return sl
+		}
+	}
+	return j.Json.MustStringArray(def)
+}
+
+func (j *jsonValue) StringMap(def map[string]string) map[string]string {
+	m, err := j.Json.Map()
+	if err != nil {
+		return def
+	}
+
+	res := map[string]string{}
+
+	for k, v := range m {
+		res[k] = fmt.Sprintf("%v", v)
+	}
+
+	return res
+}
+
+func (j *jsonValue) Scan(v interface{}) error {
+	b, err := j.Json.MarshalJSON()
+	if err != nil {
+		return err
+	}
+	return json.Unmarshal(b, v)
+}
+
+func (j *jsonValue) Bytes() []byte {
+	b, err := j.Json.Bytes()
+	if err != nil {
+		// try return marshalled
+		b, err = j.Json.MarshalJSON()
+		if err != nil {
+			return []byte{}
+		}
+		return b
+	}
+	return b
+}

config/reader/json/values_test.go

@@ -0,0 +1,39 @@
+package json
+
+import (
+	"testing"
+
+	"github.com/micro/go-micro/config/source"
+)
+
+func TestValues(t *testing.T) {
+	data := []byte(`{"foo": "bar", "baz": {"bar": "cat"}}`)
+
+	testData := []struct {
+		path  []string
+		value string
+	}{
+		{
+			[]string{"foo"},
+			"bar",
+		},
+		{
+			[]string{"baz", "bar"},
+			"cat",
+		},
+	}
+
+	values, err := newValues(&source.ChangeSet{
+		Data: data,
+	})
+
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for _, test := range testData {
+		if v := values.Get(test.path...).String(""); v != test.value {
+			t.Fatalf("Expected %s got %s for path %v", test.value, v, test.path)
+		}
+	}
+}

config/reader/options.go

@@ -0,0 +1,42 @@
+package reader
+
+import (
+	"github.com/micro/go-micro/config/encoder"
+	"github.com/micro/go-micro/config/encoder/hcl"
+	"github.com/micro/go-micro/config/encoder/json"
+	"github.com/micro/go-micro/config/encoder/toml"
+	"github.com/micro/go-micro/config/encoder/xml"
+	"github.com/micro/go-micro/config/encoder/yaml"
+)
+
+type Options struct {
+	Encoding map[string]encoder.Encoder
+}
+
+type Option func(o *Options)
+
+func NewOptions(opts ...Option) Options {
+	options := Options{
+		Encoding: map[string]encoder.Encoder{
+			"json": json.NewEncoder(),
+			"yaml": yaml.NewEncoder(),
+			"toml": toml.NewEncoder(),
+			"xml":  xml.NewEncoder(),
+			"hcl":  hcl.NewEncoder(),
+			"yml":  yaml.NewEncoder(),
+		},
+	}
+	for _, o := range opts {
+		o(&options)
+	}
+	return options
+}
+
+func WithEncoder(e encoder.Encoder) Option {
+	return func(o *Options) {
+		if o.Encoding == nil {
+			o.Encoding = make(map[string]encoder.Encoder)
+		}
+		o.Encoding[e.String()] = e
+	}
+}

config/reader/preprocessor.go

@@ -0,0 +1,23 @@
+package reader
+
+import (
+	"os"
+	"regexp"
+)
+
+func ReplaceEnvVars(raw []byte) ([]byte, error) {
+	re := regexp.MustCompile(`\$\{([A-Za-z0-9_]+)\}`)
+	if re.Match(raw) {
+		dataS := string(raw)
+		res := re.ReplaceAllStringFunc(dataS, replaceEnvVars)
+		return []byte(res), nil
+	} else {
+		return raw, nil
+	}
+}
+
+func replaceEnvVars(element string) string {
+	v := element[2 : len(element)-1]
+	el := os.Getenv(v)
+	return el
+}

config/reader/preprocessor_test.go

@@ -0,0 +1,73 @@
+package reader
+
+import (
+	"os"
+	"strings"
+	"testing"
+)
+
+func TestReplaceEnvVars(t *testing.T) {
+	os.Setenv("myBar", "cat")
+	os.Setenv("MYBAR", "cat")
+	os.Setenv("my_Bar", "cat")
+	os.Setenv("myBar_", "cat")
+
+	testData := []struct {
+		expected string
+		data     []byte
+	}{
+		// Right use cases
+		{
+			`{"foo": "bar", "baz": {"bar": "cat"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${myBar}"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "cat"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${MYBAR}"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "cat"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${my_Bar}"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "cat"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${myBar_}"}}`),
+		},
+		// Wrong use cases
+		{
+			`{"foo": "bar", "baz": {"bar": "${myBar-}"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${myBar-}"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "${}"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${}"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "$sss}"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "$sss}"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "${sss"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "${sss"}}`),
+		},
+		{
+			`{"foo": "bar", "baz": {"bar": "{something}"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "{something}"}}`),
+		},
+		// Use cases without replace env vars
+		{
+			`{"foo": "bar", "baz": {"bar": "cat"}}`,
+			[]byte(`{"foo": "bar", "baz": {"bar": "cat"}}`),
+		},
+	}
+
+	for _, test := range testData {
+		res, err := ReplaceEnvVars(test.data)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if strings.Compare(test.expected, string(res)) != 0 {
+			t.Fatalf("Expected %s got %s", test.expected, res)
+		}
+	}
+}

config/reader/reader.go

@@ -0,0 +1,36 @@
+// Package reader parses change sets and provides config values
+package reader
+
+import (
+	"time"
+
+	"github.com/micro/go-micro/config/source"
+)
+
+// Reader is an interface for merging changesets
+type Reader interface {
+	Merge(...*source.ChangeSet) (*source.ChangeSet, error)
+	Values(*source.ChangeSet) (Values, error)
+	String() string
+}
+
+// Values is returned by the reader
+type Values interface {
+	Bytes() []byte
+	Get(path ...string) Value
+	Map() map[string]interface{}
+	Scan(v interface{}) error
+}
+
+// Value represents a value of any type
+type Value interface {
+	Bool(def bool) bool
+	Int(def int) int
+	String(def string) string
+	Float64(def float64) float64
+	Duration(def time.Duration) time.Duration
+	StringSlice(def []string) []string
+	StringMap(def map[string]string) map[string]string
+	Scan(val interface{}) error
+	Bytes() []byte
+}

config/source/changeset.go

@@ -0,0 +1,13 @@
+package source
+
+import (
+	"crypto/md5"
+	"fmt"
+)
+
+// Sum returns the md5 checksum of the ChangeSet data
+func (c *ChangeSet) Sum() string {
+	h := md5.New()
+	h.Write(c.Data)
+	return fmt.Sprintf("%x", h.Sum(nil))
+}

config/source/cli/README.md

@@ -0,0 +1,71 @@
+# cli Source
+
+The cli source reads config from parsed flags via a cli.Context.
+
+## Format
+
+We expect the use of the `micro/cli` package. Upper case flags will be lower cased. Dashes will be used as delimiters for nesting.
+
+### Example
+
+```go
+micro.Flags(
+    cli.StringFlag{
+        Name: "database-address",
+        Value: "127.0.0.1",
+        Usage: "the db address",
+    },
+    cli.IntFlag{
+        Name: "database-port",
+        Value: 3306,
+        Usage: "the db port",
+    },
+)
+```
+
+Becomes
+
+```json
+{
+    "database": {
+        "address": "127.0.0.1",
+        "port": 3306
+    }
+}
+```
+
+## New and Load Source
+
+Because a cli.Context is needed to retrieve the flags and their values, it is recommended to build your source from within a cli.Action.
+
+```go
+
+func main() {
+    // New Service
+    service := micro.NewService(
+        micro.Name("example"),
+        micro.Flags(
+            cli.StringFlag{
+                Name: "database-address",
+                Value: "127.0.0.1",
+                Usage: "the db address",
+            },
+        ),
+    )
+
+    var clisrc source.Source
+
+    service.Init(
+        micro.Action(func(c *cli.Context) {
+            clisrc = cli.NewSource(
+                cli.Context(c),
+	    )
+            // Alternatively, just setup your config right here
+        }),
+    )
+    
+    // ... Load and use that source ...
+    conf := config.NewConfig()
+    conf.Load(clisrc)
+}
+```

config/source/cli/cli.go

@@ -0,0 +1,146 @@
+package cli
+
+import (
+	"flag"
+	"io/ioutil"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/imdario/mergo"
+	"github.com/micro/cli"
+	"github.com/micro/go-micro/cmd"
+	"github.com/micro/go-micro/config/source"
+)
+
+type cliSource struct {
+	opts source.Options
+	ctx  *cli.Context
+}
+
+func (c *cliSource) Read() (*source.ChangeSet, error) {
+	var changes map[string]interface{}
+
+	for _, name := range c.ctx.GlobalFlagNames() {
+		tmp := toEntry(name, c.ctx.GlobalGeneric(name))
+		mergo.Map(&changes, tmp) // need to sort error handling
+	}
+
+	for _, name := range c.ctx.FlagNames() {
+		tmp := toEntry(name, c.ctx.Generic(name))
+		mergo.Map(&changes, tmp) // need to sort error handling
+	}
+
+	b, err := c.opts.Encoder.Encode(changes)
+	if err != nil {
+		return nil, err
+	}
+
+	cs := &source.ChangeSet{
+		Format:    c.opts.Encoder.String(),
+		Data:      b,
+		Timestamp: time.Now(),
+		Source:    c.String(),
+	}
+	cs.Checksum = cs.Sum()
+
+	return cs, nil
+}
+
+func toEntry(name string, v interface{}) map[string]interface{} {
+	n := strings.ToLower(name)
+	keys := strings.FieldsFunc(n, split)
+	reverse(keys)
+	tmp := make(map[string]interface{})
+	for i, k := range keys {
+		if i == 0 {
+			tmp[k] = v
+			continue
+		}
+
+		tmp = map[string]interface{}{k: tmp}
+	}
+	return tmp
+}
+
+func reverse(ss []string) {
+	for i := len(ss)/2 - 1; i >= 0; i-- {
+		opp := len(ss) - 1 - i
+		ss[i], ss[opp] = ss[opp], ss[i]
+	}
+}
+
+func split(r rune) bool {
+	return r == '-' || r == '_'
+}
+
+func (c *cliSource) Watch() (source.Watcher, error) {
+	return source.NewNoopWatcher()
+}
+
+func (c *cliSource) String() string {
+	return "cli"
+}
+
+// NewSource returns a config source for integrating parsed flags from a micro/cli.Context.
+// Hyphens are delimiters for nesting, and all keys are lowercased. The assumption is that
+// command line flags have already been parsed.
+//
+// Example:
+//      cli.StringFlag{Name: "db-host"},
+//
+//
+//      {
+//          "database": {
+//              "host": "localhost"
+//          }
+//      }
+func NewSource(opts ...source.Option) source.Source {
+	options := source.NewOptions(opts...)
+
+	var ctx *cli.Context
+
+	c, ok := options.Context.Value(contextKey{}).(*cli.Context)
+	if ok {
+		ctx = c
+	}
+
+	// no context
+	if ctx == nil {
+		// get the default app/flags
+		app := cmd.App()
+		flags := app.Flags
+
+		// create flagset
+		set := flag.NewFlagSet(app.Name, flag.ContinueOnError)
+
+		// apply flags to set
+		for _, f := range flags {
+			f.Apply(set)
+		}
+
+		// parse flags
+		set.SetOutput(ioutil.Discard)
+		set.Parse(os.Args[1:])
+
+		// normalise flags
+		normalizeFlags(app.Flags, set)
+
+		// create context
+		ctx = cli.NewContext(app, set, nil)
+	}
+
+	return &cliSource{
+		ctx:  ctx,
+		opts: options,
+	}
+}
+
+// WithContext returns a new source with the context specified.
+// The assumption is that Context is retrieved within an app.Action function.
+func WithContext(ctx *cli.Context, opts ...source.Option) source.Source {
+	return &cliSource{
+		ctx:  ctx,
+		opts: source.NewOptions(opts...),
+	}
+}

config/source/cli/cli_test.go

@@ -0,0 +1,65 @@
+package cli
+
+import (
+	"encoding/json"
+	"os"
+	"testing"
+
+	"github.com/micro/cli"
+	"github.com/micro/go-micro/cmd"
+	"github.com/micro/go-micro/config/source"
+)
+
+func test(t *testing.T, withContext bool) {
+	var src source.Source
+
+	// setup app
+	app := cmd.App()
+	app.Name = "testapp"
+	app.Flags = []cli.Flag{
+		cli.StringFlag{Name: "db-host"},
+	}
+
+	// with context
+	if withContext {
+		// set action
+		app.Action = func(c *cli.Context) {
+			src = WithContext(c)
+		}
+
+		// run app
+		app.Run([]string{"run", "-db-host", "localhost"})
+		// no context
+	} else {
+		// set args
+		os.Args = []string{"run", "-db-host", "localhost"}
+		src = NewSource()
+	}
+
+	// test config
+	c, err := src.Read()
+	if err != nil {
+		t.Error(err)
+	}
+
+	var actual map[string]interface{}
+	if err := json.Unmarshal(c.Data, &actual); err != nil {
+		t.Error(err)
+	}
+
+	actualDB := actual["db"].(map[string]interface{})
+	if actualDB["host"] != "localhost" {
+		t.Errorf("expected localhost, got %v", actualDB["name"])
+	}
+
+}
+
+func TestCliSource(t *testing.T) {
+	// without context
+	test(t, false)
+}
+
+func TestCliSourceWithContext(t *testing.T) {
+	// with context
+	test(t, true)
+}

config/source/cli/options.go

@@ -0,0 +1,20 @@
+package cli
+
+import (
+	"context"
+
+	"github.com/micro/cli"
+	"github.com/micro/go-micro/config/source"
+)
+
+type contextKey struct{}
+
+// Context sets the cli context
+func Context(c *cli.Context) source.Option {
+	return func(o *source.Options) {
+		if o.Context == nil {
+			o.Context = context.Background()
+		}
+		o.Context = context.WithValue(o.Context, contextKey{}, c)
+	}
+}

config/source/cli/util.go

@@ -0,0 +1,50 @@
+package cli
+
+import (
+	"errors"
+	"flag"
+	"strings"
+
+	"github.com/micro/cli"
+)
+
+func copyFlag(name string, ff *flag.Flag, set *flag.FlagSet) {
+	switch ff.Value.(type) {
+	case *cli.StringSlice:
+	default:
+		set.Set(name, ff.Value.String())
+	}
+}
+
+func normalizeFlags(flags []cli.Flag, set *flag.FlagSet) error {
+	visited := make(map[string]bool)
+	set.Visit(func(f *flag.Flag) {
+		visited[f.Name] = true
+	})
+	for _, f := range flags {
+		parts := strings.Split(f.GetName(), ",")
+		if len(parts) == 1 {
+			continue
+		}
+		var ff *flag.Flag
+		for _, name := range parts {
+			name = strings.Trim(name, " ")
+			if visited[name] {
+				if ff != nil {
+					return errors.New("Cannot use two forms of the same flag: " + name + " " + ff.Name)
+				}
+				ff = set.Lookup(name)
+			}
+		}
+		if ff == nil {
+			continue
+		}
+		for _, name := range parts {
+			name = strings.Trim(name, " ")
+			if !visited[name] {
+				copyFlag(name, ff, set)
+			}
+		}
+	}
+	return nil
+}