diff --git a/README.md b/README.md
index 6ff358d9d603cb25833a36afb774cc46595e63f5..72f59413200d289fac865d328468a9712b0a7f2f 100644
--- a/README.md
+++ b/README.md
@@ -27,29 +27,66 @@ If you wanna know more about dubbo-go, please visit this reference [Project Arch
Finished List:
-- Role: Consumer, Provider
-- Transport: HTTP, TCP
-- Codec: JsonRPC v2, Hessian v2
-- Registry: ZooKeeper/[etcd v3](https://github.com/apache/dubbo-go/pull/148)/[nacos](https://github.com/apache/dubbo-go/pull/151)/[consul](https://github.com/apache/dubbo-go/pull/121)
-- Dynamic Configure Center & Service Management Configurator: Zookeeper
-- Cluster Strategy: Failover/[Failfast](https://github.com/apache/dubbo-go/pull/140)/[Failsafe/Failback](https://github.com/apache/dubbo-go/pull/136)/[Available](https://github.com/apache/dubbo-go/pull/155)/[Broadcast](https://github.com/apache/dubbo-go/pull/158)/[Forking](https://github.com/apache/dubbo-go/pull/161)
-- Load Balance: Random/[RoundRobin](https://github.com/apache/dubbo-go/pull/66)/[LeastActive](https://github.com/apache/dubbo-go/pull/65)
-- Filter: Echo Health Check/[Circuit break and service downgrade](https://github.com/apache/dubbo-go/pull/133)/[TokenFilter](https://github.com/apache/dubbo-go/pull/202)/[AccessLogFilter](https://github.com/apache/dubbo-go/pull/214)/[TpsLimitFilter](https://github.com/apache/dubbo-go/pull/237)/[ExecuteLimitFilter](https://github.com/apache/dubbo-go/pull/246)
-- Other feature: [generic invoke](https://github.com/apache/dubbo-go/pull/122)/start check/connecting certain provider/multi-protocols/multi-registries/multi-versions/service group
+- Role
+ * Consumer
+ * Provider
+
+- Transport
+ * HTTP
+ * TCP
+
+- Codec
+ * JsonRPC V2
+ * Hessian V2
+
+- Registry
+ * ZooKeeper
+ * [etcd v3](https://github.com/apache/dubbo-go/pull/148)
+ * [nacos](https://github.com/apache/dubbo-go/pull/151)
+ * [consul](https://github.com/apache/dubbo-go/pull/121)
+
+- Dynamic Configure Center & Service Management Configurator
+ * Zookeeper
+ * [apollo](https://github.com/apache/dubbo-go/pull/250)
+
+- Cluster Strategy
+ * Failover
+ * [Failfast](https://github.com/apache/dubbo-go/pull/140)
+ * [Failsafe/Failback](https://github.com/apache/dubbo-go/pull/136)
+ * [Available](https://github.com/apache/dubbo-go/pull/155)
+ * [Broadcast](https://github.com/apache/dubbo-go/pull/158)
+ * [Forking](https://github.com/apache/dubbo-go/pull/161)
+
+- Load Balance
+ * Random
+ * [RoundRobin](https://github.com/apache/dubbo-go/pull/66)
+ * [LeastActive](https://github.com/apache/dubbo-go/pull/65)
+
+- Filter
+ * Echo Health Check
+ * [Circuit break and service downgrade](https://github.com/apache/dubbo-go/pull/133)
+ * [TokenFilter](https://github.com/apache/dubbo-go/pull/202)
+ * [AccessLogFilter](https://github.com/apache/dubbo-go/pull/214)
+ * [TpsLimitFilter](https://github.com/apache/dubbo-go/pull/237)
+ * [ExecuteLimitFilter](https://github.com/apache/dubbo-go/pull/246)
+
+- Invoke
+ * [generic invoke](https://github.com/apache/dubbo-go/pull/122)
+
+- Others:
+ * start check
+ * connecting certain provider
+ * multi-protocols
+ * multi-registries
+ * multi-versions
+ * service group
Working List:
- Load Balance: ConsistentHash
- Registry: k8s
-- Configure Center: apollo
- Metadata Center (dubbo v2.7.x)
-- Metrics: Promethus(dubbo v2.7.x)
-
-Todo List:
-
-- Registry: kubernetes
-- Routing: istio
-- tracing (dubbo ecosystem)
+- Metrics: Opentracing/Promethus(dubbo v2.7.x)
You can know more about dubbo-go by its [roadmap](https://github.com/apache/dubbo-go/wiki/Roadmap).
diff --git a/README_CN.md b/README_CN.md
index 7a7b061960e8dcc49f1da0ba6490ca8bbbf09e22..86eeb8dbe2ae8c8209c7135402aaaf626e06b9cf 100644
--- a/README_CN.md
+++ b/README_CN.md
@@ -26,32 +26,68 @@ Apache License, Version 2.0
瀹炵幇鍒楄〃:
-- 瑙掕壊绔�: Consumer, Provider
-- 浼犺緭鍗忚: HTTP, TCP
-- 搴忓垪鍖栧崗璁�: JsonRPC v2, Hessian v2
-- 娉ㄥ唽涓績: ZooKeeper/[etcd v3](https://github.com/apache/dubbo-go/pull/148)/[nacos](https://github.com/apache/dubbo-go/pull/151)/[consul](https://github.com/apache/dubbo-go/pull/121)
-- 鍔ㄦ€侀厤缃腑蹇冧笌鏈嶅姟娌荤悊閰嶇疆鍣紙config center锛�: Zookeeper
-- 闆嗙兢绛栫暐: Failover/[Failfast](https://github.com/apache/dubbo-go/pull/140)/[Failsafe/Failback](https://github.com/apache/dubbo-go/pull/136)/[Available](https://github.com/apache/dubbo-go/pull/155)/[Broadcast](https://github.com/apache/dubbo-go/pull/158)/[Forking](https://github.com/apache/dubbo-go/pull/161)
-- 璐熻浇鍧囪 绛栫暐: Random/[RoundRobin](https://github.com/apache/dubbo-go/pull/66)/[LeastActive](https://github.com/apache/dubbo-go/pull/65)
-- 杩囨护鍣�: Echo Health Check/[鏈嶅姟鐔旀柇&闄嶇骇](https://github.com/apache/dubbo-go/pull/133)/[TokenFilter](https://github.com/apache/dubbo-go/pull/202)/[AccessLogFilter](https://github.com/apache/dubbo-go/pull/214)/[TpsLimitFilter](https://github.com/apache/dubbo-go/pull/237)[ExecuteLimitFilter](https://github.com/apache/dubbo-go/pull/246)
-- 鍏朵粬鍔熻兘鏀寔: [娉涘寲璋冪敤](https://github.com/apache/dubbo-go/pull/122)/鍚姩鏃舵鏌�/鏈嶅姟鐩磋繛/澶氭湇鍔″崗璁�/澶氭敞鍐屼腑蹇�/澶氭湇鍔＄増鏈�/鏈嶅姟鍒嗙粍
+- 瑙掕壊绔�
+ * Consumer
+ * Provider
+
+- 浼犺緭鍗忚
+ * HTTP
+ * TCP
+
+- 搴忓垪鍖栧崗璁�
+ * JsonRPC V2
+ * Hessian V2
+
+- 娉ㄥ唽涓績
+ * ZooKeeper
+ * [etcd v3](https://github.com/apache/dubbo-go/pull/148)
+ * [nacos](https://github.com/apache/dubbo-go/pull/151)
+ * [consul](https://github.com/apache/dubbo-go/pull/121)
+
+- 鍔ㄦ€侀厤缃腑蹇冧笌鏈嶅姟娌荤悊閰嶇疆鍣�
+ * Zookeeper
+ * [apollo](https://github.com/apache/dubbo-go/pull/250)
+
+- 闆嗙兢绛栫暐
+ * Failover
+ * [Failfast](https://github.com/apache/dubbo-go/pull/140)
+ * [Failsafe/Failback](https://github.com/apache/dubbo-go/pull/136)
+ * [Available](https://github.com/apache/dubbo-go/pull/155)
+ * [Broadcast](https://github.com/apache/dubbo-go/pull/158)
+ * [Forking](https://github.com/apache/dubbo-go/pull/161)
+
+- 璐熻浇鍧囪 绛栫暐
+ * Random
+ * [RoundRobin](https://github.com/apache/dubbo-go/pull/66)
+ * [LeastActive](https://github.com/apache/dubbo-go/pull/65)
+
+- 杩囨护鍣�
+ * Echo Health Check
+ * [鏈嶅姟鐔旀柇&闄嶇骇](https://github.com/apache/dubbo-go/pull/133)
+ * [TokenFilter](https://github.com/apache/dubbo-go/pull/202)
+ * [AccessLogFilter](https://github.com/apache/dubbo-go/pull/214)
+ * [TpsLimitFilter](https://github.com/apache/dubbo-go/pull/237)
+ * [ExecuteLimitFilter](https://github.com/apache/dubbo-go/pull/246)
+
+- 璋冪敤
+ * [娉涘寲璋冪敤](https://github.com/apache/dubbo-go/pull/122)
+
+- 鍏朵粬鍔熻兘鏀寔:
+ * 鍚姩鏃舵鏌�
+ * 鏈嶅姟鐩磋繛
+ * 澶氭湇鍔″崗璁�
+ * 澶氭敞鍐屼腑蹇�
+ * 澶氭湇鍔＄増鏈�
+ * 鏈嶅姟鍒嗙粍
寮€鍙戜腑鍒楄〃:
-- 闆嗙兢绛栫暐: Forking
- 璐熻浇鍧囪 绛栫暐: ConsistentHash
- 娉ㄥ唽涓績: k8s
-- 閰嶇疆涓績: apollo
- 鍏冩暟鎹腑蹇� (dubbo v2.7.x)
-- Metrics: Promethus(dubbo v2.7.x)
+- Metrics: Opentracing/Promethus(dubbo v2.7.x)
-浠诲姟鍒楄〃:
-
-- 娉ㄥ唽涓績: kubernetes
-- Routing: istio
-- tracing (dubbo ecosystem)
-
-浣犲彲浠ラ€氳繃璁块棶 [roadmap](https://github.com/apache/dubbo-go/wiki/Roadmap) 鐭ラ亾鏇村鍏充簬 dubbo-go 鐨勪俊鎭�
+浣犲彲浠ラ€氳繃璁块棶 [roadmap](https://github.com/apache/dubbo-go/wiki/Roadmap) 鐭ラ亾鏇村鍏充簬 dubbo-go 鐨勪俊鎭€�
## 鏂囨。
diff --git a/before_ut.bat b/before_ut.bat
index a10df71a7eeb2eadb9d86574dc374b14a4f74e05..5296d0f8769b7b9f521f82e68bf3b10f4b5d16b4 100644
--- a/before_ut.bat
+++ b/before_ut.bat
@@ -1,3 +1,19 @@
+::
+:: Licensed to the Apache Software Foundation (ASF) under one or more
+:: contributor license agreements. See the NOTICE file distributed with
+:: this work for additional information regarding copyright ownership.
+:: The ASF licenses this file to You under the Apache License, Version 2.0
+:: (the "License"); you may not use this file except in compliance with
+:: the License. You may obtain a copy of the License at
+::
+:: http://www.apache.org/licenses/LICENSE-2.0
+::
+:: Unless required by applicable law or agreed to in writing, software
+:: distributed under the License is distributed on an "AS IS" BASIS,
+:: WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+:: See the License for the specific language governing permissions and
+:: limitations under the License.
+
set zkJar=zookeeper-3.4.9-fatjar.jar
md remoting\zookeeper\zookeeper-4unittest\contrib\fatjar config_center\zookeeper\zookeeper-4unittest\contrib\fatjar registry\zookeeper\zookeeper-4unittest\contrib\fatjar
curl -L https://github.com/dubbogo/resources/raw/master/zookeeper-4unitest/contrib/fatjar/%zkJar% -o remoting/zookeeper/zookeeper-4unittest/contrib/fatjar/%zkJar%
diff --git a/before_ut.sh b/before_ut.sh
index 7acee76ce5991ac1d06bff6a6325f083904f10b9..323173bcc64c3cbe9916747e10dd3ea8538457ea 100644
--- a/before_ut.sh
+++ b/before_ut.sh
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
mkdir -p remoting/zookeeper/zookeeper-4unittest/contrib/fatjar config_center/zookeeper/zookeeper-4unittest/contrib/fatjar registry/zookeeper/zookeeper-4unittest/contrib/fatjar
wget -P "remoting/zookeeper/zookeeper-4unittest/contrib/fatjar" https://github.com/dubbogo/resources/raw/master/zookeeper-4unitest/contrib/fatjar/zookeeper-3.4.9-fatjar.jar
cp remoting/zookeeper/zookeeper-4unittest/contrib/fatjar/zookeeper-3.4.9-fatjar.jar config_center/zookeeper/zookeeper-4unittest/contrib/fatjar/
diff --git a/common/constant/default.go b/common/constant/default.go
index cb6d68af0561d44f4306f16973a89759c9a9ac37..cb66f5f0ab1cd917278b71103f34a341a1e598d6 100644
--- a/common/constant/default.go
+++ b/common/constant/default.go
@@ -46,8 +46,8 @@ const (
const (
DEFAULT_KEY = "default"
PREFIX_DEFAULT_KEY = "default."
- DEFAULT_SERVICE_FILTERS = "echo,token,accesslog,tps,execute"
- DEFAULT_REFERENCE_FILTERS = ""
+ DEFAULT_SERVICE_FILTERS = "echo,token,accesslog,tps,execute,pshutdown"
+ DEFAULT_REFERENCE_FILTERS = "cshutdown"
GENERIC_REFERENCE_FILTERS = "generic"
GENERIC = "$invoke"
ECHO = "$echo"
diff --git a/common/constant/key.go b/common/constant/key.go
index a52f15875a31a99f4d6a305454fce92aa3f29cdc..17368b45ae49d06310ecff4b9cf05e7b8b4d26f7 100644
--- a/common/constant/key.go
+++ b/common/constant/key.go
@@ -71,6 +71,8 @@ const (
EXECUTE_LIMIT_KEY = "execute.limit"
DEFAULT_EXECUTE_LIMIT = "-1"
EXECUTE_REJECTED_EXECUTION_HANDLER_KEY = "execute.limit.rejected.handler"
+ PROVIDER_SHUTDOWN_FILTER = "pshutdown"
+ CONSUMER_SHUTDOWN_FILTER = "cshutdown"
)
const (
@@ -116,6 +118,7 @@ const (
ProtocolConfigPrefix = "dubbo.protocols."
ProviderConfigPrefix = "dubbo.provider."
ConsumerConfigPrefix = "dubbo.consumer."
+ ShutdownConfigPrefix = "dubbo.shutdown."
)
const (
diff --git a/common/extension/graceful_shutdown.go b/common/extension/graceful_shutdown.go
new file mode 100644
index 0000000000000000000000000000000000000000..c8807fcc28c18c1a6fddb4e97708e9b0d5cda243
--- /dev/null
+++ b/common/extension/graceful_shutdown.go
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package extension
+
+import (
+ "container/list"
+)
+
+var (
+ customShutdownCallbacks = list.New()
+)
+
+/**
+ * you should not make any assumption about the order.
+ * For example, if you have more than one callbacks, and you wish the order is:
+ * callback1()
+ * callback2()
+ * ...
+ * callbackN()
+ * Then you should put then together:
+ * func callback() {
+ * callback1()
+ * callback2()
+ * ...
+ * callbackN()
+ * }
+ * I think the order of custom callbacks should be decided by the users.
+ * Even though I can design a mechanism to support the ordered custom callbacks,
+ * the benefit of that mechanism is low.
+ * And it may introduce much complication for another users.
+ */
+func AddCustomShutdownCallback(callback func()) {
+ customShutdownCallbacks.PushBack(callback)
+}
+
+func GetAllCustomShutdownCallbacks() *list.List {
+ return customShutdownCallbacks
+}
diff --git a/config/base_config_test.go b/config/base_config_test.go
index 54def2ae1bc60135619d86be8939710589df5b13..ab2769578072387e4686593f3c2c10fb8e49731d 100644
--- a/config/base_config_test.go
+++ b/config/base_config_test.go
@@ -40,6 +40,7 @@ func Test_refresh(t *testing.T) {
mockMap["dubbo.com.MockService.MockService.GetUser.retries"] = "10"
mockMap["dubbo.consumer.check"] = "false"
mockMap["dubbo.application.name"] = "dubbo"
+ mockMap["dubbo.shutdown.timeout"] = "12s"
config.GetEnvInstance().UpdateExternalConfigMap(mockMap)
@@ -114,6 +115,13 @@ func Test_refresh(t *testing.T) {
},
},
},
+ ShutdownConfig: &ShutdownConfig{
+ Timeout: "12s",
+ StepTimeout: "2s",
+ RejectRequestHandler: "mock",
+ RejectRequest: false,
+ RequestsFinished: false,
+ },
}
c.SetFatherConfig(father)
diff --git a/config/config_loader.go b/config/config_loader.go
index b737d3f233700f596469cfd678aa7ae7f9a82b85..414bb479025c5d6111a6373fa2626f21ffa73ef0 100644
--- a/config/config_loader.go
+++ b/config/config_loader.go
@@ -149,6 +149,8 @@ func Load() {
}
}
}
+ // init the shutdown callback
+ GracefulShutdownInit()
}
// get rpc service for consumer
diff --git a/config/consumer_config.go b/config/consumer_config.go
index b1ebdd5d8e082bf836071460e2a330632e07335c..72f60b5f77b9b9cc633d8939713c0eb93563deac 100644
--- a/config/consumer_config.go
+++ b/config/consumer_config.go
@@ -52,11 +52,12 @@ type ConsumerConfig struct {
ProxyFactory string `yaml:"proxy_factory" default:"default" json:"proxy_factory,omitempty" property:"proxy_factory"`
Check *bool `yaml:"check" json:"check,omitempty" property:"check"`
- Registry *RegistryConfig `yaml:"registry" json:"registry,omitempty" property:"registry"`
- Registries map[string]*RegistryConfig `yaml:"registries" json:"registries,omitempty" property:"registries"`
- References map[string]*ReferenceConfig `yaml:"references" json:"references,omitempty" property:"references"`
- ProtocolConf interface{} `yaml:"protocol_conf" json:"protocol_conf,omitempty" property:"protocol_conf"`
- FilterConf interface{} `yaml:"filter_conf" json:"filter_conf,omitempty" property:"filter_conf" `
+ Registry *RegistryConfig `yaml:"registry" json:"registry,omitempty" property:"registry"`
+ Registries map[string]*RegistryConfig `yaml:"registries" json:"registries,omitempty" property:"registries"`
+ References map[string]*ReferenceConfig `yaml:"references" json:"references,omitempty" property:"references"`
+ ProtocolConf interface{} `yaml:"protocol_conf" json:"protocol_conf,omitempty" property:"protocol_conf"`
+ FilterConf interface{} `yaml:"filter_conf" json:"filter_conf,omitempty" property:"filter_conf" `
+ ShutdownConfig *ShutdownConfig `yaml:"shutdown_conf" json:"shutdown_conf,omitempty" property:"shutdown_conf" `
}
func (c *ConsumerConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
diff --git a/config/graceful_shutdown.go b/config/graceful_shutdown.go
new file mode 100644
index 0000000000000000000000000000000000000000..fedb2c15ecdab62d17f0a4e83c45522f1c18acb0
--- /dev/null
+++ b/config/graceful_shutdown.go
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package config
+
+import (
+ "os"
+ "os/signal"
+ "runtime/debug"
+ "time"
+)
+
+import (
+ "github.com/dubbogo/gost/container/gxset"
+)
+
+import (
+ "github.com/apache/dubbo-go/common/constant"
+ "github.com/apache/dubbo-go/common/extension"
+ "github.com/apache/dubbo-go/common/logger"
+)
+
+/*
+ * The key point is that find out the signals to handle.
+ * The most important documentation is https://golang.org/pkg/os/signal/
+ * From this documentation, we can know that:
+ * 1. The signals SIGKILL and SIGSTOP may not be caught by signal package;
+ * 2. SIGHUP, SIGINT, or SIGTERM signal causes the program to exit
+ * 3. SIGQUIT, SIGILL, SIGTRAP, SIGABRT, SIGSTKFLT, SIGEMT, or SIGSYS signal causes the program to exit with a stack dump
+ * 4. The invocation of Notify(signal...) will disable the default behavior of those signals.
+ *
+ * So the signals SIGKILL, SIGSTOP, SIGHUP, SIGINT, SIGTERM, SIGQUIT, SIGILL, SIGTRAP, SIGABRT, SIGSTKFLT, SIGEMT, SIGSYS
+ * should be processed.
+ * syscall.SIGEMT cannot be found in CI
+ * It's seems that the Unix/Linux does not have the signal SIGSTKFLT. https://github.com/golang/go/issues/33381
+ * So this signal will be ignored.
+ * The signals are different on different platforms.
+ * We define them by using 'package build' feature https://golang.org/pkg/go/build/
+ */
+
+func GracefulShutdownInit() {
+
+ signals := make(chan os.Signal, 1)
+
+ signal.Notify(signals, ShutdownSignals...)
+
+ go func() {
+ select {
+ case sig := <-signals:
+ logger.Infof("get signal %s, application will shutdown.", sig)
+ // gracefulShutdownOnce.Do(func() {
+ BeforeShutdown()
+
+ // those signals' original behavior is exit with dump ths stack, so we try to keep the behavior
+ for _, dumpSignal := range DumpHeapShutdownSignals {
+ if sig == dumpSignal {
+ debug.WriteHeapDump(os.Stdout.Fd())
+ }
+ }
+
+ time.AfterFunc(totalTimeout(), func() {
+ logger.Warn("Shutdown gracefully timeout, application will shutdown immediately. ")
+ os.Exit(0)
+ })
+
+ os.Exit(0)
+ }
+ }()
+}
+
+func BeforeShutdown() {
+
+ destroyAllRegistries()
+ // waiting for a short time so that the clients have enough time to get the notification that server shutdowns
+ // The value of configuration depends on how long the clients will get notification.
+ waitAndAcceptNewRequests()
+
+ // reject the new request, but keeping waiting for accepting requests
+ waitForReceivingRequests()
+
+ // we fetch the protocols from Consumer.References. Consumer.ProtocolConfig doesn't contains all protocol, like jsonrpc
+ consumerProtocols := getConsumerProtocols()
+
+ // If this application is not the provider, it will do nothing
+ destroyProviderProtocols(consumerProtocols)
+
+ // reject sending the new request, and waiting for response of sending requests
+ waitForSendingRequests()
+
+ // If this application is not the consumer, it will do nothing
+ destroyConsumerProtocols(consumerProtocols)
+
+ logger.Info("Graceful shutdown --- Execute the custom callbacks.")
+ customCallbacks := extension.GetAllCustomShutdownCallbacks()
+ for callback := customCallbacks.Front(); callback != nil; callback = callback.Next() {
+ callback.Value.(func())()
+ }
+}
+
+func destroyAllRegistries() {
+ logger.Info("Graceful shutdown --- Destroy all registries. ")
+ registryProtocol := extension.GetProtocol(constant.REGISTRY_KEY)
+ registryProtocol.Destroy()
+}
+
+func destroyConsumerProtocols(consumerProtocols *gxset.HashSet) {
+ logger.Info("Graceful shutdown --- Destroy consumer's protocols. ")
+ for name := range consumerProtocols.Items {
+ extension.GetProtocol(name.(string)).Destroy()
+ }
+}
+
+/**
+ * destroy the provider's protocol.
+ * if the protocol is consumer's protocol too, we will keep it.
+ */
+func destroyProviderProtocols(consumerProtocols *gxset.HashSet) {
+
+ logger.Info("Graceful shutdown --- Destroy provider's protocols. ")
+
+ if providerConfig == nil || providerConfig.Protocols == nil {
+ return
+ }
+
+ for _, protocol := range providerConfig.Protocols {
+
+ // the protocol is the consumer's protocol too, we can not destroy it.
+ if consumerProtocols.Contains(protocol.Name) {
+ continue
+ }
+ extension.GetProtocol(protocol.Name).Destroy()
+ }
+}
+
+func waitAndAcceptNewRequests() {
+
+ logger.Info("Graceful shutdown --- Keep waiting and accept new requests for a short time. ")
+ if providerConfig == nil || providerConfig.ShutdownConfig == nil {
+ return
+ }
+
+ timeout := providerConfig.ShutdownConfig.GetStepTimeout()
+
+ // ignore this step
+ if timeout < 0 {
+ return
+ }
+ time.Sleep(timeout)
+}
+
+// for provider. It will wait for processing receiving requests
+func waitForReceivingRequests() {
+ logger.Info("Graceful shutdown --- Keep waiting until accepting requests finish or timeout. ")
+ if providerConfig == nil || providerConfig.ShutdownConfig == nil {
+ // ignore this step
+ return
+ }
+ waitingProcessedTimeout(providerConfig.ShutdownConfig)
+}
+
+// for consumer. It will wait for the response of sending requests
+func waitForSendingRequests() {
+ logger.Info("Graceful shutdown --- Keep waiting until sending requests getting response or timeout ")
+ if consumerConfig == nil || consumerConfig.ShutdownConfig == nil {
+ // ignore this step
+ return
+ }
+ waitingProcessedTimeout(consumerConfig.ShutdownConfig)
+}
+
+func waitingProcessedTimeout(shutdownConfig *ShutdownConfig) {
+ timeout := shutdownConfig.GetStepTimeout()
+ if timeout <= 0 {
+ return
+ }
+ start := time.Now()
+
+ for time.Now().After(start.Add(timeout)) && !shutdownConfig.RequestsFinished {
+ // sleep 10 ms and then we check it again
+ time.Sleep(10 * time.Millisecond)
+ }
+}
+
+func totalTimeout() time.Duration {
+ var providerShutdown time.Duration
+ if providerConfig != nil && providerConfig.ShutdownConfig != nil {
+ providerShutdown = providerConfig.ShutdownConfig.GetTimeout()
+ }
+
+ var consumerShutdown time.Duration
+ if consumerConfig != nil && consumerConfig.ShutdownConfig != nil {
+ consumerShutdown = consumerConfig.ShutdownConfig.GetTimeout()
+ }
+
+ var timeout = providerShutdown
+ if consumerShutdown > providerShutdown {
+ timeout = consumerShutdown
+ }
+ return timeout
+}
+
+/*
+ * we can not get the protocols from consumerConfig because some protocol don't have configuration, like jsonrpc.
+ */
+func getConsumerProtocols() *gxset.HashSet {
+ result := gxset.NewSet()
+ if consumerConfig == nil || consumerConfig.References == nil {
+ return result
+ }
+
+ for _, reference := range consumerConfig.References {
+ result.Add(reference.Protocol)
+ }
+ return result
+}
diff --git a/config/graceful_shutdown_config.go b/config/graceful_shutdown_config.go
new file mode 100644
index 0000000000000000000000000000000000000000..df55728565f6cf14ce4357f8c9c7927c30d80e40
--- /dev/null
+++ b/config/graceful_shutdown_config.go
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package config
+
+import (
+ "time"
+)
+
+import (
+ "github.com/apache/dubbo-go/common/constant"
+ "github.com/apache/dubbo-go/common/logger"
+)
+
+const (
+ defaultTimeout = 60 * time.Second
+ defaultStepTimeout = 10 * time.Second
+)
+
+type ShutdownConfig struct {
+ /*
+ * Total timeout. Even though we don't release all resources,
+ * the application will shutdown if the costing time is over this configuration. The unit is ms.
+ * default value is 60 * 1000 ms = 1 minutes
+ * In general, it should be bigger than 3 * StepTimeout.
+ */
+ Timeout string `default:"60s" yaml:"timeout" json:"timeout,omitempty" property:"timeout"`
+ /*
+ * the timeout on each step. You should evaluate the response time of request
+ * and the time that client noticed that server shutdown.
+ * For example, if your client will received the notification within 10s when you start to close server,
+ * and the 99.9% requests will return response in 2s, so the StepTimeout will be bigger than(10+2) * 1000ms,
+ * maybe (10 + 2*3) * 1000ms is a good choice.
+ */
+ StepTimeout string `default:"10s" yaml:"step_timeout" json:"step.timeout,omitempty" property:"step.timeout"`
+ // when we try to shutdown the application, we will reject the new requests. In most cases, you don't need to configure this.
+ RejectRequestHandler string `yaml:"reject_handler" json:"reject_handler,omitempty" property:"reject_handler"`
+ // true -> new request will be rejected.
+ RejectRequest bool
+
+ // true -> all requests had been processed. In provider side it means that all requests are returned response to clients
+ // In consumer side, it means that all requests getting response from servers
+ RequestsFinished bool
+}
+
+func (config *ShutdownConfig) Prefix() string {
+ return constant.ShutdownConfigPrefix
+}
+
+func (config *ShutdownConfig) GetTimeout() time.Duration {
+ result, err := time.ParseDuration(config.Timeout)
+ if err != nil {
+ logger.Errorf("The Timeout configuration is invalid: %s, and we will use the default value: %s, err: %v",
+ config.Timeout, defaultTimeout.String(), err)
+ return defaultTimeout
+ }
+ return result
+}
+
+func (config *ShutdownConfig) GetStepTimeout() time.Duration {
+ result, err := time.ParseDuration(config.StepTimeout)
+ if err != nil {
+ logger.Errorf("The StepTimeout configuration is invalid: %s, and we will use the default value: %s, err: %v",
+ config.StepTimeout, defaultStepTimeout.String(), err)
+ return defaultStepTimeout
+ }
+ return result
+}
diff --git a/config/graceful_shutdown_config_test.go b/config/graceful_shutdown_config_test.go
new file mode 100644
index 0000000000000000000000000000000000000000..583ed70b838a8271a47e180ee3c6eb32cbb46984
--- /dev/null
+++ b/config/graceful_shutdown_config_test.go
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package config
+
+import (
+ "testing"
+ "time"
+)
+
+import (
+ "github.com/stretchr/testify/assert"
+)
+
+func TestShutdownConfig_GetTimeout(t *testing.T) {
+ config := ShutdownConfig{}
+ assert.False(t, config.RejectRequest)
+ assert.False(t, config.RequestsFinished)
+
+ config = ShutdownConfig{
+ Timeout: "12x",
+ StepTimeout: "34a",
+ }
+
+ assert.Equal(t, 60*time.Second, config.GetTimeout())
+ assert.Equal(t, 10*time.Second, config.GetStepTimeout())
+
+ config = ShutdownConfig{
+ Timeout: "34ms",
+ StepTimeout: "79ms",
+ }
+
+ assert.Equal(t, 34*time.Millisecond, config.GetTimeout())
+ assert.Equal(t, 79*time.Millisecond, config.GetStepTimeout())
+}
diff --git a/config/graceful_shutdown_signal_darwin.go b/config/graceful_shutdown_signal_darwin.go
new file mode 100644
index 0000000000000000000000000000000000000000..59c1a5d149c2e9db8e9ac981adec107cafc863ad
--- /dev/null
+++ b/config/graceful_shutdown_signal_darwin.go
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package config
+
+import (
+ "os"
+ "syscall"
+)
+
+var ShutdownSignals = []os.Signal{os.Interrupt, os.Kill, syscall.SIGKILL, syscall.SIGSTOP,
+ syscall.SIGHUP, syscall.SIGINT, syscall.SIGQUIT, syscall.SIGILL, syscall.SIGTRAP,
+ syscall.SIGABRT, syscall.SIGSYS}
+
+var DumpHeapShutdownSignals = []os.Signal{syscall.SIGQUIT, syscall.SIGILL,
+ syscall.SIGTRAP, syscall.SIGABRT, syscall.SIGSYS}
diff --git a/config/graceful_shutdown_signal_linux.go b/config/graceful_shutdown_signal_linux.go
new file mode 100644
index 0000000000000000000000000000000000000000..59c1a5d149c2e9db8e9ac981adec107cafc863ad
--- /dev/null
+++ b/config/graceful_shutdown_signal_linux.go
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package config
+
+import (
+ "os"
+ "syscall"
+)
+
+var ShutdownSignals = []os.Signal{os.Interrupt, os.Kill, syscall.SIGKILL, syscall.SIGSTOP,
+ syscall.SIGHUP, syscall.SIGINT, syscall.SIGQUIT, syscall.SIGILL, syscall.SIGTRAP,
+ syscall.SIGABRT, syscall.SIGSYS}
+
+var DumpHeapShutdownSignals = []os.Signal{syscall.SIGQUIT, syscall.SIGILL,
+ syscall.SIGTRAP, syscall.SIGABRT, syscall.SIGSYS}
diff --git a/config/graceful_shutdown_signal_windows.go b/config/graceful_shutdown_signal_windows.go
new file mode 100644
index 0000000000000000000000000000000000000000..91b2bce7c2311ecbe9a1255be3e7b7b357a9b403
--- /dev/null
+++ b/config/graceful_shutdown_signal_windows.go
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package config
+
+import (
+ "os"
+ "syscall"
+)
+
+var ShutdownSignals = []os.Signal{os.Interrupt, os.Kill, syscall.SIGKILL,
+ syscall.SIGHUP, syscall.SIGINT, syscall.SIGQUIT, syscall.SIGILL, syscall.SIGTRAP,
+ syscall.SIGABRT}
+
+var DumpHeapShutdownSignals = []os.Signal{syscall.SIGQUIT, syscall.SIGILL, syscall.SIGTRAP, syscall.SIGABRT}
diff --git a/config/graceful_shutdown_test.go b/config/graceful_shutdown_test.go
new file mode 100644
index 0000000000000000000000000000000000000000..de203572c76281d221181dea90b0f31b43038de6
--- /dev/null
+++ b/config/graceful_shutdown_test.go
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package config
+
+import (
+ "testing"
+)
+
+import (
+ "github.com/apache/dubbo-go/common/constant"
+ "github.com/apache/dubbo-go/common/extension"
+ "github.com/apache/dubbo-go/protocol"
+)
+
+func TestGracefulShutdownInit(t *testing.T) {
+ GracefulShutdownInit()
+}
+
+func TestBeforeShutdown(t *testing.T) {
+ extension.SetProtocol("registry", func() protocol.Protocol {
+ return &mockRegistryProtocol{}
+ })
+ extension.SetProtocol(constant.DUBBO, func() protocol.Protocol {
+ return &mockRegistryProtocol{}
+ })
+
+ extension.SetProtocol("mock", func() protocol.Protocol {
+ return &mockRegistryProtocol{}
+ })
+
+ // protocolConfigs := make(map[interface{}]interface{}, 16)
+ consumerReferences := map[string]*ReferenceConfig{}
+ consumerReferences[constant.DUBBO] = &ReferenceConfig{
+ Protocol: constant.DUBBO,
+ }
+
+ // without configuration
+ BeforeShutdown()
+
+ consumerConfig = &ConsumerConfig{
+ References: consumerReferences,
+ ShutdownConfig: &ShutdownConfig{
+ Timeout: "1",
+ StepTimeout: "1s",
+ }}
+
+ providerProtocols := map[string]*ProtocolConfig{}
+ providerProtocols[constant.DUBBO] = &ProtocolConfig{
+ Name: constant.DUBBO,
+ }
+
+ providerProtocols["mock"] = &ProtocolConfig{
+ Name: "mock",
+ }
+
+ providerConfig = &ProviderConfig{
+ ShutdownConfig: &ShutdownConfig{
+ Timeout: "1",
+ StepTimeout: "1s",
+ },
+ Protocols: providerProtocols,
+ }
+ // test destroy protocol
+ BeforeShutdown()
+
+ providerConfig = &ProviderConfig{
+ ShutdownConfig: &ShutdownConfig{
+ Timeout: "1",
+ StepTimeout: "-1s",
+ },
+ Protocols: providerProtocols,
+ }
+
+ consumerConfig = &ConsumerConfig{
+ References: consumerReferences,
+ ShutdownConfig: &ShutdownConfig{
+ Timeout: "1",
+ StepTimeout: "-1s",
+ },
+ }
+
+ // test ignore steps
+ BeforeShutdown()
+}
diff --git a/config/provider_config.go b/config/provider_config.go
index 00faa1d0ab1b65a7a39d7d3548e5b89b0f250aba..0fed44c81b124cd40825695981a5394c273203fa 100644
--- a/config/provider_config.go
+++ b/config/provider_config.go
@@ -49,6 +49,7 @@ type ProviderConfig struct {
Protocols map[string]*ProtocolConfig `yaml:"protocols" json:"protocols,omitempty" property:"protocols"`
ProtocolConf interface{} `yaml:"protocol_conf" json:"protocol_conf,omitempty" property:"protocol_conf" `
FilterConf interface{} `yaml:"filter_conf" json:"filter_conf,omitempty" property:"filter_conf" `
+ ShutdownConfig *ShutdownConfig `yaml:"shutdown_conf" json:"shutdown_conf,omitempty" property:"shutdown_conf" `
}
func (c *ProviderConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
@@ -106,6 +107,7 @@ func ProviderInit(confProFile string) error {
}
logger.Debugf("provider config{%#v}\n", providerConfig)
+
return nil
}
diff --git a/config/provider_config_test.go b/config/provider_config_test.go
index db4b5f9906efb25cdb0ad8bf91f412f4510f5af5..e8a9c1f7a730f79e5bf92e1d7dd2e42b969cb0f3 100644
--- a/config/provider_config_test.go
+++ b/config/provider_config_test.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package config
import (
diff --git a/config/reference_config.go b/config/reference_config.go
index c63ac2ef28ff85d07b76ad0f5fef669d83bca3a5..8703c459bab306f98beb1668a1f9438126586f24 100644
--- a/config/reference_config.go
+++ b/config/reference_config.go
@@ -183,7 +183,7 @@ func (refconfig *ReferenceConfig) getUrlMap() url.Values {
//filter
var defaultReferenceFilter = constant.DEFAULT_REFERENCE_FILTERS
if refconfig.Generic {
- defaultReferenceFilter = constant.GENERIC_REFERENCE_FILTERS + defaultReferenceFilter
+ defaultReferenceFilter = constant.GENERIC_REFERENCE_FILTERS + "," + defaultReferenceFilter
}
urlMap.Set(constant.REFERENCE_FILTER_KEY, mergeValue(consumerConfig.Filter, refconfig.Filter, defaultReferenceFilter))
diff --git a/config/testdata/consumer_config.yml b/config/testdata/consumer_config.yml
index 9fd50bb4d35a40d8532c9a644a86ad6834f8e89b..f44ea449fd16235050f6a7ba7823a87e24791780 100644
--- a/config/testdata/consumer_config.yml
+++ b/config/testdata/consumer_config.yml
@@ -49,6 +49,10 @@ references:
"soa.com.ikurento.user.UserProvider"
"forks": 5
+shutdown_conf:
+ timeout: 60s
+ step_timeout: 10s
+
protocol_conf:
dubbo:
reconnect_interval: 0
diff --git a/config/testdata/consumer_config_with_configcenter.yml b/config/testdata/consumer_config_with_configcenter.yml
index 0550cc89741b6a490aaba9ff8906d7dda1b3ed49..ebe56fa93f9f5728aa365ee5b7a99b6bb5857a8e 100644
--- a/config/testdata/consumer_config_with_configcenter.yml
+++ b/config/testdata/consumer_config_with_configcenter.yml
@@ -17,6 +17,10 @@ references:
- name: "GetUser"
retries: "3"
+shutdown_conf:
+ timeout: 60s
+ step_timeout: 10s
+
protocol_conf:
dubbo:
reconnect_interval: 0
diff --git a/config/testdata/consumer_config_withoutProtocol.yml b/config/testdata/consumer_config_withoutProtocol.yml
index 5e57c7ddf6e82152e4f207b2d06df1443766717c..32bad8b91db3fac9c026fca36c5dc3b84f4c3fc9 100644
--- a/config/testdata/consumer_config_withoutProtocol.yml
+++ b/config/testdata/consumer_config_withoutProtocol.yml
@@ -48,6 +48,10 @@ references:
"soa.com.ikurento.user.UserProvider"
"forks": 5
+shutdown_conf:
+ timeout: 60s
+ step_timeout: 10s
+
protocol_conf:
dubbo:
reconnect_interval: 0
diff --git a/config/testdata/provider_config.yml b/config/testdata/provider_config.yml
index 080feb7dcd1cccd06ae436b2854b2531177d23e3..7c46f9101aa9a6ecb88a92953dfcec28dda4e0ff 100644
--- a/config/testdata/provider_config.yml
+++ b/config/testdata/provider_config.yml
@@ -71,6 +71,10 @@ protocols:
# ip: "127.0.0.1"
# port: 20001
+shutdown_conf:
+ timeout: 60s
+ step_timeout: 10s
+
protocol_conf:
dubbo:
session_number: 700
diff --git a/config/testdata/provider_config_withoutProtocol.yml b/config/testdata/provider_config_withoutProtocol.yml
index 2f65868d4948db9a8b99c500014ea1307569d86f..532d3005aa351820bd540b31e2721dc2a0b5c6ed 100644
--- a/config/testdata/provider_config_withoutProtocol.yml
+++ b/config/testdata/provider_config_withoutProtocol.yml
@@ -51,6 +51,10 @@ protocols:
# ip: "127.0.0.1"
# port: 20001
+shutdown_conf:
+ timeout: 60s
+ step_timeout: 10s
+
protocol_conf:
dubbo:
session_number: 700
diff --git a/dubbogo.png b/dubbogo.png
deleted file mode 100644
index 2cac434091276df102c3ae405c09621b8d8926ef..0000000000000000000000000000000000000000
Binary files a/dubbogo.png and /dev/null differ
diff --git a/filter/impl/graceful_shutdown_filter.go b/filter/impl/graceful_shutdown_filter.go
new file mode 100644
index 0000000000000000000000000000000000000000..b912ea88e4ba4741b7d7fe36b8bbd3ba158abe63
--- /dev/null
+++ b/filter/impl/graceful_shutdown_filter.go
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package impl
+
+import (
+ "sync/atomic"
+)
+
+import (
+ "github.com/apache/dubbo-go/common/constant"
+ "github.com/apache/dubbo-go/common/extension"
+ "github.com/apache/dubbo-go/common/logger"
+ "github.com/apache/dubbo-go/config"
+ "github.com/apache/dubbo-go/filter"
+ "github.com/apache/dubbo-go/filter/common"
+ "github.com/apache/dubbo-go/protocol"
+)
+
+func init() {
+ var consumerFiler = &gracefulShutdownFilter{
+ shutdownConfig: config.GetConsumerConfig().ShutdownConfig,
+ }
+ var providerFilter = &gracefulShutdownFilter{
+ shutdownConfig: config.GetProviderConfig().ShutdownConfig,
+ }
+
+ extension.SetFilter(constant.CONSUMER_SHUTDOWN_FILTER, func() filter.Filter {
+ return consumerFiler
+ })
+
+ extension.SetFilter(constant.PROVIDER_SHUTDOWN_FILTER, func() filter.Filter {
+ return providerFilter
+ })
+}
+
+type gracefulShutdownFilter struct {
+ activeCount int32
+ shutdownConfig *config.ShutdownConfig
+}
+
+func (gf *gracefulShutdownFilter) Invoke(invoker protocol.Invoker, invocation protocol.Invocation) protocol.Result {
+ if gf.rejectNewRequest() {
+ logger.Info("The application is closing, new request will be rejected.")
+ return gf.getRejectHandler().RejectedExecution(invoker.GetUrl(), invocation)
+ }
+ atomic.AddInt32(&gf.activeCount, 1)
+ return invoker.Invoke(invocation)
+}
+
+func (gf *gracefulShutdownFilter) OnResponse(result protocol.Result, invoker protocol.Invoker, invocation protocol.Invocation) protocol.Result {
+ atomic.AddInt32(&gf.activeCount, -1)
+ // although this isn't thread safe, it won't be a problem if the gf.rejectNewRequest() is true.
+ if gf.shutdownConfig != nil && gf.activeCount <= 0 {
+ gf.shutdownConfig.RequestsFinished = true
+ }
+ return result
+}
+
+func (gf *gracefulShutdownFilter) rejectNewRequest() bool {
+ if gf.shutdownConfig == nil {
+ return false
+ }
+ return gf.shutdownConfig.RejectRequest
+}
+
+func (gf *gracefulShutdownFilter) getRejectHandler() common.RejectedExecutionHandler {
+ handler := constant.DEFAULT_KEY
+ if gf.shutdownConfig != nil && len(gf.shutdownConfig.RejectRequestHandler) > 0 {
+ handler = gf.shutdownConfig.RejectRequestHandler
+ }
+ return extension.GetRejectedExecutionHandler(handler)
+}
diff --git a/filter/impl/graceful_shutdown_filter_test.go b/filter/impl/graceful_shutdown_filter_test.go
new file mode 100644
index 0000000000000000000000000000000000000000..21da167ea0f201ea357c51cab0ecb4f8ebec0957
--- /dev/null
+++ b/filter/impl/graceful_shutdown_filter_test.go
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package impl
+
+import (
+ "net/url"
+ "testing"
+)
+
+import (
+ "github.com/stretchr/testify/assert"
+)
+
+import (
+ "github.com/apache/dubbo-go/common"
+ "github.com/apache/dubbo-go/common/constant"
+ "github.com/apache/dubbo-go/common/extension"
+ "github.com/apache/dubbo-go/config"
+ filterCommon "github.com/apache/dubbo-go/filter/common"
+ "github.com/apache/dubbo-go/filter/common/impl"
+ "github.com/apache/dubbo-go/protocol"
+ "github.com/apache/dubbo-go/protocol/invocation"
+)
+
+func TestGenericFilter_Invoke(t *testing.T) {
+ invoc := invocation.NewRPCInvocation("GetUser", []interface{}{"OK"}, make(map[string]string, 0))
+
+ invokeUrl := common.NewURLWithOptions(
+ common.WithParams(url.Values{}))
+
+ shutdownFilter := extension.GetFilter(constant.PROVIDER_SHUTDOWN_FILTER).(*gracefulShutdownFilter)
+
+ providerConfig := config.GetProviderConfig()
+
+ assert.False(t, shutdownFilter.rejectNewRequest())
+ assert.Nil(t, providerConfig.ShutdownConfig)
+
+ assert.Equal(t, extension.GetRejectedExecutionHandler(constant.DEFAULT_KEY),
+ shutdownFilter.getRejectHandler())
+
+ result := shutdownFilter.Invoke(protocol.NewBaseInvoker(*invokeUrl), invoc)
+ assert.NotNil(t, result)
+ assert.Nil(t, result.Error())
+
+ providerConfig.ShutdownConfig = &config.ShutdownConfig{
+ RejectRequest: true,
+ RejectRequestHandler: "mock",
+ }
+ shutdownFilter.shutdownConfig = providerConfig.ShutdownConfig
+
+ assert.True(t, shutdownFilter.rejectNewRequest())
+ result = shutdownFilter.OnResponse(nil, protocol.NewBaseInvoker(*invokeUrl), invoc)
+
+ rejectHandler := &impl.OnlyLogRejectedExecutionHandler{}
+ extension.SetRejectedExecutionHandler("mock", func() filterCommon.RejectedExecutionHandler {
+ return rejectHandler
+ })
+ assert.True(t, providerConfig.ShutdownConfig.RequestsFinished)
+ assert.Equal(t, rejectHandler, shutdownFilter.getRejectHandler())
+
+}
diff --git a/go.sum b/go.sum
index ee7072bd1dfecf798071264d1c49072a97220ca4..9855250a90f72eca314bf54cd9bea03a619b6a5e 100644
--- a/go.sum
+++ b/go.sum
@@ -37,8 +37,6 @@ github.com/aliyun/alibaba-cloud-sdk-go v0.0.0-20190802083043-4cd0c391755e/go.mod
github.com/aliyun/aliyun-oss-go-sdk v0.0.0-20190307165228-86c17b95fcd5/go.mod h1:T/Aws4fEfogEE9v+HPhhw+CntffsBHJ8nXQCwKr0/g8=
github.com/apache/dubbo-go-hessian2 v1.2.5-0.20191029001541-894e45c9aaaa h1:11TO1wiM5bvGAVrmfN5atD8gZqUSPE1TBoIs8sI6Abk=
github.com/apache/dubbo-go-hessian2 v1.2.5-0.20191029001541-894e45c9aaaa/go.mod h1:LWnndnrFXZmJLAzoyNAPNHSIJ1KOHVkTSsHgC3YYWlo=
-github.com/apache/dubbo-go-hessian2 v1.3.0 h1:ZhQYDm8GHqIp6i53T4ZJHQBN11nAYAjxlwoVznfyvD8=
-github.com/apache/dubbo-go-hessian2 v1.3.0/go.mod h1:LWnndnrFXZmJLAzoyNAPNHSIJ1KOHVkTSsHgC3YYWlo=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e h1:QEF07wC0T1rKkctt1RINW/+RMTVmiwxETico2l3gxJA=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
@@ -104,8 +102,6 @@ github.com/docker/go-connections v0.3.0 h1:3lOnM9cSzgGwx8VfK/NGOW5fLQ0GjIlCkaktF
github.com/docker/go-connections v0.3.0/go.mod h1:Gbd7IOopHjR8Iph03tsViu4nIes5XhDvyHbTtUxmeec=
github.com/docker/go-units v0.3.3 h1:Xk8S3Xj5sLGlG5g67hJmYMmUgXv5N4PhkjJHHqrwnTk=
github.com/docker/go-units v0.3.3/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
-github.com/dubbogo/getty v1.3.0 h1:GImOCANdts7dlRqi9GMVsZJnfst9EPyjTVTR1AesOD8=
-github.com/dubbogo/getty v1.3.0/go.mod h1:K4b3MkGLf7T+lMgQNFgpg0dI1Wvv1PTisFs1Psf86kU=
github.com/dubbogo/getty v1.3.1 h1:9fehwTo/D6+z6/+kADMbhbKeMkP80o/3g+XwV5lFLTY=
github.com/dubbogo/getty v1.3.1/go.mod h1:dtLOEb1v6EMHsQNYRWEACiRLmTWB2kJGUAj1aXayPOg=
github.com/dubbogo/gost v1.1.1 h1:JCM7vx5edPIjDA5ovJTuzEEXuw2t7xLyrlgi2mi5jHI=
diff --git a/protocol/jsonrpc/http.go b/protocol/jsonrpc/http.go
index b64a3a344e95164b8f49556cdc155bf34642a83b..3d99786624c71818cc5f787c8695d1c116c35707 100644
--- a/protocol/jsonrpc/http.go
+++ b/protocol/jsonrpc/http.go
@@ -66,8 +66,8 @@ type HTTPOptions struct {
}
var defaultHTTPOptions = HTTPOptions{
- HandshakeTimeout: 3e9,
- HTTPTimeout: 3e9,
+ HandshakeTimeout: 3 * time.Second,
+ HTTPTimeout: 3 * time.Second,
}
type HTTPClient struct {
@@ -115,7 +115,7 @@ func (c *HTTPClient) Call(ctx context.Context, service common.URL, req *Request,
reqTimeout := c.options.HTTPTimeout
if reqTimeout <= 0 {
- reqTimeout = 1e8
+ reqTimeout = 100 * time.Millisecond
}
httpHeader.Set("Timeout", reqTimeout.String())
if md, ok := ctx.Value(constant.DUBBOGO_CTX_KEY).(map[string]string); ok {
diff --git a/protocol/mock/mock_invoker.go b/protocol/mock/mock_invoker.go
index 557dafa277e95c39d0b960436ac10ba8842c9186..c509cef054f5a23fe504486e01d7cc0e8772711d 100644
--- a/protocol/mock/mock_invoker.go
+++ b/protocol/mock/mock_invoker.go
@@ -1,3 +1,19 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+//
+
// Code generated by MockGen. DO NOT EDIT.
// Source: invoker.go
diff --git a/registry/etcdv3/listener.go b/registry/etcdv3/listener.go
index e0dc09908567ffcd8b8f77a9627c83876906e792..31d62fa916e5659cf424839cedf8f063fabedaa0 100644
--- a/registry/etcdv3/listener.go
+++ b/registry/etcdv3/listener.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package etcdv3
import (
diff --git a/registry/etcdv3/listener_test.go b/registry/etcdv3/listener_test.go
index 00024d28949f8b517d49f45ce6f16422d67b0a6b..c064f99c6c4b447a6c81093b87d99e1d1ba6d17a 100644
--- a/registry/etcdv3/listener_test.go
+++ b/registry/etcdv3/listener_test.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package etcdv3
import (
diff --git a/registry/etcdv3/registry.go b/registry/etcdv3/registry.go
index 4ee90969e57fc50344c914f5332134e6f7f01b89..b058113c69b8007803a8a18c1b5e0c3af8c184f4 100644
--- a/registry/etcdv3/registry.go
+++ b/registry/etcdv3/registry.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package etcdv3
import (
diff --git a/registry/etcdv3/registry_test.go b/registry/etcdv3/registry_test.go
index 6da9ad9d7d13bb0b6f8d1dab9b582669735ceec8..3f8c0f4cfccc2bcc68fc1e55fa69d74e9f0f8c0f 100644
--- a/registry/etcdv3/registry_test.go
+++ b/registry/etcdv3/registry_test.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package etcdv3
import (
diff --git a/registry/nacos/listener.go b/registry/nacos/listener.go
index 1f264ec9e4ba9b8e970eba7da3b46d8792831dea..25cd3d09b5711e4e7db56cd8e40f3283f3252e10 100644
--- a/registry/nacos/listener.go
+++ b/registry/nacos/listener.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package nacos
import (
diff --git a/registry/nacos/registry.go b/registry/nacos/registry.go
index 810a1cb05fec780868cf7767a9fcf8a7ccbd2b41..a8b9fa83fa73858064e570722341c14f974f5c9e 100644
--- a/registry/nacos/registry.go
+++ b/registry/nacos/registry.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package nacos
import (
diff --git a/registry/nacos/registry_test.go b/registry/nacos/registry_test.go
index 023ff788091c0c0f7c83ab213d8ab52006cfdc81..e6ab693cd3f5432fe30c2b83011cd56e44ac509f 100644
--- a/registry/nacos/registry_test.go
+++ b/registry/nacos/registry_test.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package nacos
import (
diff --git a/registry/protocol/protocol.go b/registry/protocol/protocol.go
index 534a4b945965f332e49ff343557fa20355921454..ffdb2753d6bfa0712b8fb9c962c8433a5c281083 100644
--- a/registry/protocol/protocol.go
+++ b/registry/protocol/protocol.go
@@ -338,10 +338,10 @@ func setProviderUrl(regURL *common.URL, providerURL *common.URL) {
}
func GetProtocol() protocol.Protocol {
- if regProtocol != nil {
- return regProtocol
+ if regProtocol == nil {
+ regProtocol = newRegistryProtocol()
}
- return newRegistryProtocol()
+ return regProtocol
}
type wrappedInvoker struct {
diff --git a/registry/protocol/protocol_test.go b/registry/protocol/protocol_test.go
index 0c19da59df6e4fd2f663f9e8d541165fe26c3ffa..761d14006680a3e0f3a111458d32155b19c26968 100644
--- a/registry/protocol/protocol_test.go
+++ b/registry/protocol/protocol_test.go
@@ -291,3 +291,8 @@ func TestExportWithApplicationConfig(t *testing.T) {
v2, _ := regProtocol.bounds.Load(getCacheKey(newUrl))
assert.NotNil(t, v2)
}
+
+func TestGetProtocol(t *testing.T) {
+ singleton := GetProtocol()
+ assert.True(t, singleton == GetProtocol())
+}
diff --git a/registry/zookeeper/listener.go b/registry/zookeeper/listener.go
index 857421f07706d6bdfec5a3ec21ba674627633458..5a4cc2c66e506360c02b9289f0606692ac168a23 100644
--- a/registry/zookeeper/listener.go
+++ b/registry/zookeeper/listener.go
@@ -92,7 +92,7 @@ func (l *RegistryConfigurationListener) Next() (*registry.ServiceEvent, error) {
return nil, perrors.New("listener stopped")
case <-l.registry.done:
- logger.Warnf("zk consumer register has quit, so zk event listener exit asap now.")
+ logger.Warnf("zk consumer register has quit, so zk event listener exit now.")
return nil, perrors.New("listener stopped")
case e := <-l.events:
@@ -109,7 +109,13 @@ func (l *RegistryConfigurationListener) Next() (*registry.ServiceEvent, error) {
}
}
func (l *RegistryConfigurationListener) Close() {
- l.registry.wg.Done()
+ if l.registry.IsAvailable() {
+ /**
+ * if the registry is not available, it means that the registry has been destroy
+ * so we don't need to call Done(), or it will cause the negative count panic for registry.wg
+ */
+ l.registry.wg.Done()
+ }
}
func (l *RegistryConfigurationListener) valid() bool {
diff --git a/remoting/etcdv3/client.go b/remoting/etcdv3/client.go
index 57d1211fe30e00dcb1ad16733f36b7969ebaf505..050968565387fd31871b0aa8e9969496d39f6534 100644
--- a/remoting/etcdv3/client.go
+++ b/remoting/etcdv3/client.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package etcdv3
import (
diff --git a/remoting/etcdv3/client_test.go b/remoting/etcdv3/client_test.go
index 187789e0abfac6a0e195bebd68ce4b91e0f9bdec..8f9b80cd30a9791709a0b2e83b9e59e0046f4c6c 100644
--- a/remoting/etcdv3/client_test.go
+++ b/remoting/etcdv3/client_test.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package etcdv3
import (
diff --git a/remoting/etcdv3/facade.go b/remoting/etcdv3/facade.go
index e75b39d6bcd7f67f7606c6b212f59e7a42178fd8..499044b8d77d3dcd8d32b0cb70cb78f84fae8ec4 100644
--- a/remoting/etcdv3/facade.go
+++ b/remoting/etcdv3/facade.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package etcdv3
import (
diff --git a/remoting/etcdv3/listener.go b/remoting/etcdv3/listener.go
index f5401917e208f41d3ea84e47c46f535b344e2784..a4d5805a6dbf3c76f43cb6085653c791b33ab119 100644
--- a/remoting/etcdv3/listener.go
+++ b/remoting/etcdv3/listener.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package etcdv3
import (
diff --git a/remoting/etcdv3/listener_test.go b/remoting/etcdv3/listener_test.go
index 33904a21345ec0ac7ee1adbb239a0a7a44852387..7da85819730eaeb1c6931b0e13de26e2e171bec2 100644
--- a/remoting/etcdv3/listener_test.go
+++ b/remoting/etcdv3/listener_test.go
@@ -1,3 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
package etcdv3
import (