memcache tomcat 集群

Search    Search within:       All wiki pages    Featured pages    Current pages         Deprecated pages      for  
Configure logging

 

Introduction

For the most simple integration you just need to have a tomcat (6, 7 or 8) and a memcached installed (or s.th. supporting the memcached protocol). In your production environment you probably will have several tomcats and you should also have several memcached nodes available, on different pieces of hardware. You can use sticky sessions or non-sticky sessions, memcached-session-manager (msm) supports both operation modes.

The following description shows an example for a setup with sticky sessions, with two instances of tomcat and two instances of memcached installed.

Tomcat-1 (t1) will primarily store its sessions in memcached-2 (m2) which is running on another machine (m2 is a regular node for t1). Only if m2 is not available, t1 will store its sessions in memcached-1 (m1, m1 is the failoverNode for t1). With this configuration, sessions wont be lost when machine 1 (serving t1 and m1) crashes. The following really nice ASCII art shows this setup.

 t1   t2 

  . \ / .

  .  X  .

  . / \ .

 m1   m2 

So what needs to be done for this?

Decide which serialization strategy to use

There are several session serialization strategies available, as they are described on SerializationStrategies. The default strategy uses Javaserialization and is already provided by the memcached-session-manager jar. Other strategies are provided by separate jars, in the section below youll see which jars are required for which strategy.

Configure tomcat

The configuration of tomcat requires two things: you need to drop some jars in your $CATALINA_HOME/lib/ and WEB-INF/lib/ directories and you have to configure the memcached session manager in the related  Context  element (e.g. in META-INF/context.xml inside the application files).

Add memcached-session-manager jars to tomcat

Independent of the chosen serialization strategy you always need the memcached-session-manager-${version}.jar and either memcached-session-manager-tc6-${version}.jar for tomcat6, memcached-session-manager-tc7-${version}.jar for tomcat7 (attention: tomcat 7.0.23+) ormemcached-session-manager-tc8-${version}.jar for tomcat8.

If youre using memcached, you also need the spymemcached-2.11.1.jar.

If youre using couchbase, you need additionally these jars: couchbase-client-1.4.0.jar jettison-1.3.jar, commons-codec-1.5.jar, httpcore-4.3.jar,httpcore-nio-4.3.jar, netty-3.5.5.Final.jar.

Please download the appropriate jars and put them in $CATALINA_HOME/lib/.

Add custom serializers to your webapp (optional)

If you want to use javas built in serialization nothing more has to be done. If you want to use a custom serialization strategy (e.g. because ofbetter performance) this has to be deployed with your webapp so that theyre available in WEB-INF/lib/.

As msm is available in maven central (under groupId de.javakaffee.msm) you can just pull it in using the dependency management of your build system. With maven you can use this dependency definition for the kryo-serializer:

 dependency 

    groupId de.javakaffee.msm /groupId 

    artifactId msm-kryo-serializer /artifactId 

    version 1.8.0 /version 

    scope runtime /scope 

 /dependency 

For javolution the artifactId is msm-javolution-serializer, for xstream msm-xstream-serializer and for flexjson its msm-flexjson-serializer.

If youre not using a dependency management based on maven repositories these are the jars you need for the different serializers:

kryo-serializer: msm-kryo-serializer, kryo-serializers-0.11 (0.11 is needed, as 0.20+ is for kryo2), kryo, minlog, reflectasm, asm-3.2 javolution-serializer: msm-javolution-serializer, javolution-5.4.3.1 xstream-serializer: msm-xstream-serializer, xstream, xmlpull, xpp3_min flexjson-serializer: msm-flexjson-serializer, flexjson
Configure memcached-session-manager as  Context  Manager

Update the  Context  element (in META-INF/context.xml or what else you choose for context definition, please check the related tomcat documentation for this) so that it contains the Manager configuration for the memcached-session-manager, like in the examples below.

The following examples show configurations for sticky sessions and non-sticky sessions with memcached servers and for non-sticky sessions with membase. The examples with memcached servers assume that there are two memcacheds running, one on host1 and another one on host2. All sample configurations assume that you want to use kryo based serialization.

Example for sticky sessions + kryo

The following example shows the configuration of the first tomcat, assuming that it runs on host1, together with memcached "n1". The attributefailoverNodes="n1" tells msm to store sessions preferably in memcached "n2" and only store sessions in "n1" (running on the same host/machine) if no other memcached node (here only n2) is available (even if host1 goes down completely, the session is still available in memcached "n2" and could be served by the tomcat on host2). For the second tomcat (on host2) you just need to change the failover node to "n2", so that it prefers the memcached "n1". Everything else should be left unchanged.

 Context 

  ...

  ManagerclassName="de.javakaffee.web.msm.MemcachedBackupSessionManager"

    memcachedNodes="n1:host1.yourdomain.com:11211,n2:host2.yourdomain.com:11211"

    failoverNodes="n1"

    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

    transcoderFactory     / 

 /Context 

Example for non-sticky sessions + kryo

The following example shows a configuration for non-sticky sessions. In this case theres no need for failoverNodes, as sessions are served by all tomcats round-robin and theyre not bound to a single tomcat. For non-sticky sessions the configuration (for both/all tomcats) would look like this:

 Context 

  ...

  ManagerclassName="de.javakaffee.web.msm.MemcachedBackupSessionManager"

    memcachedNodes="n1:host1.yourdomain.com:11211,n2:host2.yourdomain.com:11211"

    sticky="false"

    sessionBackupAsync="false"

    lockingMode="uriPattern:/path1|/path2"

    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

    transcoderFactory     / 

 /Context 

Example for non-sticky sessions with couchbase + kryo

To connect to a membase bucket "bucket1" the configuration would look like this:

 Context 

  ...

  ManagerclassName="de.javakaffee.web.msm.MemcachedBackupSessionManager"

    memcachedNodes="http://host1.yourdomain.com:8091/pools"

    username="bucket1"

    password="topsecret"

    memcachedProtocol="binary"

    sticky="false"

    sessionBackupAsync="false"

    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

    transcoderFactory     / 

 /Context 

Example for multiple contexts sharing the same session id

If you are running multiple webapps/contexts sharing the same session id (e.g. by having set sessionCookiePath="/" - oremptySessionPath="true" in tomcat 6) you must tell memcached session manager to add a prefix to the session id when storing a session in memcached. For this you can use the storageKeyPrefix attribute as shown by this example (see also the more detailed attribute description below):

 ContextsessionCookiePath="/" 

  ...

  ManagerclassName="de.javakaffee.web.msm.MemcachedBackupSessionManager"

    memcachedNodes="n1:host1.yourdomain.com:11211,n2:host2.yourdomain.com:11211"

    failoverNodes="n1"

    storageKeyPrefix="context"

    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

    transcoderFactory     / 

 /Context 

More details for all configuration attributes are provided in the section below.

After you have configured msm in the Context/Manager element, you can just start your application and sessions will be stored in the memcached nodes or in membase as configured. Now you should do some tests with a simulated tomcat failure, a restart of a memcached node etc. - have fun! :-)

Overview over memcached-session-manager configuration attributes

className (required)

This should be set to de.javakaffee.web.msm.MemcachedBackupSessionManager to get sessions stored in memcached. However, since version 1.3.6 theres also a de.javakaffee.web.msm.DummyMemcachedBackupSessionManager that can be used for development purposes: it simply serializes sessions to a special in memory map and deserializes the serialized data at the next request when a session is requested (without really using this session) - just to see if deserialization is working for each request. Your application is still using sessions as if the memcached-session-manager (or DummyMemcachedBackupSessionManager) would not be existing. Session serialization/deserialization is done asynchronously. The configuration attributes memcachedNodes and failoverNode are not used to create a memcached client, so serialized session data will not be sent to memcached - and therefore no running memcacheds are required.

 

memcachedNodes (required)

This attribute must contain all memcached nodes you have running, or the membase bucket uri(s). It should be the same for all tomcats.

memcached nodes: Each memcached node is defined as  id : host : port . Several definitions are separated by space or comma (e.g. memcachedNodes="n1:app01:11211,n2:app02:11211"). For a single node the  id  is optional so that its allowed to set only host : port  (e.g. memcachedNodes="localhost:11211"). Then the sessionId will be left unchanged (no node id appended). This option is useful for the usage with e.g. membase+moxi, where each tomcat knows only a single "memcached" node (moxi actually).

membase bucket uris (since 1.6.0): For usage with membase its possible to specify one or more membase bucket uris, e.g.http://host1:8091/pools,http://host2:8091/pools. Bucket name and password are specified via the attributes username andpassword (see below). Connecting to membase buckets requires the binary memcached protocol. Also remember to have thejettison.jar and netty.jar available in CATALINA_HOME/lib/.

 

failoverNodes (optional, must not be used for non-sticky sessions)

This attribute must contain the ids of the memcached nodes that must only be used for session backup when none of the other memcached nodes are available. Therefore, you should list those memcached nodes, that are running on the same machine as this tomcat. E.g. if you have tomcat1 and memcached1 (n1) running on host1, and tomcat2 and memcached2 (n2) on host2, then you set n1 as failover node for tomcat1, so that tomcat1 does only store sessions in memcached1 if memcached2 is not available (for tomcat2 failoverNodes would be set to n2). With this setting host1 could completely crash and sessions from tomcat1 would still be available.

 

For non-sticky sessions failoverNodes must not be specified as a session is not tied to a single tomcat. For membase buckets this attribute should also be left out. Several memcached node ids are separated by space or comma.

username (since 1.6.0, optional)

Specifies the username used for a membase bucket or SASL. If the memcachedNodes contains a membase bucket uri (or multiple) this is the bucket name. If the memcachedNodes contains memcached node definitions this is the username used for SASL authentication. Both require the binary memcached protocol.

 

password (since 1.6.0, optional)

Specifies the password used for membase bucket or SASL authentication (can be left empty / omitted if the "default" membase bucket without a password shall be used).

 

memcachedProtocol (since 1.3, optional, default text)

This attribute specifies the memcached protocol to use, one of text or binary.

 

sticky (since 1.4.0, optional, default true)

Specifies if sticky or non-sticky sessions are used.

 

lockingMode (since 1.4.0, optional, for non-sticky sessions only, default none)

Specifies the locking strategy for non-sticky sessions. Session locking is useful to prevent concurrent modifications and lost updates of the session in the case of parallel requests (tabbed browsing with long requests, ajax etc.). Session locking is done using memcached. Possible values for lockingMode are: none: sessions are never locked all: the session is locked when requested by the app until the end of the request auto: readonly requests are detected, for them the session is not locked. For requests that are not classified as "readonly" the session is locked uriPattern: regexp : the provided regular expression pattern is compared with the requestURI + "?" + queryString and the session is locked if they match.

requestUriIgnorePattern (optional)

This attribute contains a regular expression for request URIs, that shall not trigger a session backup. If static resources like css, javascript, images etc. are delivered by the same tomcat and the same web application context these requests will also pass the memcached-session-manager. However, as these requests should not change anything in a http session, they should also not trigger a session backup. So you should check if any static resources are delivered by tomcat and in this case you should exclude them by using this attribute. The requestUriIgnorePattern must follow the java regex Pattern.

 

sessionBackupAsync (optional, default true)

Specifies if the session shall be stored asynchronously in memcached. If this is true, the backupThreadCount setting is evaluated. If this is false, the timeout set via sessionBackupTimeout is evaluated.

 

backupThreadCount (since 1.3, optional, default number-of-cpu-cores)

The number of threads that are used for asynchronous session backup (if sessionBackupAsync="true"). For the default value the number of available processors (cores) is used.

 

sessionBackupTimeout (optional, default 100)

The timeout in milliseconds after that a session backup is considered as beeing failed. This property is only evaluated if sessions are stored synchronously (set via sessionBackupAsync). The default value is 100 milliseconds.

 

operationTimeout (since 1.6.0, optional, default 1000)

The memcached operation timeout used at various places, e.g. used for the spymemcached ConnectionFactory.

 

storageKeyPrefix (since 1.8.0, optional, default webappVersion)

Allows to configure a prefix thats added to the session id when a session is stored in memcached. This is useful if there are multiple webapps sharing the same session id, e.g. because contexts are configured with sessionCookiePath="/" (tomcat 7, oremptySessionPath="true" in tomcat 6)

 

When tomcats parallel deployment is used there are also multiple webapps sharing the same session id. This is supported by default, because the default value (if storageKeyPrefix is not set explicitely) is webappVersion, so that the webapp version is used as prefix for the session id / key in memcached. The storageKeyPrefix attribute supports both a static prefix (static:somePrefix) and dynamic values (host, context,webappVersion), multiple values can be specified seperated by comma. Here are some examples that demonstrate which config would create which storage key (for a session id "foo" with context path "ctxt" and host "hst"): static:x -  x_foo host -  hst_foo host.hash -  e93c085e_foo context -  ctxt_foo context.hash -  45e6345f_foo host,context -  hst:ctxt_foo webappVersion -  001_foo host.hash,context.hash,webappVersion -  e93c085e:45e6345f:001_foo

sessionAttributeFilter (since 1.5.0, optional)

A regular expression to control which session attributes are serialized to memcached. The regular expression is evaluated with session attribute names. E.g. sessionAttributeFilter="^(userName|sessionHistory)$" specifies that only "userName" and "sessionHistory" attributes are stored in memcached. Works independently from the chosen serialization strategy.

 

transcoderFactoryClass (since 1.1, optional, default de.javakaffee.web.msm.JavaSerializationTranscoderFactory)

The class name of the factory that creates the transcoder to use for serializing/deserializing sessions to/from memcached. The specified class must implement de.javakaffee.web.msm.TranscoderFactory and provide a no-args constructor. Other TranscoderFactoryimplementations are available through other packages/jars like msm-kryo-serializer, msm-xstream-serializer and msm-javolution-serializer (as describe above), those are listed and compared on SerializationStrategies.

 

Available TranscoderFactory implementations:

Java serialization: de.javakaffee.web.msm.JavaSerializationTranscoderFactory Kryo based serialization: de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory Javolution based serialization: de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory XStream based serialization: de.javakaffee.web.msm.serializer.xstream.XStreamTranscoderFactory

copyCollectionsForSerialization (since 1.1, optional, default false)

A boolean value that specifies, if iterating over collection elements shall be done on a copy of the collection or on the collection itself. This configuration property must be supported by the serialization strategy specified with transcoderFactoryClass. Which strategy supports this feature can be seen in the column Copy Collections before serialization in the list of available serialization strategies. This feature and its motivation is described more deeply at SerializationStrategies#Concurrent_modifications_of_collections.

 

customConverter (since 1.2, optional)

Custom converter allow you to provide custom serialization of application specific types. Multiple custom converter class names are specified separated by comma (with optional space following the comma). Converter classes must be available in the classpath of the web application (place jars in WEB-INF/lib).

This option is useful if you need some special serialization for a certain type or if reflection based serialization is just very verbose and you want to provide a more efficient serialization for a specific type.

Custom converter must be supported by the serialization strategy specified with transcoderFactoryClass. Requirements regarding the specific custom converter classes depend on the actual serialization strategy, but a common requirement would be that they must provide a default/no-args constructor. For more details have a look at available serialization strategies.

 

Available converter implementations:

Kryo (converters provided by msm-kryo-serializer jar) de.javakaffee.web.msm.serializer.kryo.JodaDateTimeRegistration: A more efficient serialization of Jodas DateTimewith kryo. de.javakaffee.web.msm.serializer.kryo.CGLibProxySerializerFactory: serializes/deserializes CGLIB proxies. de.javakaffee.web.msm.serializer.kryo.HibernateCollectionsSerializerFactory: serializes/deserializes hibernate persistent collections (required if you store collections loaded by hibernate in your session). de.javakaffee.web.msm.serializer.kryo.WicketSerializerFactory: required if youre running a wicket web application. de.javakaffee.web.msm.serializer.kryo.FacesLRUMapRegistration: Needed if kryo serialization shall be used in combination with JSF2/Mojarra (see also issue #97). de.javakaffee.web.msm.serializer.kryo.GrailsFlashScopeRegistration: Support for grails flash scope (see also issue #107). de.javakaffee.web.msm.serializer.kryo.SpringSecurityUserRegistration: Support for Spring Security User class (see also issue #145).
Javolution (converters provided by msm-javolution-serializer jar) de.javakaffee.web.msm.serializer.javolution.JodaDateTimeFormat: A more efficient serialization of Jodas DateTime, see also issue #32. de.javakaffee.web.msm.serializer.javolution.CGLibProxyFormat : serializes/deserializes CGLIB proxies, see alsoissue #59. de.javakaffee.web.msm.serializer.javolution.HibernateCollectionsXMLFormat: serializes/deserializes hibernate persistent collections (required if you store collections loaded by hibernate in your session).

enableStatistics (since 1.2, optional, default true)

A boolean value that specifies, if statistics shall be gathered. For more info see the JMXStatistics page.

 

enabled (since 1.4.0, optional, default true)

Specifies if session storage in memcached is enabled or not, can also be changed at runtime via JMX. Only allowed in sticky mode.

 

Overview over available system properties (all optional)

msm.maxReconnectDelay (since 1.7, default 30)

The max reconnect delay for the spymemcached MemcachedClient, in seconds.

 

msm.nodeAvailabilityCacheTTL (since 1.7, default 1000)

The TTL for msm NodeAvailabilityCache entries, in millis.

 

msm.kryo.bufferSize.initial (since 1.1, default 102400)

The initial size in bytes of the kryo object buffer when (de)serializing a session. Is passed as initialCapacity when creating a kryoObjectBuffer.

 

msm.kryo.bufferSize.max (since 1.1, default 2048000)

The maximum size in bytes of the kryo object buffer when (de)serializing a session - the maximum size in bytes that a serialized session can take. Is passed as maxCapacity when creating a kryo ObjectBuffer.

 

msm.kryo.defaultSerializerFactory (since 1.8, default de.javakaffee.web.msm.serializer.kryo.ReferenceFieldSerializerFactory)

The factory to create the Serializer used by Kryo when Kryo#newDefaultSerializer is called. Must implementde.javakaffee.web.msm.serializer.kryo.KryoDefaultSerializerFactory.

 

Currently available implementations: de.javakaffee.web.msm.serializer.kryo.ReferenceFieldSerializerFactory: Factory to create a ReferenceFieldSerializer, handling object references and cyclic graphs. de.javakaffee.web.msm.serializer.kryo.CompatibleFieldSerializerFactory: Factory to create aCompatibleFieldSerializer, with limited support for forward and backward compatibility.
Configure logging

If you want to enable fine grained / debug logging you can add

de.javakaffee.web.msm.level=FINE

to $CATALINA_HOME/conf/logging.properties.

As the memcached-session-manager uses spymemcached also the logging hints of spymemcached might be interesting to you. A short summary how you can make spymemcached more silent:

Add the following to $CATALINA_HOME/bin/catalina.sh:

 

CATALINA_OPTS="-Dnet.spy.log.LoggerImpl=net.spy.memcached.compat.log.SunLogger"

Add this to $CATALINA_HOME/conf/logging.properties:

# A handlers log level threshold can be set using SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST or ALL

net.spy.memcached.level = WARNING

# To make only the MemcachedConnection less verbose:

#net.spy.memcached.MemcachedConnection.level = WARNING

More info about logging in tomcat can be found in the tomcat logging documentation.

Tomcat集群session复制与Oracle的坑。。 问题描述 公司某个系统使用了tomcat自带的集群session复制功能，然后后报了一个oracle驱动包里面的连接不能被序列化的异常。
基于Docker部署 Tomcat集群、 Nginx负载均衡 当作一百世一样。这里的道理很明白:我思故我在,既然我存在,就不能装作不存在。无论如何,我要为自己负起责任。——王小波《三十而立》
1、系统环境变量配置： 首先要实现Tomcat的集群就得拥有多个tomcat，所以我在本地电脑下载了两个Tomcat，我这里使用的是Tomcat7，当然，配置与Tomcat的版本没多大关系~
Grafana+Telegraf+Influxdb监控Tomcat集群方案 前一段时间自家养的几只猫经常出问题，由于没有有效的监控预警手段，以至于问题出现或者许久一段时间才会被通知到。凌晨一点这个锅可谁都不想背，为此基于目前的情况搭建了以下这么一套监控预警系统。 Nginx：代理访问 Grafana Grafana： 可视化面板(Dashboard)，有.
nginx反向代理和tomcat集群(适用于ubutnu16.04及其centos7) 下面示例，本人亲测有效 为什么要反向代理和集群？ 因为并发问题，很多请求如果全部分发给一个tomcat,一个tomcat优化最好的话，据说可达到800负载，但是面对成千上万的请求，单单一个tomcat是不行的，不管是apr还是nio,面对高并发高负载时，采取Nginx反向代理策略是最好的。
反向代理负载均衡（如Apache+JK2+Tomcat这种组合） 使用代理服务器可以将请求转发给内部的Web服务器，让代理服务器将请求均匀地转发给多台内部Web服务器之一上，从而达到负载均衡的目的。