Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Classloading issues with jboss-marshalling and redisson-tomcat #2984

Closed
ghollies opened this issue Aug 17, 2020 · 5 comments
Closed

Classloading issues with jboss-marshalling and redisson-tomcat #2984

ghollies opened this issue Aug 17, 2020 · 5 comments
Labels
bug
Milestone
3.13.4

Comments

@ghollies
Copy link
Contributor

ghollies commented Aug 17, 2020 •
edited

Expected behavior
When using redisson-tomcat to provide a SessionManager for our tomcat installation, we expect the classloader used to instantiate objects should be the current thread classloader, regardless of which Codec is being used. Using FSTCodec this worked as expected. This is to allow the right classloader, Tomcat vs WAR classloader to create the objects which it knows how to create.

Actual behavior
After the switch to the MarshallingCodec as default we are running into issues with the RiverUnmarshaller is throwing errors after calling classResolver.resolveClass(). When a thread from my WAR attempts to unmarshall a class it has loaded, for example org.springframework.security.web.savedrequest.DefaultSavedRequest the classresolver on the RiverUnmarshaller for some threads points to tomcat’s Classloader (URLClassLoader with apache-tomcat paths)

This RiverUnmashaller is a FastThreadLocal<Unmarshaller> decoderThreadLocal which is field on the MarshallingCodec.
I can see in the debugger that the MarshallingCodec has classLoader field of my WAR’s class loader, not a reference to tomcat’s. The configuration object on the same MarshallingCodec also points to the WAR class loader.

The number of threads in this broken state where the classloader being used to unmarshall isn't shared by the MarshallingCodec seems to change on each run, but reproduced pretty consistently.
image

Steps to reproduce or test case
Running Tomcat 8.5.57.0, deploy a WAR with a context xml with a manager as described https://github.com/redisson/redisson/tree/master/redisson-tomcat#1-add-redissonsessionmanager
Place objects onto the session and attempt to retrieve them that are instances of classes not on the tomcat classpath directly (only within the war, spring security for example)
See requests fail sometimes, depending which thread picks up the work.

Redis version
6.0.6

Redisson version
3.13.3

Redisson configuration

singleServerConfig:
  address: "redis://redis-server:6379"
  password: "${REDIS_PASSWORD}"
@mrniko mrniko added this to the 3.13.4 milestone Aug 18, 2020
@mrniko mrniko added the bug label Aug 18, 2020
@mrniko
Copy link
Member

mrniko commented Aug 18, 2020

Thanks for report! Please try both MarshallingCodecV2 and MarshallingCodec codecs included in attached version.

redisson-3.13.4-SNAPSHOT.jar.zip

@ghollies
Copy link
Contributor Author

Thanks for the quick feedback!
I tried both Codec's out this morning and still ran into the same issue. It felt harder to reproduce on V2 (it would work a little more often). By using this Spring class to trigger it, I'm just hitting f5 on my login page I have setup (each time triggering a SavedRequest to be stored on my session), but given that i only restarted on each Codec a couple of times each, the sample size might not mean much.

Here are the two stack traces,

MarshallingCodec from redisson-3.13.4-SNAPSHOT.jar.zip

org.redisson.client.RedisException: Unexpected exception while processing command
		at org.redisson.command.CommandAsyncService.convertException(CommandAsyncService.java:350)
		at org.redisson.command.CommandAsyncService.get(CommandAsyncService.java:147)
		at org.redisson.RedissonObject.get(RedissonObject.java:90)
		at org.redisson.RedissonMap.get(RedissonMap.java:267)
		at org.redisson.tomcat.RedissonSession.getAttribute(RedissonSession.java:111)
		at org.apache.catalina.session.StandardSessionFacade.getAttribute(StandardSessionFacade.java:102)
		at org.springframework.security.web.savedrequest.HttpSessionRequestCache.getRequest(HttpSessionRequestCache.java:73)
		at org.springframework.security.web.savedrequest.HttpSessionRequestCache.getMatchingRequest(HttpSessionRequestCache.java:91)
		at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:60)
		… Filter Chain
		at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:524)
		at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
		at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
		at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1626)
		at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.lang.Thread.run(Thread.java:748)
	Caused by: java.io.IOException: java.lang.ClassNotFoundException: org.springframework.security.web.savedrequest.DefaultSavedRequest
		at org.redisson.codec.MarshallingCodec$3.decode(MarshallingCodec.java:153)
		at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:378)
		at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:196)
		at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:134)
		at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:104)
		at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
		at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		... 1 more
	Caused by: java.lang.ClassNotFoundException: org.springframework.security.web.savedrequest.DefaultSavedRequest
		at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
		at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
		at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
		at java.lang.Class.forName0(Native Method)
		at java.lang.Class.forName(Class.java:348)
		at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:129)
		at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:110)
		at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:1033)
		at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1366)
		at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:283)
		at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:216)
		at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
		at org.redisson.codec.MarshallingCodec$3.decode(MarshallingCodec.java:151)

MarshallingCodecV2:

org.redisson.client.RedisException: Unexpected exception while processing command
		at org.redisson.command.CommandAsyncService.convertException(CommandAsyncService.java:350)
		at org.redisson.command.CommandAsyncService.get(CommandAsyncService.java:147)
		at org.redisson.RedissonObject.get(RedissonObject.java:90)
		at org.redisson.RedissonMap.get(RedissonMap.java:267)
		at org.redisson.tomcat.RedissonSession.getAttribute(RedissonSession.java:111)
		at org.apache.catalina.session.StandardSessionFacade.getAttribute(StandardSessionFacade.java:102)
		at org.springframework.security.web.savedrequest.HttpSessionRequestCache.getRequest(HttpSessionRequestCache.java:73)
		at org.springframework.security.web.savedrequest.HttpSessionRequestCache.getMatchingRequest(HttpSessionRequestCache.java:91)
		at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:60)
		… Filter Chain
		at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:524)
		at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
		at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
		at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1626)
		at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.lang.Thread.run(Thread.java:748)
	Caused by: java.io.IOException: java.lang.ClassNotFoundException: org.springframework.security.web.savedrequest.DefaultSavedRequest
		at org.redisson.codec.MarshallingCodecV2$3.decode(MarshallingCodecV2.java:152)
		at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:378)
		at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:196)
		at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:134)
		at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:104)
		at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
		at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		... 1 more
	Caused by: java.lang.ClassNotFoundException: org.springframework.security.web.savedrequest.DefaultSavedRequest
		at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
		at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
		at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
		at java.lang.Class.forName0(Native Method)
		at java.lang.Class.forName(Class.java:348)
		at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:129)
		at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:110)
		at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:1033)
		at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1366)
		at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:283)
		at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:216)
		at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
		at org.redisson.codec.MarshallingCodecV2$3.decode(MarshallingCodecV2.java:150)
		... 23 more

@mrniko
Copy link
Member

mrniko commented Aug 20, 2020

Please try MarshallingCodec with version below.

redisson-3.13.4-SNAPSHOT.jar.zip

@ghollies
Copy link
Contributor Author

I can no longer reproduce the issue using that jar 🎉
It seem to work for me!

mrniko pushed a commit that referenced this issue Aug 20, 2020
Fixed - Classloading issues when MarshallingCodec used in Tomcat. #2984
6edad47
@mrniko
Copy link
Member

mrniko commented Aug 20, 2020

Fixed

@mrniko mrniko closed this as completed Aug 20, 2020
@mrniko mrniko mentioned this issue Sep 2, 2020
Closed
@nab0310 nab0310 mentioned this issue Mar 17, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Milestone
3.13.4
Development

No branches or pull requests
2 participants
@mrniko @ghollies