A distributed in-memory data store for the cloud

Related tags

EVCache
Overview

EVCache

Join the chat at https://gitter.im/Netflix/EVCache

EVCache is a memcached & spymemcached based caching solution that is mainly used for AWS EC2 infrastructure for caching frequently used data.

EVCache is an abbreviation for:

  • Ephemeral - The data stored is for a short duration as specified by its TTL (Time To Live).
  • Volatile - The data can disappear any time (Evicted).
  • Cache - An in-memory key-value store.

ScreenShot

Issues
  • EvCache - Eureka

    EvCache - Eureka

    I would like to ask question related with the EvCache and Eureka setup. I spent quite a while checking EvCache - testing and reading about different setup's. At this moment, I am not sure what is the best implementation with Eureka.

    Basically, sample deployment on wiki page explains how to run this on localhost (or on ec2 instances) with 2 replicas (with 2 shards each). But with hard coded memcached replicas/shards endpoint.

    In case I want to use dynamic method of discovering servers with memcached, what I need to do in order to initialize EvCache client?

    Right now, for example in Sample deployment we are creating instance of the EvCache using following: deploymentDescriptor = "SERVERGROUP1=localhost:11211;SERVERGROUP2=localhost:11212"; System.setProperty("EVCACHE_APP1-NODES", deploymentDescriptor); evCache = new EVCache.Builder().setAppName("EVCACHE_APP1").build();

    In case of the Eureka, do we need to use discovery service manually and then recreate instance of evCache or this is done automatically? Is there any kind of technical wiki document which explains this setup little bit more detailed?

    opened by mersed 11
  • Difference in hashing algorithm for evache-client v1.0.5 vs v4.38.0

    Difference in hashing algorithm for evache-client v1.0.5 vs v4.38.0

    Hello, After trying to upgrade evcache-client from v 1.0.5 to v 4.38.0 we faced found increase in cache misses during Blue-Green deployment, on environments using more then one cache nodes. Performing test where number of SETs performed from old client and GETs from new client, we can see that in ~50% of cases new client misses values. Backward test ( SETs from new and GETs from old) show almost same results. On environments with single cache node given tests pass 100% correctly. Both clients using default - KETAMA hashing algorithm.

    Assuming, this we think that something in hashing algorithm implementation have been changed. Is this known issue, and are there some workaround?

    opened by akhitev 9
  • tomcat 7/8 crashes when using the evcache with much data

    tomcat 7/8 crashes when using the evcache with much data

    We use the evcache as the memcached client (one note). Recently, the Tomcat 7 (we upgrade to tomcat 8.5) down several times every day. I have spent much time on it. But I only find few clue in the tomcat log (there isn't hs_err_pid file. I think the jvm crashed before it wrote the log file).

    Could you help me on this issue?

    1. java runtime error.

    A fatal error has been detected by the Java Runtime Environment:

    SIGSEGV (0xb) at pc=0x00007f9959e43e10, pid=15465, tid=0x00007f9829cec700

    JRE version: Java(TM) SE Runtime Environment (8.0_151-b12) (build 1.8.0_151-b12)

    Java VM: Java HotSpot(TM) 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)

    Problematic frame:

    C [libc.so.6+0x81e10]2017-11-20 23:10:39.453 INFO com.netflix.evcache.EVCacheTranscoder: Compression increased the size of [B from 136 to 151

    [ timer expired, abort... ]

    1. Sometimes, tomcat crashes when the evcache reconnecting. Here is the log. 2017-11-20 20:58:18.970 INFO net.spy.memcached.EVCacheConnection: Reconnecting due to exception on {QA sa=m-bp175bd9d8160bc4.memcache.rds.aliyuncs.com/10.50.117.204:11211, #Rops=0, #Wops=0, #iq=54, topRop=null, topWop=null, toWrite=0, interested=1} java.lang.IllegalStateException: No read operation. at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:842) at net.spy.memcached.MemcachedConnection.handleReadsAndWrites(MemcachedConnection.java:732) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:697) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:450) at net.spy.memcached.EVCacheConnection.run(EVCacheConnection.java:50) 2017-11-20 20:58:18.972 WARN net.spy.memcached.EVCacheConnection: Closing, and reopening {QA sa=m-bp175bd9d8160bc4.memcache.rds.aliyuncs.com/10.50.117.204:11211, #Rops=0, #Wops=0, #iq=87, topRop=null, topWop=null, toWrite=0, interested=1}, attempt 0.
    opened by shengang1978 6
  • EVCache Touch method throw NoSuchMethodError exception

    EVCache Touch method throw NoSuchMethodError exception

    Here are the jar files version: EVCache Client: 4.104 spymemcached: 2.12.3

    I use the EVCache.touch(key, 120000) to refresh the time. But it throws the below exception. java.lang.NoSuchMethodError: net.spy.memcached.OperationFactory.touch(Ljava/lang/String;ILnet/spy/memcached/ops/OperationCallback;)Lnet/spy/memcached/ops/KeyedOperation; net.spy.memcached.EVCacheMemcachedClient.touch(EVCacheMemcachedClient.java:322) com.netflix.evcache.pool.EVCacheClient.touch(EVCacheClient.java:1099) com.netflix.evcache.EVCacheImpl.touchData(EVCacheImpl.java:755) com.netflix.evcache.EVCacheImpl.touch(EVCacheImpl.java:712) com.netflix.evcache.EVCacheImpl.touch(EVCacheImpl.java:663) ...

    Do I use the wrong jar files?

    opened by shengang1978 5
  • Using Memcached with Prana

    Using Memcached with Prana

    I am facing difficulty trying to configure healthurl "prana.host.healthcheck.url" for memcached . I have one instance running on port 11211 for memcached and prana is running on 8078. My understanding is this url should be reachable but memcached doesn't have any direct url that can be polled from prana. Please suggest.

    opened by userswat 5
  • Spymemcached client version 2.8 and above is not compatible

    Spymemcached client version 2.8 and above is not compatible

    I found out through testing, that i managed to start up the EvCache client using spymemcached client version 2.7.3. Everything from 2.8 and up breaks the contract of the HashAlgorithm interface.

    It would be nice if this could work this latest version present in Maven central. They only uploaded versions 2.9 and later.

    opened by pparth 4
  • Cache calls taking 10 seconds

    Cache calls taking 10 seconds

    I have deployed evcache to a highish load production environment and we appeared to have some kind of network glitch where cache calls ended up taking 10 seconds per call (or multiples of 10 i.e. 20 or 30 seconds). Even though I had sent the max timeout to 100ms, this resulted in threads backing up and server grinding to a hault.

    As it was production we had very limited logging and I after extensive testing in QA I was unable to replicate the issue. So would anyone have any idea what could cause this?

    evcache version:1.0.5 spymemcache version:2.7.3

    ConfigurationManager.getConfigInstance(); EVCacheClientPoolManager.getInstance().initEVCache(applicationName);

    evCache = (new EVCache.Builder()).setAppName(applicationName).enableZoneFallback().build();

    Config EC2_AVAILABILITY_ZONE=zone1 ${evcache.pool.provider}=com.netflix.evcache.pool.standalone.ZoneClusteredEVCacheClientPoolImpl ${evcache.application.name}.${evcache.server.zone.a.name}.EVCacheClientPool.hosts">${evcache.server.zone.a.hostnames}=server1,server2,server3 ${evcache.application.name}.${evcache.server.zone.b.name}.EVCacheClientPool.hosts">${evcache.server.zone.b.hostnames}=server4,server5,server6 ${evcache.application.name}.EVCacheClientPool.poolSize">${evcache.connection.poolsize}=1 ${evcache.application.name}.EVCacheClientPool.readTimeout">${evcache.read.timeout}=100

    Logs Before problem We recieved a number of these warnings from a heartbeat thread we had running MemcachedConnection] - Could not redistribute to another node, retrying primary node for key11111111

    The about 1 minute later (where there were a number of successful cache calls) we started having the issue.

    Logs around problem time [ n.s.m.MemcachedConnection] - Closing, and reopening {QA sa=/serverip_1:serverport, #Rops=2, #Wops=162, #iq=0, topRop=Cmd: -1 Opaque: -1 Keys: , topWop=Cmd: 0 Opaque: 1865615 Key: Anotherkey_V11, toWrite=0, interested=5}, attempt 0. [ n.s.m.p.b.BinaryMemcachedNodeImpl] - Discarding partially completed op: Cmd: -1 Opaque: -1 Keys: [5ad1c8ab56d9445fa5aa4b0034c8c93d n.s.m.MemcachedConnection] - Could not redistribute to another node, retrying primary node for Anotherkey2_V11. [ n.s.m.MemcachedConnection] - Closing, and reopening {QA sa=/serverip_2:serverport, #Rops=2, #Wops=220, #iq=0, topRop=Cmd: 1 Opaque: 1865530 Key: cckey_V11 Cas: 0 Exp: 0 Flags: 1 Data Length: 222, topWop=Cmd: 0 Opaque: 1865532 Key: cckey_V11, toWrite=0, interested=8}, attempt 0. [ n.s.m.p.b.BinaryMemcachedNodeImpl] - Discarding partially completed op: Cmd: 1 Opaque: 1865530 Key: cckey_V11 Cas: 0 Exp: 0 Flags: 1 Data Length: 222 [ n.s.m.MemcachedConnection] - Closing, and reopening {QA sa=/serverip_3:serverport, #Rops=2, #Wops=180, #iq=0, topRop=Cmd: -1 Opaque: -1 Keys: , topWop=Cmd: 0 Opaque: 1865589 Key: cckey2_V11, toWrite=0, interested=5}, attempt 0. [ n.s.m.p.b.BinaryMemcachedNodeImpl] - Discarding partially completed op: Cmd: -1 Opaque: -1 Keys:

    Also I noticed the value DEFAULT_OP_QUEUE_MAX_BLOCK_TIME in the spymemcache code which coincidentally = 10 seconds. Which is quite strange, anyone know what this value is for?

    DefaultConnectionFactory public static final long DEFAULT_OP_QUEUE_MAX_BLOCK_TIME = TimeUnit.SECONDS.toMillis(10);

    Any ideas or help would be most welcome.

    opened by pat-humphreys 4
  • MultiZone PoolSize

    MultiZone PoolSize

    Can anyone recommend a poolsize to use for an application that would have around 1000 interactions per second with memcache? I had this set to 5, but I was seeing some very unusual results, almost looking like a threading issue. I am still trying to collect more data around that, but a poolsize recommendation would be great.

    opened by pat-humphreys 4
  • Large amount of PoolRefresher threads

    Large amount of PoolRefresher threads

    Hello, We are using v4.76.0 with stand alone (non-eureka mode) and have two nodes memcached cluster. We faced situation when several thousands of PoolRefresher threads remain in BLOCKED state

    at com.netflix.evcache.pool.EVCacheClientPool.refresh(EVCacheClientPool.java:680) at com.netflix.evcache.pool.EVCacheClientPool.access$200(EVCacheClientPool.java:47) at com.netflix.evcache.pool.EVCacheClientPool$7.run(EVCacheClientPool.java:859)

    waiting for lock for com.netflix.evcache.pool.EVCacheClientPool owned by single PoolRefresher. Comparing samples we can see that it's not deadlock situation and PoolRefresher which owns the lock is changing constantly. All PoolRefresher related to single memcached node. Netstat shows that connections to this node in SYN_SENT status. Are there workaround for this situation ? From code I can see that it is result of EVCacheClientPool#refreshAsync. Would it make sense to limit number of concurrent refreshAsync operations?

    opened by akhitev 4
  • Add a metric to track

    Add a metric to track "Wrong key returned" errors

    This change adds "wrongKeyReturned" as a metric and increments it everytime we see it in action.

    This change was added as we noticed some of these error messages in the logs, and we'd like to track the frequence of these errors.

    Note that even prior to this change, we did not return the wrong key back to the calling application but instead count it as a miss.

    Print stack trace when we notice a Wrong Key error

    Consolidate duplicate code under checkWrongKeyReturned()

    Address review comments for wrong key returned metric

    • Better logging to catch if returned key is from a different host
    • Make sure stack trace goes to logs instead of stdout
    • Naming nits addressed
    opened by smukil 0
  • Add a metric to track

    Add a metric to track "Wrong key returned" errors

    This change adds "wrongKeyReturned" as a metric and increments it everytime we see it in action.

    This change was added as we noticed some of these error messages in the logs, and we'd like to track the frequence of these errors.

    Note that even prior to this change, we did not return the wrong key back to the calling application but instead count it as a miss.

    opened by smukil 1
  • Adding a basic Meta arithmetic implementation

    Adding a basic Meta arithmetic implementation

    This will get binary incr/decr compatibility for the ascii protocol

    opened by tharanga 0
  • Fix heading typo `Failues` to `Failures`

    Fix heading typo `Failues` to `Failures`

    Simple typo fix

    opened by dainmiller 0
  • Netflix

    Netflix

    opened by odedsemach 0
  • docs: fix typo

    docs: fix typo

    Failues -> Failures

    opened by HollowMan6 0
  • .travis.yml: The 'sudo' tag is now deprecated in Travis CI

    .travis.yml: The 'sudo' tag is now deprecated in Travis CI

    opened by cclauss 0
  • Two issues on the evcache

    Two issues on the evcache

    1. The evcache.EVCacheClientPool.poolSize=2 setting will cause the memcached unstable. If we set the poolSize as 2, the cache client becomes unstable. After several days/hours running and there are about 1M data in cache server, the get/set/delete operation become more and more slow. They needs more than tens seconds.
    2. In default, the evcache writes so many logs with info level. After we upgrade to the 4.131. 0, the evcache writes so many data, our log database is easy to be full. And it will slow our servers.
    opened by shengang1978 1
  • Any particular reason compare-and-set is not supported?

    Any particular reason compare-and-set is not supported?

    while "add" is supported

    opened by lcf 0
  • How to improve the set/delete performance?

    How to improve the set/delete performance?

    We use evcache with one memcache server node. We find there are much EVCache.set/delete operations which need more than 500ms. I'm not sure if the performance is up to the limitation. Here are some monitor log. Key total number in memcache: 700k QPS: 400 to 600

    opened by shengang1978 5
Releases(v5.18.21-rc.1)
Owner
Netflix, Inc.
Netflix Open Source Platform
Netflix, Inc.
CrateDB is a distributed SQL database that makes it simple to store and analyze massive amounts of machine data in real-time.

About CrateDB is a distributed SQL database that makes it simple to store and analyze massive amounts of machine data in real-time. CrateDB offers the

Crate.io 3k Mar 11, 2021
A distributed in-memory data store for the cloud

EVCache EVCache is a memcached & spymemcached based caching solution that is mainly used for AWS EC2 infrastructure for caching frequently used data.

Netflix, Inc. 1.6k Mar 10, 2021
Transactional schema-less embedded database used by JetBrains YouTrack and JetBrains Hub.

JetBrains Xodus is a transactional schema-less embedded database that is written in Java and Kotlin. It was initially developed for JetBrains YouTrack

JetBrains 858 Mar 12, 2021
Transactional schema-less embedded database used by JetBrains YouTrack and JetBrains Hub.

JetBrains Xodus is a transactional schema-less embedded database that is written in Java and Kotlin. It was initially developed for JetBrains YouTrack

JetBrains 858 Mar 12, 2021
Replicate your Key Value Store across your network, with consistency, persistance and performance.

Chronicle Map Version Overview Chronicle Map is a super-fast, in-memory, non-blocking, key-value store, designed for low-latency, and/or multi-process

Chronicle Software : Open Source 2.1k Mar 10, 2021
MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap-memory. It is a fast and easy to use embedded Java database engine.

MapDB: database engine MapDB combines embedded database engine and Java collections. It is free under Apache 2 license. MapDB is flexible and can be u

Jan Kotek 4.2k Mar 12, 2021
MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap-memory. It is a fast and easy to use embedded Java database engine.

MapDB: database engine MapDB combines embedded database engine and Java collections. It is free under Apache 2 license. MapDB is flexible and can be u

Jan Kotek 4.2k Mar 12, 2021
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)

Trino is a fast distributed SQL query engine for big data analytics. See the User Manual for deployment instructions and end user documentation. Devel

Trino 2.8k Mar 13, 2021
R2DBC Driver for Oracle Database

About Oracle R2DBC The Oracle R2DBC Driver is a Java library that supports reactive programming with Oracle Database. Oracle R2DBC implements the R2DB

Oracle 59 Mar 24, 2021
A blazingly small and sane redis java client

Jedis Jedis is a blazingly small and sane Redis java client. Jedis was conceived to be EASY to use. Jedis is fully compatible with redis 2.8.x, 3.x.x

Redis 9.7k Mar 13, 2021
A blazingly small and sane redis java client

Jedis Jedis is a blazingly small and sane Redis java client. Jedis was conceived to be EASY to use. Jedis is fully compatible with redis 2.8.x, 3.x.x

Redis 9.7k Mar 13, 2021
Apache Hive

Apache Hive (TM) The Apache Hive (TM) data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storag

The Apache Software Foundation 3.6k Mar 13, 2021
Distributed ID Generate Service

Leaf There are no two identical leaves in the world. — Leibnitz 中文文档 | English Document Introduction Leaf refers to some common ID generation schemes

美团 4.3k Mar 12, 2021