A better compressed bitset in Java

Overview

RoaringBitmap

docs-badge Java 8 CI Java 11 CI Java 12 CI Java 13 CI Java 14 CI

Bitsets, also called bitmaps, are commonly used as fast data structures. Unfortunately, they can use too much memory. To compensate, we often use compressed bitmaps.

Roaring bitmaps are compressed bitmaps which tend to outperform conventional compressed bitmaps such as WAH, EWAH or Concise. In some instances, roaring bitmaps can be hundreds of times faster and they often offer significantly better compression. They can even be faster than uncompressed bitmaps.

Roaring bitmaps are found to work well in many important applications:

Use Roaring for bitmap compression whenever possible. Do not use other bitmap compression methods (Wang et al., SIGMOD 2017)

kudos for making something that makes my software run 5x faster (Charles Parker from BigML)

This library is used by

The YouTube SQL Engine, Google Procella, uses Roaring bitmaps for indexing. Apache Lucene uses Roaring bitmaps, though they have their own independent implementation. Derivatives of Lucene such as Solr and Elastic also use Roaring bitmaps. Other platforms such as Whoosh, Microsoft Visual Studio Team Services (VSTS) and Pilosa also use Roaring bitmaps with their own implementations. You find Roaring bitmaps in InfluxDB, Bleve, Cloud Torrent, and so forth.

There is a serialized format specification for interoperability between implementations: https://github.com/RoaringBitmap/RoaringFormatSpec/ We have interoperable C/C++, Java and Go implementations.

(c) 2013-... the RoaringBitmap authors

This code is licensed under Apache License, Version 2.0 (AL2.0).

When should you use a bitmap?

Sets are a fundamental abstraction in software. They can be implemented in various ways, as hash sets, as trees, and so forth. In databases and search engines, sets are often an integral part of indexes. For example, we may need to maintain a set of all documents or rows (represented by numerical identifier) that satisfy some property. Besides adding or removing elements from the set, we need fast functions to compute the intersection, the union, the difference between sets, and so on.

To implement a set of integers, a particularly appealing strategy is the bitmap (also called bitset or bit vector). Using n bits, we can represent any set made of the integers from the range [0,n): the ith bit is set to one if integer i is present in the set. Commodity processors use words of W=32 or W=64 bits. By combining many such words, we can support large values of n. Intersections, unions and differences can then be implemented as bitwise AND, OR and ANDNOT operations. More complicated set functions can also be implemented as bitwise operations.

When the bitset approach is applicable, it can be orders of magnitude faster than other possible implementation of a set (e.g., as a hash set) while using several times less memory.

However, a bitset, even a compressed one is not always applicable. For example, if the you have 1000 random-looking integers, then a simple array might be the best representation. We refer to this case as the "sparse" scenario.

When should you use compressed bitmaps?

An uncompressed BitSet can use a lot of memory. For example, if you take a BitSet and set the bit at position 1,000,000 to true and you have just over 100kB. That is over 100kB to store the position of one bit. This is wasteful even if you do not care about memory: suppose that you need to compute the intersection between this BitSet and another one that has a bit at position 1,000,001 to true, then you need to go through all these zeroes, whether you like it or not. That can become very wasteful.

This being said, there are definitively cases where attempting to use compressed bitmaps is wasteful. For example, if you have a small universe size. E.g., your bitmaps represent sets of integers from [0,n) where n is small (e.g., n=64 or n=128). If you are able to uncompressed BitSet and it does not blow up your memory usage, then compressed bitmaps are probably not useful to you. In fact, if you do not need compression, then a BitSet offers remarkable speed.

The sparse scenario is another use case where compressed bitmaps should not be used. Keep in mind that random-looking data is usually not compressible. E.g., if you have a small set of 32-bit random integers, it is not mathematically possible to use far less than 32 bits per integer, and attempts at compression can be counterproductive.

How does Roaring compares with the alternatives?

Most alternatives to Roaring are part of a larger family of compressed bitmaps that are run-length-encoded bitmaps. They identify long runs of 1s or 0s and they represent them with a marker word. If you have a local mix of 1s and 0, you use an uncompressed word.

There are many formats in this family:

  • Oracle's BBC is an obsolete format at this point: though it may provide good compression, it is likely much slower than more recent alternatives due to excessive branching.
  • WAH is a patented variation on BBC that provides better performance.
  • Concise is a variation on the patented WAH. It some specific instances, it can compress much better than WAH (up to 2x better), but it is generally slower.
  • EWAH is both free of patent, and it is faster than all the above. On the downside, it does not compress quite as well. It is faster because it allows some form of "skipping" over uncompressed words. So though none of these formats are great at random access, EWAH is better than the alternatives.

There is a big problem with these formats however that can hurt you badly in some cases: there is no random access. If you want to check whether a given value is present in the set, you have to start from the beginning and "uncompress" the whole thing. This means that if you want to intersect a big set with a large set, you still have to uncompress the whole big set in the worst case...

Roaring solves this problem. It works in the following manner. It divides the data into chunks of 216 integers (e.g., [0, 216), [216, 2 x 216), ...). Within a chunk, it can use an uncompressed bitmap, a simple list of integers, or a list of runs. Whatever format it uses, they all allow you to check for the present of any one value quickly (e.g., with a binary search). The net result is that Roaring can compute many operations much faster than run-length-encoded formats like WAH, EWAH, Concise... Maybe surprisingly, Roaring also generally offers better compression ratios.

API docs

http://www.javadoc.io/doc/org.roaringbitmap/RoaringBitmap/

Scientific Documentation

  • Daniel Lemire, Owen Kaser, Nathan Kurz, Luca Deri, Chris O'Hara, François Saint-Jacques, Gregory Ssi-Yan-Kai, Roaring Bitmaps: Implementation of an Optimized Software Library, Software: Practice and Experience 48 (4), 2018 arXiv:1709.07821
  • Samy Chambi, Daniel Lemire, Owen Kaser, Robert Godin, Better bitmap performance with Roaring bitmaps, Software: Practice and Experience Volume 46, Issue 5, pages 709–719, May 2016 http://arxiv.org/abs/1402.6407 This paper used data from http://lemire.me/data/realroaring2014.html
  • Daniel Lemire, Gregory Ssi-Yan-Kai, Owen Kaser, Consistently faster and smaller compressed bitmaps with Roaring, Software: Practice and Experience 46 (11), 2016. http://arxiv.org/abs/1603.06549
  • Samy Chambi, Daniel Lemire, Robert Godin, Kamel Boukhalfa, Charles Allen, Fangjin Yang, Optimizing Druid with Roaring bitmaps, IDEAS 2016, 2016. http://r-libre.teluq.ca/950/

Code sample

import org.roaringbitmap.RoaringBitmap;

public class Basic {

  public static void main(String[] args) {
        RoaringBitmap rr = RoaringBitmap.bitmapOf(1,2,3,1000);
        RoaringBitmap rr2 = new RoaringBitmap();
        rr2.add(4000L,4255L);
        rr.select(3); // would return the third value or 1000
        rr.rank(2); // would return the rank of 2, which is index 1
        rr.contains(1000); // will return true
        rr.contains(7); // will return false

        RoaringBitmap rror = RoaringBitmap.or(rr, rr2);// new bitmap
        rr.or(rr2); //in-place computation
        boolean equals = rror.equals(rr);// true
        if(!equals) throw new RuntimeException("bug");
        // number of values stored?
        long cardinality = rr.getLongCardinality();
        System.out.println(cardinality);
        // a "forEach" is faster than this loop, but a loop is possible:
        for(int i : rr) {
          System.out.println(i);
        }
  }
}

Please see the examples folder for more examples, which you can run with ./gradlew :examples:runAll, or run a specific one with ./gradlew :examples:runExampleBitmap64, etc.

Unsigned integers

Java lacks native unsigned integers but integers are still considered to be unsigned within Roaring and ordered according to Integer.compareUnsigned. This means that Java will order the numbers like so 0, 1, ..., 2147483647, -2147483648, -2147483647,..., -1. To interpret correctly, you can use Integer.toUnsignedLong and Integer.toUnsignedString.

Working with memory-mapped bitmaps

If you want to have your bitmaps lie in memory-mapped files, you can use the org.roaringbitmap.buffer package instead. It contains two important classes, ImmutableRoaringBitmap and MutableRoaringBitmap. MutableRoaringBitmaps are derived from ImmutableRoaringBitmap, so that you can convert (cast) a MutableRoaringBitmap to an ImmutableRoaringBitmap in constant time.

An ImmutableRoaringBitmap that is not an instance of a MutableRoaringBitmap is backed by a ByteBuffer which comes with some performance overhead, but with the added flexibility that the data can reside anywhere (including outside of the Java heap).

At times you may need to work with bitmaps that reside on disk (instances of ImmutableRoaringBitmap) and bitmaps that reside in Java memory. If you know that the bitmaps will reside in Java memory, it is best to use MutableRoaringBitmap instances, not only can they be modified, but they will also be faster. Moreover, because MutableRoaringBitmap instances are also ImmutableRoaringBitmap instances, you can write much of your code expecting ImmutableRoaringBitmap.

If you write your code expecting ImmutableRoaringBitmap instances, without attempting to cast the instances, then your objects will be truly immutable. The MutableRoaringBitmap has a convenience method (toImmutableRoaringBitmap) which is a simple cast back to an ImmutableRoaringBitmap instance. From a language design point of view, instances of the ImmutableRoaringBitmap class are immutable only when used as per the interface of the ImmutableRoaringBitmap class. Given that the class is not final, it is possible to modify instances, through other interfaces. Thus we do not take the term "immutable" in a purist manner, but rather in a practical one.

One of our motivations for this design where MutableRoaringBitmap instances can be casted down to ImmutableRoaringBitmap instances is that bitmaps are often large, or used in a context where memory allocations are to be avoided, so we avoid forcing copies. Copies could be expected if one needs to mix and match ImmutableRoaringBitmap and MutableRoaringBitmap instances.

The following code sample illustrates how to create an ImmutableRoaringBitmap from a ByteBuffer. In such instances, the constructor only loads the meta-data in RAM while the actual data is accessed from the ByteBuffer on demand.

        import org.roaringbitmap.buffer.*;

        //...

        MutableRoaringBitmap rr1 = MutableRoaringBitmap.bitmapOf(1, 2, 3, 1000);
        MutableRoaringBitmap rr2 = MutableRoaringBitmap.bitmapOf( 2, 3, 1010);
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        DataOutputStream dos = new DataOutputStream(bos);
        // If there were runs of consecutive values, you could
        // call rr1.runOptimize(); or rr2.runOptimize(); to improve compression
        rr1.serialize(dos);
        rr2.serialize(dos);
        dos.close();
        ByteBuffer bb = ByteBuffer.wrap(bos.toByteArray());
        ImmutableRoaringBitmap rrback1 = new ImmutableRoaringBitmap(bb);
        bb.position(bb.position() + rrback1.serializedSizeInBytes());
        ImmutableRoaringBitmap rrback2 = new ImmutableRoaringBitmap(bb);

Alternatively, we can serialize directly to a ByteBuffer with the serialize(ByteBuffer) method.

Operations on an ImmutableRoaringBitmap such as and, or, xor, flip, will generate a RoaringBitmap which lies in RAM. As the name suggest, the ImmutableRoaringBitmap itself cannot be modified.

This design was inspired by Druid.

One can find a complete working example in the test file TestMemoryMapping.java.

Note that you should not mix the classes from the org.roaringbitmap package with the classes from the org.roaringbitmap.buffer package. They are incompatible. They serialize to the same output however. The performance of the code in org.roaringbitmap package is generally superior because there is no overhead due to the use of ByteBuffer instances.

Kryo

Many applications use Kryo for serialization/deserialization. One can use Roaring bitmaps with Kryo efficiently thanks to a custom serializer (Kryo 5):

public class RoaringSerializer extends Serializer<RoaringBitmap> {
    @Override
    public void write(Kryo kryo, Output output, RoaringBitmap bitmap) {
        try {
            bitmap.serialize(new KryoDataOutput(output));
        } catch (IOException e) {
            e.printStackTrace();
            throw new RuntimeException();
        }
    }
    @Override
    public RoaringBitmap read(Kryo kryo, Input input, Class<? extends RoaringBitmap> type) {
        RoaringBitmap bitmap = new RoaringBitmap();
        try {
            bitmap.deserialize(new KryoDataInput(input));
        } catch (IOException e) {
            e.printStackTrace();
            throw new RuntimeException();
        }
        return bitmap;
    }

}

64-bit integers (long)

Though Roaring Bitmaps were designed with the 32-bit case in mind, we have extensions to 64-bit integers. We offer two classes for this purpose: Roaring64NavigableMap and Roaring64Bitmap.

The Roaring64NavigableMap relies on a conventional red-black-tree. The keys are 32-bit integers representing the most significant 32~bits of elements whereas the values of the tree are 32-bit Roaring bitmaps. The 32-bit Roaring bitmaps represent the least significant bits of a set of elements.

The newer Roaring64Bitmap approach relies on the ART data structure to hold the key/value pair. The key is made of the most significant 48~bits of elements whereas the values are 16-bit Roaring containers. It is inspired by The Adaptive Radix Tree: ARTful Indexing for Main-Memory Databases by Leis et al. (ICDE '13).

    import org.roaringbitmap.longlong.*;

    
    // first Roaring64NavigableMap
    LongBitmapDataProvider r = Roaring64NavigableMap.bitmapOf(1,2,100,1000);
    r.addLong(1234);
    System.out.println(r.contains(1)); // true
    System.out.println(r.contains(3)); // false
    LongIterator i = r.getLongIterator();
    while(i.hasNext()) System.out.println(i.next());


    // second Roaring64Bitmap
    bitmap1 = new Roaring64Bitmap();
    bitmap2 = new Roaring64Bitmap();
    int k = 1 << 16;
    long i = Long.MAX_VALUE / 2;
    long base = i;
    for (; i < base + 10000; ++i) {
       bitmap1.add(i * k);
       bitmap2.add(i * k);
    }
    b1.and(bitmap2);

Prerequisites

  • Version 0.7.x requires JDK 8 or better
  • Version 0.6.x requires JDK 7 or better
  • Version 0.5.x requires JDK 6 or better

To build the project you need maven (version 3).

Download

You can download releases from github: https://github.com/RoaringBitmap/RoaringBitmap/releases

Maven repository

If your project depends on roaring, you can specify the dependency in the Maven "pom.xml" file:

        <dependencies>
          <dependency>
            <groupId>org.roaringbitmap</groupId>
            <artifactId>RoaringBitmap</artifactId>
            <version>0.9.9</version>
          </dependency>
        </dependencies>

where you should replace the version number by the version you require.

For up-to-date releases, we recommend configuring maven and gradle to depend on the Jitpack repository.

Usage

  • Get java

  • ./gradlew assemble will compile

  • ./gradlew build will compile and run the unit tests

  • ./gradlew test will run the tests

  • ./gradlew :roaringbitmap:test --tests TestIterators.testIndexIterator4 run just the test TestIterators.testIndexIterator4

  • ./gradlew checkstyleMain will check that you abide by the code style and that the code compiles. We enforce a strict style so that there is no debate as to the proper way to format the code.

IntelliJ and Eclipse

If you plan to contribute to RoaringBitmap, you can have load it up in your favorite IDE.

  • For IntelliJ, in the IDE create a new project, possibly from existing sources, choose import, gradle.
  • For Eclipse: File, Import, Existing Gradle Projects, Select RoaringBitmap on my disk

Contributing

Contributions are invited. We enforce the Google Java style. Please run ./gradlew checkstyleMain on your code before submitting a patch.

FAQ

  • I am getting an error about a bad cookie. What is this about?

In the serialized files, part of the first 4 bytes are dedicated to a "cookie" which serves to indicate the file format.

If you try to deserialize or map a bitmap from data that has an unrecognized "cookie", the code will abort the process and report an error.

This problem will occur to all users who serialized Roaring bitmaps using versions prior to 0.4.x as they upgrade to version 0.4.x or better. These users need to refresh their serialized bitmaps.

  • How big can a Roaring bitmap get?

Given N integers in [0,x), then the serialized size in bytes of a Roaring bitmap should never exceed this bound:

8 + 9 * ((long)x+65535)/65536 + 2 * N

That is, given a fixed overhead for the universe size (x), Roaring bitmaps never use more than 2 bytes per integer. You can call RoaringBitmap.maximumSerializedSize for a more precise estimate.

  • What is the worst case scenario for Roaring bitmaps?

There is no such thing as a data structure that is always ideal. You should make sure that Roaring bitmaps fit your application profile. There are at least two cases where Roaring bitmaps can be easily replaced by superior alternatives compression-wise:

  1. You have few random values spanning in a large interval (i.e., you have a very sparse set). For example, take the set 0, 65536, 131072, 196608, 262144 ... If this is typical of your application, you might consider using a HashSet or a simple sorted array.

  2. You have dense set of random values that never form runs of continuous values. For example, consider the set 0,2,4,...,10000. If this is typical of your application, you might be better served with a conventional bitset (e.g., Java's BitSet class).

  • How do I select an element at random?

       Random random = new Random();
       bitmap.select(random.nextInt(bitmap.getCardinality()));
    

Benchmark

To run JMH benchmarks, use the following command:

     $ ./gradlew jmhJar

You can also run specific benchmarks...

     $ ./jmh/run.sh 'org.roaringbitmap.aggregation.and.identical.*'

Mailing list/discussion group

https://groups.google.com/forum/#!forum/roaring-bitmaps

Funding

This work was supported by NSERC grant number 26143.

Issues
  • Port most of the build to gradle.

    Port most of the build to gradle.

    This is still missing some stuff but I wanted to get feedback on if this was useful / interesting before sinking more time into it.

    Typical gradle subproject structure plus:

    • Multi-release jar for shims subproject
    • Move examples into typical src/main/java and set up tasks for each one, removing the need for a separate shell script
    • Move JMH benchmarks from src/main/java to src/jmh/java since that's how the JMH plugin prefers it. (This allows benchmarks to live in the same module as the normal code, if desired.)

    Missing things:

    • Deploying releases (use Bintray to avoid Sonatype Nexus pain?)
    • jxr
    • perhaps other subtleties I missed/forgot
    opened by marshallpierce 106
  • Introduce RoaringTreeMap to handle a bitmap holding any long value

    Introduce RoaringTreeMap to handle a bitmap holding any long value

    Hello,

    This refers to https://github.com/RoaringBitmap/RoaringBitmap/issues/109

    I needed a bitmap holding any long values to improve management of object references in a fork of MAT. Here is a my current state-of-work: it handles .rank, .select and .getCardinality . I added a few unit-tests.

    If this would actually be integrated in RoaringBitmap, please provide feedbacks about what would be the relevant next-steps, especially regarding naming conventions, or API (e.g. should .getCardinality returns a long).

    new feature 
    opened by blacelle 86
  • Faster deserialization by loading BitmapContainer in a single .readFully

    Faster deserialization by loading BitmapContainer in a single .readFully

    The IO effect of reading bytes one by one seems quite big. With a single .readFully:

    recommended: 247 ms 107 ms 83 ms 177 ms 81 ms 81 ms 107 ms 83 ms 83 ms 116 ms 154 ms 97 ms 82 ms 85 ms 83 ms 84 ms 97 ms 84 ms 84 ms 83 ms 83 ms 117 ms 161 ms 132 ms 82 ms 84 ms 84 ms 84 ms 84 ms 82 ms average: 103 via ByteArrayInputStream: 257 ms 86 ms 86 ms 86 ms 86 ms 84 ms 82 ms 82 ms 180 ms 88 ms 83 ms 85 ms 84 ms 84 ms 85 ms 86 ms 102 ms 90 ms 84 ms 82 ms 83 ms 84 ms 83 ms 85 ms 94 ms 82 ms 81 ms 82 ms 84 ms 82 ms average: 94 via Immutable: 130 ms 69 ms 77 ms 67 ms 67 ms 70 ms 67 ms 64 ms 62 ms 64 ms 77 ms 65 ms 64 ms 63 ms 64 ms 64 ms 64 ms 62 ms 66 ms 64 ms 66 ms 67 ms 64 ms 65 ms 64 ms 62 ms 78 ms 66 ms 65 ms 64 ms average: 68

    Reference: https://github.com/RoaringBitmap/RoaringBitmap/issues/319

    opened by blacelle 46
  • Range operations performance

    Range operations performance

    Fixes #103

    opened by ppiotrow 40
  • Convert

    Convert "full" bitsets to runs #124

    When ORing two bitsets, the result can be "full" (cardinality = 1<<16). Checking for this occurrence is cheap. When it happens, it should be converted to a "full" run container.

    RunContainer.full helper method to cheaply create "full" run container

    opened by ppiotrow 40
  • Implement parallel OR and XOR

    Implement parallel OR and XOR

    This PR provides a way to execute arbitrary boolean reductions on bitmaps in parallel.

    Parallel AND and ANDNOT were implemented but found to be unprofitable. They can be implemented as circuits if necessary.

    opened by richardstartin 35
  • Broken releases due(?) to broken maven setup (RoaringBitmapParent artifact)

    Broken releases due(?) to broken maven setup (RoaringBitmapParent artifact)

    I used to issue releases by going to the main RoaringBitmap repository and typing mvn release:clean && mvn release:prepare && mvn release:perform. Since PR https://github.com/RoaringBitmap/RoaringBitmap/pull/181 this no longer works. Sadly, CI does not check that releases are possible, so I did not catch the issue.

    As I recall the point of the release was to move to a "classic maven project" described as follow:

    Cannot we do a classic maven project by moving RoaringBitmap actual source to a specific submodule, and leaving all other submodules where they are?

    Technically, I can issue the commands, but it does not generate an actual release and the result is a broken repository. I left the broken commits on purpose, so you can see the result...

    https://github.com/RoaringBitmap/RoaringBitmap/commits/master

    Note that to be able to do this in a sane way, I had to make a small change to the pom file so that jmh is excluded as a submodule by default, because otherwise, we run the full jmh unit test which takes 3 hours on my machine, and fails for an unclear reason (it looks like -DBITMAP_TYPES=ROARING_ONLY has no effect when doing mvn perform), with The forked VM terminated without saying properly goodbye.

    Anyhow, the broken aspect of the release with mvn release:clean && mvn release:prepare && mvn release:perform has to do with the fact that this attempts to release an artifact RoaringBitmapParent. I am not sure what that is, and I am not sure that it is correct. Yet this seems to have been deliberate, see the pom file: https://github.com/RoaringBitmap/RoaringBitmap/blob/master/pom.xml#L5

    To still be able to issue releases, I did a deploy instead from within the roaringbitmap subdirectory, hoping it would save us from RoaringBitmapParent artifact nonesense...

      cd roaringbitmap
      mvn versions:set -DnewVersion=0.6.53
      mvn clean deploy
      git commit -a
      git tag -a v0.6.53 -m "version 0.6.53"
      git push --tags
      # go to https://oss.sonatype.org/#nexus-search;quick~roaringbitmap and initial the release manually
    

    Sadly, this result in something that is broken as well as you can see from a report on the mailing list:

    I'm having some problems pulling the latest release from maven central via Leiningen:

    [~/repos/streaming-lda] $ lein deps

    Retrieving org/roaringbitmap/RoaringBitmap/0.6.53/RoaringBitmap-0.6.53.pom from central

    Could not find artifact org.roaringbitmap:RoaringBitmapParent:pom:0.6.53 in central > (https://repo1.maven.org/maven2/)

    This could be due to a typo in :dependencies or network issues.

    You can see it's looking for a artifact with "RoaringBitmapParent" in the name. Leiningen doesn't do this when I try to pull down any other version, it just grabs the pom and jar having the usual names, like this: [~/repos/streaming-lda] $ lein deps Retrieving org/roaringbitmap/RoaringBitmap/0.6.49/RoaringBitmap-0.6.49.pom from central Retrieving org/roaringbitmap/RoaringBitmap/0.6.49/RoaringBitmap-0.6.49.jar from central
    I can see from the history your pom file that this is some sort of intentional change. My maven knowledge is fairly low, but I'm suspicious that this is something Leiningen either doesn't support requires some workaround for. Do you know anything more about this than I do?

    Anyhow, it seems clear that either the maven project is broken or else, I am lacking the right release instructions.

    I think that this is an urgent problem.

    URGENT bug help wanted 
    opened by lemire 33
  • Try to go back to publishing directly to Maven Central

    Try to go back to publishing directly to Maven Central

    It appears that the sonatype folks are willing to help us. So if we reconfigure our build for sonatype, they may provide us with a faster access...

    So we need to go through the song and dance and retool our build setup accordingly.

    https://central.sonatype.org/pages/gradle.html

    https://central.sonatype.org/pages/ossrh-guide.html

    cc @richardstartin @marshallpierce

    opened by lemire 31
  • AC.andNot(RC)

    AC.andNot(RC)

    I've decided to try using Util.advanceUntil in andNot between ArrayContainer and RunContainer. The implementation present in the commit was verified with present tests and a new one, but for sure I'd test it more if going to be merged.

    I run two benchmarks part{3,4}_andNotArrayContainerVSRunContainerContainer on master and the new branch.

    BasicAndNotContainerBenchmark.part3_andNotArrayContainerVSRunContainerContainer  avgt   15  3327,444 ± 49,353  ns/op
    BasicAndNotContainerBenchmark.part4_andNotArrayContainerVSRunContainerContainer  avgt   15  1877,370 ± 25,888  ns/op
    
    BasicAndNotContainerBenchmark.part3_andNotArrayContainerVSRunContainerContainer  avgt   15  7790,214 ± 81,502  ns/op
    BasicAndNotContainerBenchmark.part4_andNotArrayContainerVSRunContainerContainer  avgt   15  638,899 ± 4,385  ns/op
    

    The code is slower for the present benchmark which andNot Array Container of 1024 elements with very dense Run Container of 1009 runs with 64512 cardinality. The new benchmark uses Run Container with 5 very long runs and the new code is 3x faster.

    I'm opening this Pull Request to discuss

    • is it worth to use new approach?
    • should there be a switch when to use old and new approach?

    I think the new approach will be faster when there are long sequences of array elements covered by runs or between two runs. If there are very few array elements per run or between runs the old approach should be faster.

    opened by ppiotrow 30
  • Migrate to s01.oss.sonatype.org

    Migrate to s01.oss.sonatype.org

    It seems that we need to migrate to s01.oss.sonatype.org.

    Sonatype includes the relevant information in their guide...

    https://central.sonatype.org/publish/publish-gradle/

    But our current build does not reference a URI for the repository (it is somehow implicit?).

    @marshallpierce can you help?

    opened by lemire 3
  • partialRadixSort optimisation

    partialRadixSort optimisation

    I took a look at change you did in #471 and removing the memory move wasn't enough, don't ask me why but you need the loop unroll which gives the performance gain.

    Benchmark                              (listSize)  Mode  Cnt   Score   Error  Units
    PartialRadixSortBenchmark.BM_fullSort     1000000  avgt    2  65.492          ms/op
    PartialRadixSortBenchmark.BM_original     1000000  avgt    2   5.586          ms/op
    PartialRadixSortBenchmark.BM_new          1000000  avgt    2   5.648          ms/op
    PartialRadixSortBenchmark.BM_likeNew      1000000  avgt    2   3.310          ms/op
    PartialRadixSortBenchmark.BM_unroll       1000000  avgt    2   3.164          ms/op
    
    package benchmark;
    
    import org.openjdk.jmh.annotations.Benchmark;
    import org.openjdk.jmh.annotations.BenchmarkMode;
    import org.openjdk.jmh.annotations.Fork;
    import org.openjdk.jmh.annotations.Level;
    import org.openjdk.jmh.annotations.Measurement;
    import org.openjdk.jmh.annotations.Mode;
    import org.openjdk.jmh.annotations.OutputTimeUnit;
    import org.openjdk.jmh.annotations.Param;
    import org.openjdk.jmh.annotations.Scope;
    import org.openjdk.jmh.annotations.Setup;
    import org.openjdk.jmh.annotations.State;
    import org.openjdk.jmh.annotations.Warmup;
    import org.openjdk.jmh.infra.Blackhole;
    
    import java.util.Arrays;
    import java.util.concurrent.ThreadLocalRandom;
    import java.util.concurrent.TimeUnit;
    
    @BenchmarkMode(Mode.AverageTime)
    @OutputTimeUnit(TimeUnit.MILLISECONDS)
    @Warmup(iterations = 2)
    @Measurement(batchSize = 1, iterations = 2)
    @Fork(1)
    public class PartialRadixSortBenchmark {
    
    	public static void originalPartialRadixSort(int[] data) {
    		final int radix = 8;
    
    		int shift = 16;
    		int mask = 0xFF0000;
    		int[] copy = new int[data.length];
    		int[] histogram = new int[(1 << radix) + 1];
    		while (shift < 32) {
    			for (int i = 0; i < data.length; ++i) {
    				++histogram[((data[i] & mask) >>> shift) + 1];
    			}
    			for (int i = 0; i < 1 << radix; ++i) {
    				histogram[i + 1] += histogram[i];
    			}
    			for (int i = 0; i < data.length; ++i) {
    				copy[histogram[(data[i] & mask) >>> shift]++] = data[i];
    			}
    			System.arraycopy(copy, 0, data, 0, data.length);
    			shift += radix;
    			mask <<= radix;
    			Arrays.fill(histogram, 0);
    		}
    	}
    
    	public static void newPartialRadixSort(int[] data) {
    		final int radix = 8;
    		int shift = 16;
    		int mask = 0xFF0000;
    		int[] copy = new int[data.length];
    		int[] histogram = new int[(1 << radix) + 1];
    		// We want to avoid copying the data, see
    		// https://github.com/RoaringBitmap/RoaringBitmap/issues/470
    		int[] primary = data;
    		int[] secondary = copy;
    		while (shift < 32) {
    			for (int i = 0; i < data.length; ++i) {
    				++histogram[((primary[i] & mask) >>> shift) + 1];
    			}
    			for (int i = 0; i < 1 << radix; ++i) {
    				histogram[i + 1] += histogram[i];
    			}
    			for (int i = 0; i < primary.length; ++i) {
    				secondary[histogram[(primary[i] & mask) >>> shift]++] = primary[i];
    			}
    			// swap
    			int[] tmp = primary;
    			primary = secondary;
    			secondary = tmp;
    			//
    			shift += radix;
    			mask <<= radix;
    			Arrays.fill(histogram, 0);
    		}
    	}
    
    	public static void unrollPartialRadixSort(int[] primary) {
    		final int radix = 8;
    
    		final int[] secondary = new int[primary.length];
    		final int[] histogram = new int[(1 << radix) + 1];
    		{
    			final int shift = 16;
    			final int mask = 0x00FF0000;
    			for (int i = 0; i < primary.length; ++i) {
    				++histogram[((primary[i] & mask) >>> shift) + 1];
    			}
    			for (int i = 0; i < 1 << radix; ++i) {
    				histogram[i + 1] += histogram[i];
    			}
    			for (int i = 0; i < primary.length; ++i) {
    				secondary[histogram[(primary[i] & mask) >>> shift]++] = primary[i];
    			}
    		}
    		Arrays.fill(histogram, 0);
    		{
    			final int shift = 24;
    			final int mask = 0xFF000000;
    			for (int i = 0; i < secondary.length; ++i) {
    				++histogram[((secondary[i] & mask) >>> shift) + 1];
    			}
    			for (int i = 0; i < 1 << radix; ++i) {
    				histogram[i + 1] += histogram[i];
    			}
    			for (int i = 0; i < secondary.length; ++i) {
    				primary[histogram[(secondary[i] & mask) >>> shift]++] = secondary[i];
    			}
    		}
    	}
    
    	public static void likeNewPartialRadixSort(int[] data) {
    		final int radix = 8;
    		int shift = 16;
    		int mask = 0xFF0000;
    		int[] copy = new int[data.length];
    		int[] histogram = new int[(1 << radix) + 1];
    		// We want to avoid copying the data, see
    		// https://github.com/RoaringBitmap/RoaringBitmap/issues/470
    		int[] primary = data;
    		int[] secondary = copy;
    		for (int i = 0; i < data.length; ++i) {
    			++histogram[((primary[i] & mask) >>> shift) + 1];
    		}
    		for (int i = 0; i < 1 << radix; ++i) {
    			histogram[i + 1] += histogram[i];
    		}
    		for (int i = 0; i < primary.length; ++i) {
    			secondary[histogram[(primary[i] & mask) >>> shift]++] = primary[i];
    		}
    		// swap
    		int[] tmp = primary;
    		primary = secondary;
    		secondary = tmp;
    		//
    		shift += radix;
    		mask <<= radix;
    		Arrays.fill(histogram, 0);
    		for (int i = 0; i < data.length; ++i) {
    			++histogram[((primary[i] & mask) >>> shift) + 1];
    		}
    		for (int i = 0; i < 1 << radix; ++i) {
    			histogram[i + 1] += histogram[i];
    		}
    		for (int i = 0; i < primary.length; ++i) {
    			secondary[histogram[(primary[i] & mask) >>> shift]++] = primary[i];
    		}
    		// swap
    		tmp = primary;
    		primary = secondary;
    		secondary = tmp;
    		//
    		shift += radix;
    		mask <<= radix;
    		Arrays.fill(histogram, 0);
    	}
    
    	@State(Scope.Benchmark)
    	public static class BenchmarkState {
    		@Param({/*"100", "10000",*/ "1000000"})
    		public int listSize;
    
    		private int[] origArr;
    		public int[] testArr;
    
    		@Setup(Level.Trial)
    		public void setUp() {
    			origArr = ThreadLocalRandom.current()
    					.ints()
    					.limit(listSize)
    					.toArray();
    			testArr = new int[origArr.length];
    		}
    
    		@Setup(Level.Invocation)
    		public void perInvocation(){
    			System.arraycopy(origArr, 0, testArr, 0, origArr.length);
    		}
    	}
    
    	@Benchmark
    	public void BM_original(Blackhole blackhole, BenchmarkState state) {
    		originalPartialRadixSort(state.testArr);
    		blackhole.consume(state.testArr);
    	}
    	@Benchmark
    	public void BM_new(Blackhole blackhole, BenchmarkState state) {
    		newPartialRadixSort(state.testArr);
    		blackhole.consume(state.testArr);
    	}
    
    	@Benchmark
    	public void BM_unroll(Blackhole blackhole, BenchmarkState state) {
    		unrollPartialRadixSort(state.testArr);
    		blackhole.consume(state.testArr);
    	}
    
    	@Benchmark
    	public void BM_likeNew(Blackhole blackhole, BenchmarkState state) {
    		likeNewPartialRadixSort(state.testArr);
    		blackhole.consume(state.testArr);
    	}
    
    	@Benchmark
    	public void BM_fullSort(Blackhole blackhole, BenchmarkState state) {
    		Arrays.sort(state.testArr);
    		blackhole.consume(state.testArr);
    	}
    
    }
    
    opened by Ignition 17
  • Rename package to org.roaringbitmap.v1 or org.roaringbitmap.v2

    Rename package to org.roaringbitmap.v1 or org.roaringbitmap.v2

    We are about to merge this API breaking changes to https://github.com/RoaringBitmap/RoaringBitmap/tree/1.0-SNAPSHOT

    It has been proposed in https://github.com/RoaringBitmap/RoaringBitmap/pull/461 that we rename the package.

    I do not mind jumping directly to v2, but consider that we never reached v1... and I am not exactly sure why would skip v1.

    cc @blacelle @richardstartin

    opened by lemire 2
  • Remove Iterable<Integer>

    Remove Iterable

    Implementing Iterable allows applications to use enhanced for loops, syntactic sugar, at the cost of performance (caused by auto-boxing). Since RoaringBitmap already implements forEach with IntConsumer, this patch removes the Iterable interface implementation from RoaringBitmap. This should encourage applications to use more efficient APIs and existing applications that use the enhanced for loop can easily convert the code to loop the iterator directly.

    opened by Maithem 23
  • Remove iterable

    Remove iterable

    As remarked by @richardstartin and @blacelle, users should not rely on iterable and should favour IntConsumer or something of the sort (for performance).

    opened by lemire 0
  • Comment away jcenter

    Comment away jcenter

    opened by blacelle 4
  • Question/advice: iterating over an a.and(b) bitmap whilst keeping ranks of elements from 'a'

    Question/advice: iterating over an a.and(b) bitmap whilst keeping ranks of elements from 'a'

    Hi,

    Sorry if it's not the place for questions or advice.

    I'm using RoaringBitmap as indexes in an internal tool, and the performance kick I get from them is amazing compared to sparse matrices or my homemade clumsy solutions...

    Still there's one operation I'm doing very often that I'm not sure how to implement efficiently.

    I have 2 bitmaps : blocks and index. blocks usually has very high cardinality, index usually lower, sometimes the same, sometimes higher.

    I want to iterate over blocks.and(index) BUT I need to keep the positions/ranks of elements from blocks. So for each element of blocks.and(index) I'd like to have that element's rank/position in blocks.

    What I do today is I iterate (forEach) over blocks.and(index) and for each element I get their rank from blocks through rank() or select(). The performance is... not great.

    I'm wondering whether there'd be a better, more idiomatic way of doing this? Should I be writing a specific andIterator that would give me this info? Anything I should be aware whilst going this path? Am I completely out of RoaringBitmaps' use case?

    Thanks in advance for your help. Lionel

    enhancement 
    opened by touisteur 11
  • `FastAggregation` for 64 bit Bitmap

    `FastAggregation` for 64 bit Bitmap

    In our case, Roaring64NavigableMap are used in customized Spark TypedImperativeAggregate, and aggregate looks like the following:

    Roaring64NavigableMap _64Map = new Roaring64NavigableMap();
    for (Bitmap _bitmap: bitmaps) {
      _64Map.or(_bitmap)
    }
    

    From the paper, Wide Unions will benefit from calculating cardinality only at the last iteration. so, I implemented a prototype :

    /**
     * In-place bitwise OR (union) operation without maintaining cardinality.
     * Don't forget to call repairAfterLazy() afterward. The current bitmap is modified.
     *
     * @param x2 other bitmap
     */
    public void naivelazyor(final Roaring64NavigableMap x2) {
      boolean firstBucket = true;
      for (Entry<Integer, BitmapDataProvider> e2 : x2.highToBitmap.entrySet()) {
        // Keep object to prevent auto-boxing
        Integer high = e2.getKey();
    
        BitmapDataProvider lowBitmap1 = this.highToBitmap.get(high);
    
        BitmapDataProvider lowBitmap2 = e2.getValue();
    
        if ((lowBitmap1 == null || lowBitmap1 instanceof RoaringBitmap)
                && lowBitmap2 instanceof RoaringBitmap) {
          if (lowBitmap1 == null) {
            // Clone to prevent future modification of this modifying the input Bitmap
            RoaringBitmap lowBitmap2Clone = ((RoaringBitmap) lowBitmap2).clone();
    
            pushBitmapForHigh(high, lowBitmap2Clone);
          } else {
            ((RoaringBitmap) lowBitmap1).naivelazyor((RoaringBitmap) lowBitmap2);
          }
        } else {
          throw new UnsupportedOperationException(
                  ".naivelazyor is not between " + this.getClass() + " and " + lowBitmap2.getClass());
        }
      }
    }
    

    Here are benchmark:

    wideOr_naive        mac        false  roaring64_with_run  avgt   25  382429.831 ± 34007.762  us/op
     pairwiseIOr        mac        false  roaring64_with_run  avgt   25  604916.850 ± 50736.895  us/op
    

    Any ideas?

    enhancement 
    opened by baibaichen 4
  • MappeableRunContainer.and(MappeableRunContainer) is slow

    MappeableRunContainer.and(MappeableRunContainer) is slow

    See druid. A benchmark intersecting 100 bitmaps regressed after a change meaning bitmaps were built in such a way that data would be run compressed during bitmap construction.

    The benchmark only regresses on quite dense bitmaps (10% and 50%). Here is where JMH perfasm identified that the time was spent in version 0.8.18, which corresponds to intersecting run containers:

    [email protected]:~/workspace/druid/benchmarks$ java -jar target/benchmarks.jar -wi 5 -w 1 -i 5 -r 1 -f 1 -p bitmapAlgo=roaring -p n=100 -p prob=0.1 -prof perfasm  BitmapIterationBenchmark.intersectionAndIter
    WARNING: An illegal reflective access operation has occurred
    WARNING: Illegal reflective access by org.openjdk.jmh.util.Utils (file:/home/richard/workspace/druid/benchmarks/target/benchmarks.jar) to field java.io.Console.cs
    WARNING: Please consider reporting this to the maintainers of org.openjdk.jmh.util.Utils
    WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
    WARNING: All illegal access operations will be denied in a future release
    # JMH version: 1.21
    # VM version: JDK 11.0.7, OpenJDK 64-Bit Server VM, 11.0.7+10-post-Ubuntu-2ubuntu218.04
    # VM invoker: /usr/lib/jvm/java-11-openjdk-amd64/bin/java
    # VM options: <none>
    # Warmup: 5 iterations, 1 s each
    # Measurement: 5 iterations, 1 s each
    # Timeout: 10 min per iteration
    # Threads: 1 thread, will synchronize iterations
    # Benchmark mode: Average time, time/op
    # Benchmark: org.apache.druid.benchmark.BitmapIterationBenchmark.intersectionAndIter
    # Parameters: (bitmapAlgo = roaring, n = 100, prob = 0.1, size = 1000000)
    
    # Run progress: 0.00% complete, ETA 00:00:10
    # Fork: 1 of 1
    # Preparing profilers: LinuxPerfAsmProfiler 
    # Profilers consume stdout and stderr from target VM, use -v EXTRA to copy to console
    # Warmup Iteration   1: 350404086.000 ns/op
    # Warmup Iteration   2: 210305542.000 ns/op
    # Warmup Iteration   3: 206197575.800 ns/op
    # Warmup Iteration   4: 206942536.000 ns/op
    # Warmup Iteration   5: 204609723.600 ns/op
    Iteration   1: 202870838.400 ns/op
    Iteration   2: 200174849.200 ns/op
    Iteration   3: 199120412.667 ns/op
    Iteration   4: 198925495.667 ns/op
    Iteration   5: 200660082.000 ns/op
    # Processing profiler results: LinuxPerfAsmProfiler 
    
    
    Result "org.apache.druid.benchmark.BitmapIterationBenchmark.intersectionAndIter":
      200350335.587 ±(99.9%) 6095264.032 ns/op [Average]
      (min, avg, max) = (198925495.667, 200350335.587, 202870838.400), stdev = 1582920.713
      CI (99.9%): [194255071.555, 206445599.619] (assumes normal distribution)
    
    Secondary result "org.apache.druid.benchmark.BitmapIterationBenchmark.intersectionAndIter:·asm":
    PrintAssembly processed: 203149 total address lines.
    Perf output processed (skipped 9.870 seconds):
     Column 1: cycles (5748 events)
    
    Hottest code regions (>10.00% "cycles" events):
    
    ....[Hottest Region 1]..............................................................................
    c2, level 4, org.roaringbitmap.buffer.MappeableRunContainer::and, version 683 (506 bytes) 
    
                                                               ; - org.roaringbitmap.buffer.MappeableContainer::&lt;init&gt;@1 (line 19)
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::&lt;init&gt;@1 (line 319)
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
                 0x00007f4ac84268ca: cmp    0x20(%rsp),%ebp
                 0x00007f4ac84268ce: jae    0x00007f4ac84272ca  ;*getfield limit {reexecute=0 rethrow=0 return_oop=0}
                                                               ; - java.nio.Buffer::[email protected] (line 681)
                                                               ; - java.nio.HeapCharBuffer::[email protected] (line 169)
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 974)
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 478)
                 0x00007f4ac84268d4: vmovq  %xmm4,%r10
      0.02%      0x00007f4ac84268d9: movzwl 0x16(%r10,%r14,2),%r9d
                 0x00007f4ac84268df: add    0x34(%rsp),%r9d
      0.33%      0x00007f4ac84268e4: inc    %r9d               ;*invokevirtual get {reexecute=0 rethrow=0 return_oop=0}
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 937)
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 479)
      0.02%      0x00007f4ac84268e7: mov    0x108(%r15),%r10   ; ImmutableOopMap{xmm0=NarrowOop xmm1=NarrowOop xmm2=NarrowOop xmm3=NarrowOop xmm4=Oop xmm5=NarrowOop xmm6=Oop xmm7=Oop [0]=Oop [8]=Oop [16]=Oop }
                                                               ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 488)
                 0x00007f4ac84268ee: test   %eax,(%r10)        ;*putfield valueslength {reexecute=0 rethrow=0 return_oop=0}
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::&lt;init&gt;@36 (line 324)
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
                                                               ;   {poll}
                 0x00007f4ac84268f1: vmovd  %xmm9,%r8d
                 0x00007f4ac84268f6: vmovaps %xmm8,%xmm10      ;*invokevirtual getLength {reexecute=0 rethrow=0 return_oop=0}
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 481)
      0.19%      0x00007f4ac84268fb: vmovd  %r8d,%xmm9
                 0x00007f4ac8426900: vmovd  %xmm10,%r10d       ;*invokespecial position {reexecute=0 rethrow=0 return_oop=0}
                                                               ; - java.nio.CharBuffer::[email protected] (line 1086)
                                                               ; - java.nio.CharBuffer::[email protected] (line 262)
                                                               ; - java.nio.Buffer::&lt;init&gt;@39 (line 223)
                                                               ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
                                                               ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
                                                               ; - java.nio.CharBuffer::[email protected] (line 348)
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
                 0x00007f4ac8426905: cmp    0x30(%rsp),%r10d
                 0x00007f4ac842690a: jge    0x00007f4ac8426c7e  ;*invokevirtual limit {reexecute=0 rethrow=0 return_oop=0}
                                                               ; - java.nio.Buffer::&lt;init&gt;@33 (line 222)
                                                               ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
                                                               ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
                                                               ; - java.nio.CharBuffer::[email protected] (line 348)
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
                 0x00007f4ac8426910: vmovd  %r10d,%xmm10
      0.23%  ╭   0x00007f4ac8426915: jmpq   0x00007f4ac84269b0
             │   0x00007f4ac842691a: nopw   0x0(%rax,%rax,1)
      0.09%  │↗  0x00007f4ac8426920: vmovq  %xmm4,%r11
      1.83%  ││  0x00007f4ac8426925: movzwl 0x16(%r11,%r14,2),%r11d
      0.05%  ││  0x00007f4ac842692b: add    %r8d,%r11d
      0.14%  ││  0x00007f4ac842692e: inc    %r11d              ;*invokevirtual limit {reexecute=0 rethrow=0 return_oop=0}
             ││                                                ; - java.nio.CharBuffer::[email protected] (line 262)
             ││                                                ; - java.nio.Buffer::&lt;init&gt;@33 (line 222)
             ││                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
             ││                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
             ││                                                ; - java.nio.CharBuffer::[email protected] (line 348)
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.21%  ││  0x00007f4ac8426931: mov    0x10(%rsp),%r8
      2.00%  ││  0x00007f4ac8426936: mov    0xc(%r8),%edi      ;*invokespecial &lt;init&gt; {reexecute=0 rethrow=0 return_oop=0}
             ││                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
             ││                                                ; - java.nio.CharBuffer::[email protected] (line 348)
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.05%  ││  0x00007f4ac842693a: mov    %edi,%ecx
      0.17%  ││  0x00007f4ac842693c: shl    %ecx               ;*invokevirtual limit {reexecute=0 rethrow=0 return_oop=0}
             ││                                                ; - java.nio.CharBuffer::[email protected] (line 262)
             ││                                                ; - java.nio.Buffer::&lt;init&gt;@33 (line 222)
             ││                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
             ││                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
             ││                                                ; - java.nio.CharBuffer::[email protected] (line 348)
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.21%  ││  0x00007f4ac842693e: cmp    0x18(%rsp),%ecx
             ││  0x00007f4ac8426942: jae    0x00007f4ac8426cc7  ;*invokevirtual get {reexecute=0 rethrow=0 return_oop=0}
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 937)
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 488)
      2.04%  ││  0x00007f4ac8426948: vmovq  %xmm7,%r8
      0.05%  ││  0x00007f4ac842694d: mov    %r10w,0x10(%r8,%rcx,2)
     18.28%  ││  0x00007f4ac8426953: vmovd  %xmm11,%r8d
      0.02%  ││  0x00007f4ac8426958: sub    %r10d,%r8d
      0.03%  ││  0x00007f4ac842695b: dec    %r8d               ;*synchronization entry
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 473)
      0.02%  ││  0x00007f4ac842695e: mov    %ecx,%r9d
      2.54%  ││  0x00007f4ac8426961: inc    %r9d               ;*invokespecial position {reexecute=0 rethrow=0 return_oop=0}
             ││                                                ; - java.nio.CharBuffer::[email protected] (line 1086)
             ││                                                ; - java.nio.CharBuffer::[email protected] (line 262)
             ││                                                ; - java.nio.Buffer::&lt;init&gt;@39 (line 223)
             ││                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
             ││                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
             ││                                                ; - java.nio.CharBuffer::[email protected] (line 348)
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.03%  ││  0x00007f4ac8426964: cmp    0x18(%rsp),%r9d
             ││  0x00007f4ac8426969: jae    0x00007f4ac8426d26  ;*aload_0 {reexecute=0 rethrow=0 return_oop=0}
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 937)
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 488)
      0.02%  ││  0x00007f4ac842696f: inc    %edi
      0.02%  ││  0x00007f4ac8426971: mov    0x10(%rsp),%r10
      2.28%  ││  0x00007f4ac8426976: mov    %edi,0xc(%r10)
      0.14%  ││  0x00007f4ac842697a: movslq %ecx,%r10
      0.05%  ││  0x00007f4ac842697d: vmovq  %xmm7,%r9
      0.02%  ││  0x00007f4ac8426982: mov    %r8w,0x12(%r9,%r10,2)  ;*invokevirtual get {reexecute=0 rethrow=0 return_oop=0}
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 937)
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 479)
      2.16%  ││  0x00007f4ac8426988: mov    0x108(%r15),%r10   ; ImmutableOopMap{r9=Oop xmm0=NarrowOop xmm1=NarrowOop xmm2=NarrowOop xmm3=NarrowOop xmm4=Oop xmm5=NarrowOop xmm6=Oop xmm7=Oop [0]=Oop [8]=Oop [16]=Oop }
             ││                                                ;*goto {reexecute=1 rethrow=0 return_oop=0}
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 531)
      0.07%  ││  0x00007f4ac842698f: test   %eax,(%r10)        ;   {poll}
      0.03%  ││  0x00007f4ac8426992: mov    (%rsp),%r10
      0.05%  ││  0x00007f4ac8426996: mov    0xc(%r10),%r10d
      2.33%  ││  0x00007f4ac842699a: vmovd  %xmm8,%r8d
      0.07%  ││  0x00007f4ac842699f: cmp    %r10d,%r8d
             ││  0x00007f4ac84269a2: jge    0x00007f4ac8426c7e  ;*invokevirtual limit {reexecute=0 rethrow=0 return_oop=0}
             ││                                                ; - java.nio.Buffer::&lt;init&gt;@33 (line 222)
             ││                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
             ││                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
             ││                                                ; - java.nio.CharBuffer::[email protected] (line 348)
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.02%  ││  0x00007f4ac84269a8: vmovaps %xmm8,%xmm10
      0.02%  ││  0x00007f4ac84269ad: mov    %r11d,%r9d         ;*putfield position {reexecute=0 rethrow=0 return_oop=0}
             ││                                                ; - java.nio.Buffer::[email protected] (line 294)
             ││                                                ; - java.nio.CharBuffer::[email protected] (line 1086)
             ││                                                ; - java.nio.CharBuffer::[email protected] (line 262)
             ││                                                ; - java.nio.Buffer::&lt;init&gt;@39 (line 223)
             ││                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
             ││                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
             ││                                                ; - java.nio.CharBuffer::[email protected] (line 348)
             ││                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      2.00%  ↘│  0x00007f4ac84269b0: mov    0x8(%rsp),%r10
      0.02%   │  0x00007f4ac84269b5: mov    0xc(%r10),%r10d    ;*iflt {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.Buffer::[email protected] (line 681)
              │                                                ; - java.nio.HeapCharBuffer::[email protected] (line 169)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 974)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 478)
      0.16%   │  0x00007f4ac84269b9: mov    %r10d,0x44(%rsp)
      0.31%   │  0x00007f4ac84269be: vmovd  %xmm9,%r10d
      2.31%   │  0x00007f4ac84269c3: cmp    0x44(%rsp),%r10d
              │  0x00007f4ac84269c8: jge    0x00007f4ac8426c7e  ;*invokevirtual limit {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.Buffer::&lt;init&gt;@33 (line 222)
              │                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
              │                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 348)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      3.34%   │  0x00007f4ac84269ce: mov    (%rsp),%r10
      0.02%   │  0x00007f4ac84269d2: mov    0xc(%r10),%r10d    ;*caload {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.HeapCharBuffer::[email protected] (line 169)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 937)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 479)
      0.24%   │  0x00007f4ac84269d6: mov    %r10d,0x30(%rsp)
      0.09%   │  0x00007f4ac84269db: vmovd  %xmm10,%r10d
      2.23%   │  0x00007f4ac84269e0: inc    %r10d              ;*invokespecial position {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.CharBuffer::[email protected] (line 1086)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 262)
              │                                                ; - java.nio.Buffer::&lt;init&gt;@39 (line 223)
              │                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
              │                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 348)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.02%   │  0x00007f4ac84269e3: vmovd  %r10d,%xmm8
      0.17%   │  0x00007f4ac84269e8: vmovd  %xmm10,%r11d
      0.09%   │  0x00007f4ac84269ed: shl    %r11d              ;*invokespecial &lt;init&gt; {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      2.52%   │  0x00007f4ac84269f0: mov    %r11d,%edi
              │  0x00007f4ac84269f3: add    $0x2,%edi          ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.Buffer::[email protected] (line 292)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 1086)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 262)
              │                                                ; - java.nio.Buffer::&lt;init&gt;@39 (line 223)
              │                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
              │                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 348)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.12%   │  0x00007f4ac84269f6: mov    %r11d,%eax
      0.12%   │  0x00007f4ac84269f9: add    0x1c(%rsp),%eax    ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::&lt;init&gt;@16 (line 320)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      2.14%   │  0x00007f4ac84269fd: add    $0x3,%r11d         ;*invokespecial position {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.CharBuffer::[email protected] (line 1086)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 262)
              │                                                ; - java.nio.Buffer::&lt;init&gt;@39 (line 223)
              │                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
              │                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 348)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.03%   │  0x00007f4ac8426a01: movslq %eax,%r14
      0.17%   │  0x00007f4ac8426a04: mov    %eax,%ebp
      0.17%   │  0x00007f4ac8426a06: add    $0x3,%ebp          ;*putfield hb {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.CharBuffer::&lt;init&gt;@12 (line 282)
              │                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 348)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      1.74%   │  0x00007f4ac8426a09: add    $0x2,%eax          ;*invokespecial &lt;init&gt; {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
              │                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 348)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.02%   │  0x00007f4ac8426a0c: cmp    0x2c(%rsp),%r9d
              │  0x00007f4ac8426a11: jle    0x00007f4ac8426870  ;*invokevirtual get {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 974)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 480)
      0.31%   │  0x00007f4ac8426a17: vmovd  %r9d,%xmm11
      0.21%   │  0x00007f4ac8426a1c: vmovd  %xmm9,%r9d
      2.66%   │  0x00007f4ac8426a21: shl    %r9d               ;*return {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.lang.Object::&lt;init&gt;@0 (line 50)
              │                                                ; - org.roaringbitmap.buffer.MappeableContainer::&lt;init&gt;@1 (line 19)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::&lt;init&gt;@1 (line 319)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.07%   │  0x00007f4ac8426a24: vmovd  %xmm9,%r8d
      0.23%   │  0x00007f4ac8426a29: inc    %r8d               ;*invokespecial position {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.CharBuffer::[email protected] (line 1086)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 262)
              │                                                ; - java.nio.Buffer::&lt;init&gt;@39 (line 223)
              │                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
              │                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 348)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.10%   │  0x00007f4ac8426a2c: mov    %r9d,%ecx
      1.84%   │  0x00007f4ac8426a2f: add    0x28(%rsp),%ecx    ;*return {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.lang.Object::&lt;init&gt;@0 (line 50)
              │                                                ; - org.roaringbitmap.buffer.MappeableContainer::&lt;init&gt;@1 (line 19)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::&lt;init&gt;@1 (line 319)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.03%   │  0x00007f4ac8426a33: mov    %r9d,%ebx
      0.07%   │  0x00007f4ac8426a36: add    $0x2,%ebx
      0.12%   │  0x00007f4ac8426a39: add    $0x3,%r9d          ;*invokevirtual position {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.CharBuffer::[email protected] (line 262)
              │                                                ; - java.nio.Buffer::&lt;init&gt;@39 (line 223)
              │                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
              │                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 348)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      2.23%   │  0x00007f4ac8426a3d: movslq %ecx,%rsi
      0.09%   │  0x00007f4ac8426a40: mov    %ecx,%edx
      0.09%   │  0x00007f4ac8426a42: add    $0x3,%edx          ;*invokespecial &lt;init&gt; {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
              │                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 348)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.19%   │  0x00007f4ac8426a45: add    $0x2,%ecx          ;*invokevirtual position {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.Buffer::&lt;init&gt;@39 (line 223)
              │                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
              │                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 348)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      2.09%   │  0x00007f4ac8426a48: mov    0x40(%rsp),%r10d
      0.05%   │  0x00007f4ac8426a4d: cmp    0x34(%rsp),%r10d
              │  0x00007f4ac8426a52: jle    0x00007f4ac84267e0  ;*invokevirtual checkIndex {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.HeapCharBuffer::[email protected] (line 169)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 974)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 480)
      0.21%   │  0x00007f4ac8426a58: mov    0x34(%rsp),%r10d
      0.24%   │  0x00007f4ac8426a5d: cmp    0x2c(%rsp),%r10d
      2.63%   │  0x00007f4ac8426a62: cmovl  0x2c(%rsp),%r10d   ;*invokevirtual limit {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.CharBuffer::[email protected] (line 262)
              │                                                ; - java.nio.Buffer::&lt;init&gt;@33 (line 222)
              │                                                ; - java.nio.CharBuffer::&lt;init&gt;@6 (line 281)
              │                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 348)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.03%   │  0x00007f4ac8426a68: vmovd  %xmm11,%r13d
      0.14%   │  0x00007f4ac8426a6d: cmp    0x40(%rsp),%r13d
              │  0x00007f4ac8426a72: je     0x00007f4ac8426b9f  ;*getfield valueslength {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 974)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 480)
      0.12%   │  0x00007f4ac8426a78: cmp    0x40(%rsp),%r13d
              │  0x00007f4ac8426a7d: jge    0x00007f4ac8426b16  ;*caload {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.HeapCharBuffer::[email protected] (line 169)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 937)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 479)
      2.02%   │  0x00007f4ac8426a83: vmovd  %xmm8,%r8d
      0.09%   │  0x00007f4ac8426a88: cmp    0x30(%rsp),%r8d
              │  0x00007f4ac8426a8d: jge    0x00007f4ac8426ca7  ;*getfield hb {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.HeapCharBuffer::[email protected] (line 169)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 974)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 478)
      0.17%   │  0x00007f4ac8426a93: test   %edi,%edi
              │  0x00007f4ac8426a95: jl     0x00007f4ac8426def
      0.09%   │  0x00007f4ac8426a9b: cmp    0x38(%rsp),%edi
              │  0x00007f4ac8426a9f: jge    0x00007f4ac8426eba  ;*putfield valueslength {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::&lt;init&gt;@36 (line 324)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      2.37%   │  0x00007f4ac8426aa5: cmp    0x20(%rsp),%eax
              │  0x00007f4ac8426aa9: jae    0x00007f4ac8426d82  ;*getfield valueslength {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 937)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 488)
      0.03%   │  0x00007f4ac8426aaf: vmovq  %xmm4,%r8
      0.05%   │  0x00007f4ac8426ab4: movzwl 0x14(%r8,%r14,2),%r8d  ;*putfield hb {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - java.nio.CharBuffer::&lt;init&gt;@12 (line 282)
              │                                                ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
              │                                                ; - java.nio.CharBuffer::[email protected] (line 348)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.99%   │  0x00007f4ac8426aba: mov    %r8d,0x34(%rsp)
      2.38%   │  0x00007f4ac8426abf: test   %r11d,%r11d
              │  0x00007f4ac8426ac2: jl     0x00007f4ac8426e4a
      0.02%   │  0x00007f4ac8426ac8: cmp    0x38(%rsp),%r11d
              │  0x00007f4ac8426acd: jge    0x00007f4ac8426f1e  ;*putfield valueslength {reexecute=0 rethrow=0 return_oop=0}
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::&lt;init&gt;@36 (line 324)
              │                                                ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
      0.09%   │  0x00007f4ac8426ad3: cmp    0x20(%rsp),%ebp
              ╰  0x00007f4ac8426ad7: jb     0x00007f4ac8426920  ;*putfield hb {reexecute=0 rethrow=0 return_oop=0}
                                                               ; - java.nio.CharBuffer::&lt;init&gt;@12 (line 282)
                                                               ; - java.nio.HeapCharBuffer::&lt;init&gt;@9 (line 61)
                                                               ; - java.nio.CharBuffer::[email protected] (line 348)
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 474)
                 0x00007f4ac8426add: mov    $0xffffffe4,%esi   ;*invokevirtual get {reexecute=0 rethrow=0 return_oop=0}
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 937)
                                                               ; - org.roaringbitmap.buffer.MappeableRunContainer::[email protected] (line 488)
    ....................................................................................................
     75.92%  <total for region 1>
    
    ....[Hottest Regions]...............................................................................
     75.92%         c2, level 4  org.roaringbitmap.buffer.MappeableRunContainer::and, version 683 (506 bytes) 
      5.51%         c2, level 4  org.roaringbitmap.buffer.MappeableRunContainer::and, version 683 (70 bytes) 
      3.83%         c2, level 4  org.roaringbitmap.buffer.MappeableRunContainer::and, version 683 (196 bytes) 
      3.13%         c2, level 4  org.roaringbitmap.buffer.MappeableRunContainer::and, version 683 (49 bytes) 
      3.08%              kernel  [unknown] (6 bytes) 
      1.17%        libc-2.27.so  __memset_avx2_erms (0 bytes) 
      0.80%              kernel  [unknown] (8 bytes) 
      0.38%        libc-2.27.so  __memmove_avx_unaligned_erms (0 bytes) 
      0.33%              kernel  [unknown] (74 bytes) 
      0.33%         c2, level 4  org.roaringbitmap.buffer.MappeableRunContainer::and, version 683 (26 bytes) 
      0.31%         c2, level 4  org.roaringbitmap.buffer.MappeableRunContainer::and, version 683 (67 bytes) 
      0.30%         c2, level 4  org.roaringbitmap.buffer.MappeableRunContainer::and, version 683 (0 bytes) 
      0.30%         c2, level 4  org.roaringbitmap.buffer.MappeableRunContainer::and, version 683 (17 bytes) 
      0.14%              kernel  [unknown] (30 bytes) 
      0.14%              kernel  [unknown] (0 bytes) 
      0.09%              kernel  [unknown] (41 bytes) 
      0.09%              kernel  [unknown] (15 bytes) 
      0.09%              kernel  [unknown] (0 bytes) 
      0.09%         c2, level 4  org.roaringbitmap.buffer.MappeableRunContainer::and, version 683 (27 bytes) 
      0.09%         c2, level 4  org.apache.druid.benchmark.BitmapIterationBenchmark::iter, version 732 (57 bytes) 
      3.88%  <...other 184 warm regions...>
    ....................................................................................................
     99.98%  <totals>
    
    ....[Hottest Methods (after inlining)]..............................................................
     89.77%         c2, level 4  org.roaringbitmap.buffer.MappeableRunContainer::and, version 683 
      6.51%              kernel  [unknown] 
      1.17%        libc-2.27.so  __memset_avx2_erms 
      1.11%           libjvm.so  [unknown] 
      0.38%        libc-2.27.so  __memmove_avx_unaligned_erms 
      0.31%         c2, level 4  org.apache.druid.benchmark.BitmapIterationBenchmark::iter, version 732 
      0.10%        libc-2.27.so  vfprintf 
      0.07%         c2, level 4  org.roaringbitmap.buffer.MutableRoaringBitmap::and, version 741 
      0.05%      hsdis-amd64.so  print_insn 
      0.05%        libc-2.27.so  __strchr_avx2 
      0.03%         interpreter  invokeinterface  185 invokeinterface  
      0.03%        libc-2.27.so  _int_malloc 
      0.03%  libpthread-2.27.so  __pthread_disable_asynccancel 
      0.03%        libc-2.27.so  _IO_old_init 
      0.02%        libc-2.27.so  __GI___libc_write 
      0.02%         interpreter  invoke return entry points  
      0.02%         interpreter  aaload  50 aaload  
      0.02%        libc-2.27.so  __stpcpy_sse2_unaligned 
      0.02%        libc-2.27.so  pthread_mutex_unlock 
      0.02%         interpreter  method entry point (kind = zerolocals)  
      0.21%  <...other 12 warm methods...>
    ....................................................................................................
     99.98%  <totals>
    
    ....[Distribution by Source]........................................................................
     90.15%         c2, level 4
      6.51%              kernel
      1.93%        libc-2.27.so
      1.11%           libjvm.so
      0.10%         interpreter
      0.07%      hsdis-amd64.so
      0.03%  libpthread-2.27.so
      0.03%         c1, level 3
      0.02%              [vdso]
      0.02%          ld-2.27.so
    ....................................................................................................
     99.98%  <totals>
    
    
    performance 
    opened by richardstartin 10
  • Efficiency of certain operations

    Efficiency of certain operations

    Hi,

    recently while calculating AND cardinality I have hit the wall, unexpectedly performance did drop significantly. After some research and debugging I came up with code which reproduces such situation. Would it be correct assumption that whenever run container appear in bitmap, it will significantly slowdown operations ?

    `

    long rangeEnd = 100_000_000;
    
    RoaringBitmap r1 = new RoaringBitmap();
    r1.add(0L, rangeEnd);
    
    Random rand = new Random();
    RoaringBitmap temp = new RoaringBitmap();
    for (int i = 0; i < rangeEnd; i++) {
        if (rand.nextBoolean()) {
            temp.add(i);
        }
    }
    RoaringBitmap r2 = RoaringBitmap.bitmapOf(temp.toArray());
    
    System.out.println("r1 bytes: " + r1.getSizeInBytes());
    System.out.println("r2 bytes: " + r2.getSizeInBytes());
    
    long cards = 0;
    List<Long> timer = new ArrayList<>();
    for (int i = 0; i < 100; i++) {
        long st = System.currentTimeMillis();
        cards += RoaringBitmap.andCardinality(r1, r2); // ~ 60 to 80 times slower than below
        // cards += RoaringBitmap.and(r1,r2).getCardinality(); // faster
        timer.add(System.currentTimeMillis() - st);
    }
    System.out.println("Millis: " + timer.stream().mapToLong(Long::longValue).summaryStatistics());
    System.out.println(cards);
    

    ` Update: What do you think about reusing bitmapContainer functionality for range cardinality scan? https://github.com/Cheappie/RoaringBitmap/commit/762de42f469634b4fab4ba0f39bb2e27f3311a68

    opened by Cheappie 6
Releases(0.9.10)
Owner
Roaring bitmaps: A better compressed bitset
Roaring bitmaps are compressed bitmaps. They can be hundreds of times faster. (Picture credit: tambako)
Roaring bitmaps: A better compressed bitset
Fast integer compression in C using the StreamVByte codec

streamvbyte StreamVByte is a new integer compression technique that applies SIMD instructions (vectorization) to Google's Group Varint approach. The n

Daniel Lemire 195 Feb 19, 2021
A simple integer compression library in Java

JavaFastPFOR: A simple integer compression library in Java License This code is released under the Apache License Version 2.0 http://www.apache.org/li

Daniel Lemire 425 Mar 3, 2021
Competitive Programming.

ℹ️ 30 Days Of Code : Coding journey! ℹ️ Ressources : ?? playlist to watch Days Solutions Challenges Points Day00 : Question Solution Hello, World. 30

Abdessamad Laamimi 16 Mar 23, 2021
Parallel boolean circuit evaluation

splitmap This library builds on top of RoaringBitmap to provide a parallel implementation of boolean circuits (multidimensional filters) and arbitrary

Richard Startin 16 Dec 10, 2020
Chronicle Bytes has a similar purpose to Java NIO's ByteBuffer with many extensions

Chronicle-Bytes Chronicle-Bytes Chronicle Bytes contains all the low level memory access wrappers. It is built on Chronicle Core’s direct memory and O

Chronicle Software : Open Source 239 Mar 12, 2021
Immutable key/value store with efficient space utilization and fast reads. They are ideal for the use-case of tables built by batch processes and shipped to multiple servers.

Minimal Perfect Hash Tables About Minimal Perfect Hash Tables are an immutable key/value store with efficient space utilization and fast reads. They a

Indeed Engineering 84 Mar 12, 2021
Micro second messaging that stores everything to disk

Chronicle Queue Contents Table of Contents Contents About Chronicle Software What is Chronicle Queue Java Docs Usage More benchmarks Downloading Chron

Chronicle Software : Open Source 2.3k Mar 13, 2021
Immutable in-memory R-tree and R*-tree implementations in Java with reactive api

rtree In-memory immutable 2D R-tree implementation in java using RxJava Observables for reactive processing of search results. Status: released to Mav

Dave Moten 860 Mar 10, 2021
Java library for the HyperLogLog algorithm

java-hll A Java implementation of HyperLogLog whose goal is to be storage-compatible with other similar offerings from Aggregate Knowledge. NOTE: This

Aggregate Knowledge (a Neustar service) 274 Feb 5, 2021
LWJGL is a Java library that enables cross-platform access to popular native APIs useful in the development of graphics (OpenGL, Vulkan), audio (OpenAL), parallel computing (OpenCL, CUDA) and XR (OpenVR, LibOVR) applications.

LWJGL - Lightweight Java Game Library 3 LWJGL (https://www.lwjgl.org) is a Java library that enables cross-platform access to popular native APIs usef

Lightweight Java Game Library 3.1k Mar 13, 2021
A java.util.HashMap compatible map that won't stall puts or gets when resizing

/* * Written by Gil Tene, based on Apache Harmony version of java.util.HashMap. */ PauselessHashMap: A java.util.HashMap compatible Map implementat

Gil Tene 144 Mar 1, 2021
A fork of Cliff Click's High Scale Library. Improved with bug fixes and a real build system.

High Scale Lib This is Boundary's fork of Cliff Click's high scale lib. We will be maintaining this fork with bug fixes, improvements and versioned bu

BMC TrueSight Pulse (formerly Boundary) 375 Feb 15, 2021
RTree2D is a 2D immutable R-tree with STR (Sort-Tile-Recursive) packing for ultra-fast nearest and intersection queries

RTree2D RTree2D is a 2D immutable R-tree with STR (Sort-Tile-Recursive) packing for ultra-fast nearest and intersection queries. Goals Main our requir

Andriy Plokhotnyuk 92 Mar 13, 2021
Reactive Streams Utilities - Future standard utilities library for Reactive Streams.

Reactive Streams Utilities This is an exploration of what a utilities library for Reactive Streams in the JDK might look like. Glossary: A short gloss

Lightbend 59 Mar 9, 2021