Sawmill is a JSON transformation Java library

Overview

Sawmill Logo

Build Status Maven Central

Update: June 25, 2020 The 2.0 release of Sawmill introduces a breaking change to the GeoIpProcessor to comply with the updated license of the MaxMind Lite database. See https://github.com/logzio/sawmill/wiki/GeoIp-Processor for additional details.

Sawmill is a JSON transformation open source library.

It enables you to enrich, transform, and filter your JSON documents.

Using Sawmill pipelines you can integrate your favorite groks, geoip, user-agent resolving, add or remove fields/tags and more in a descriptive manner, using configuration files or builders, in a simple DSL, allowing you to dynamically change transformations.

Download

Get Sawmill Java via Maven:

<dependency>
    <groupId>io.logz.sawmill</groupId>
    <artifactId>sawmill-core</artifactId>
    <version>2.0.13</version>
</dependency>

or Gradle:

compile 'io.logz.sawmill:sawmill-core:2.0.13'

Documentation

The full Sawmill documentation can be found here.

Simple configuration example

{
  "steps": [
    {
      "grok": {
        "config": {
          "field": "message",
          "overwrite": [
            "message"
          ],
          "patterns": [
            "(%{IPORHOST:client_ip}|-) %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \\\"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion:float})?|%{DATA:rawrequest})\\\" %{NUMBER:response:int} (?:%{NUMBER:bytes:float}|-) B %{DATA:thread} %{NUMBER:response_time:float} ms %{DATA:servername} %{DATA:client_id:int}(\\;%{NOTSPACE})? %{DATA:device_id} %{DATA}"
          ]
        }
      }
    },
    {
      "removeField": {
        "config": {
          "path": "message"
        }
      }
    }
  ]
}
Issues
  • Automate tests

    Automate tests

    opened by roiravhon 11
  • change version to 1.1.18

    change version to 1.1.18

    opened by ofir76 4
  • Does Sawmill Stream Large Documents?

    Does Sawmill Stream Large Documents?

    Just wondering if Sawmill streams the data into memory as processors perform their work or does it load the files all at once? Some files can be really large and wondering if it buffers data to perform the transformations?

    opened by adamfisher 2
  • Bump jackson-databind from 2.9.10.1 to 2.9.10.4

    Bump jackson-databind from 2.9.10.1 to 2.9.10.4

    Bumps jackson-databind from 2.9.10.1 to 2.9.10.4.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 2
  • DEV-3735: dns processor

    DEV-3735: dns processor

    opened by oririechman 2
  • Added sawmill micro benchmark via jmh

    Added sawmill micro benchmark via jmh

    Added a set of micro-benchmarks tests for comparing performance of individual pipelines and processors.

    opened by barakm 2
  • Failing Commit after the passing one

    Failing Commit after the passing one

    The commit 8f2981b933051f6061777b861d50560d3c322492 successfully builds the project. Author: templatevk [email protected] Date: Fri Jun 12 12:19:32 2020 +0300

    _Introduced GeoIpConfiguration configuration_

    Whereas the previous commit 871b4ceacbc9a9e7e13d95ad8cc12585a5f8291d leads to a build failure. Author: templatevk [email protected] Date: Fri Jun 12 12:14:05 2020 +0300

    Added processor factory configuration to inject processor creation stage dependencies

    Changes in the failing commit: it compiles the code but the test fails. $ git show 871b4ceacbc9a9e7e13d95ad8cc12585a5f8291d

    Link for the changes in the failing commit. The build has failed too.

    ..... Show Error- All saw-mill core files have changed and the below sawmill-core shows failure ... [WARNING] The project io.logz.sawmill:sawmill-benchmark:jar:0-SNAPSHOT uses prerequisites which is only intended for maven-plugin projects but not for non maven-plugin projects. For such purposes you should use the maven-enforcer-plugin. See https://maven.apache.org/enforcer/enforcer-rules/requireMavenVersion.html ... [INFO] Reactor Summary for Sawmill 0-SNAPSHOT: [INFO] [INFO] Sawmill ............................................ SUCCESS [ 0.109 s] [INFO] sawmill-core ....................................... FAILURE [ 4.377 s] [INFO] sawmill-benchmark .................................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.200 s [INFO] Finished at: 2020-09-11T21:21:47-05:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.3:run (download-files) on project sawmill-core: An Ant BuildException has occured: java.net.UnknownHostException: geolite.maxmind.com: Unknown host geolite.maxmind.com -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :sawmill-core

    Maven Version-

    [[email protected] sawmill]$ mvn --version Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) Maven home: /usr/local/src/apache-maven Java version: 1.8.0_262, vendor: Oracle Corporation, runtime: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-1127.18.2.el7.x86_64", arch: "amd64", family: "unix"

    Java Version-

    [[email protected] sawmill]$ mvn --version Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) Maven home: /usr/local/src/apache-maven Java version: 1.8.0_262, vendor: Oracle Corporation, runtime: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-1127.18.2.el7.x86_64", arch: "amd64", family: "unix" [[email protected] sawmill]$ java -version openjdk version "1.8.0_262" OpenJDK Runtime Environment (build 1.8.0_262-b10) OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)

    INFERENCE - in the passing commit after the failing one:

    Failing commit -

    ... 
    [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.3:run (download-files) on project sawmill-core: An Ant BuildException has occured: java.net.UnknownHostException: geolite.maxmind.com: Unknown host geolite.maxmind.com -> [Help 1] 
    ...
    

    Passing commit, the following was removed from the POM file: . .. http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz ... from the build ...

    opened by Shilpa-Rani 0
  • Bump junit from 4.11 to 4.13.1

    Bump junit from 4.11 to 4.13.1

    Bumps junit from 4.11 to 4.13.1.

    Release notes

    Sourced from junit's releases.

    JUnit 4.13.1

    Please refer to the release notes for details.

    JUnit 4.13

    Please refer to the release notes for details.

    JUnit 4.13 RC 2

    Please refer to the release notes for details.

    JUnit 4.13 RC 1

    Please refer to the release notes for details.

    JUnit 4.13 Beta 3

    Please refer to the release notes for details.

    JUnit 4.13 Beta 2

    Please refer to the release notes for details.

    JUnit 4.13 Beta 1

    Please refer to the release notes for details.

    JUnit 4.12

    Please refer to the release notes for details.

    JUnit 4.12 Beta 3

    Please refer to the release notes for details.

    JUnit 4.12 Beta 2

    No release notes provided.

    JUnit 4.12 Beta 1

    No release notes provided.

    Commits
    • 1b683f4 [maven-release-plugin] prepare release r4.13.1
    • ce6ce3a Draft 4.13.1 release notes
    • c29dd82 Change version to 4.13.1-SNAPSHOT
    • 1d17486 Add a link to assertThrows in exception testing
    • 543905d Use separate line for annotation in Javadoc
    • 510e906 Add sub headlines to class Javadoc
    • 610155b Merge pull request from GHSA-269g-pwp5-87pp
    • b6cfd1e Explicitly wrap float parameter for consistency (#1671)
    • a5d205c Fix GitHub link in FAQ (#1672)
    • 3a5c6b4 Deprecated since jdk9 replacing constructor instance of Double and Float (#1660)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • The project uses

    The project uses "jackson-module-afterburner" which does not work well with Java 9+

    The Jackson module uses illegal reflective access. You can see https://github.com/FasterXML/jackson-modules-base/issues/37 for more details.

    opened by sergeykad 0
  • Bump jackson-databind from 2.9.10.1 to 2.10.0.pr1

    Bump jackson-databind from 2.9.10.1 to 2.10.0.pr1

    Bumps jackson-databind from 2.9.10.1 to 2.10.0.pr1.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 1
  • Update README.md

    Update README.md

    Updating the date we expect this to be fixed.

    opened by jkowall 0
  • Base64 Decode Processor

    Base64 Decode Processor

    A customer has requested the ability to decode Base64 encoded fields

    opened by bgeveritt 0
  • JSONpath support

    JSONpath support

    How is it difficult to add JSON path support? There are cases when doted fqdn is just impossible to know upfront.

    {  
      addField: {  
        config: {  
           jsonpath: "$..author"
           value: "replace"
        }
      }
    }
    
    opened by Mihailoff 1
  • add patterns for pfsense / opnsense firewall logs

    add patterns for pfsense / opnsense firewall logs

    copied from https://github.com/patrickjennings/logstash-pfsense/blob/master/patterns/pfsense2-4.grok

    opened by ctramnitz 0
Releases(2.0.15)
Owner
Logz.io
Logz.io combines open source log analytics and behavioral learning intelligence to pinpoint what’s actually important
Logz.io
MapNeat is a JVM library written in Kotlin that provides an easy to use DSL (Domain Specific Language) for transforming JSON to JSON, XML to JSON, POJO to JSON in a declarative way.

MapNeat is a JVM library written in Kotlin that provides an easy to use DSL (Domain Specific Language) for transforming JSON to JSON, XML to JSON, POJ

Andrei Ciobanu 17 Mar 4, 2021
Compare JSON in your Unit Tests

JsonUnit JsonUnit is a library that simplifies JSON comparison in tests. APIs AssertJ integration Hamcrest matchers Spring MVC assertions Spring WebTe

Lukáš Křečan 529 Mar 12, 2021
High performance JVM JSON library

DSL-JSON library Fastest JVM (Java/Android/Scala/Kotlin) JSON library with advanced compile-time databinding support. Compatible with DSL Platform. Ja

New Generation Software Ltd 704 Mar 11, 2021
A modern JSON library for Kotlin and Java.

Moshi Moshi is a modern JSON library for Android and Java. It makes it easy to parse JSON into Java objects: String json = ...; Moshi moshi = new Mos

Square 7.1k Mar 13, 2021
Main Portal page for the Jackson project

Jackson Project Home @github This is the home page of the Jackson Project. What is New? Oct 1, 2020: Jackson participates in Hacktoberfest2020 and we

FasterXML, LLC 6.5k Apr 29, 2021
A reference implementation of a JSON package in Java.

JSON in Java [package org.json] Click here if you just want the latest release jar file. Overview JSON is a light-weight language-independent data int

Sean Leary 3.8k Apr 29, 2021
JSON to JSON transformation library written in Java.

Jolt JSON to JSON transformation library written in Java where the "specification" for the transform is itself a JSON document. Useful For Transformin

Bazaarvoice 1k Mar 11, 2021
Convert Java to JSON. Convert JSON to Java. Pretty print JSON. Java JSON serializer.

json-io Perfect Java serialization to and from JSON format (available on Maven Central). To include in your project: <dependency> <groupId>com.cedar

John DeRegnaucourt 272 Feb 20, 2021
A JSON Schema validation implementation in pure Java, which aims for correctness and performance, in that order

Read me first The current version of this project is licensed under both LGPLv3 (or later) and ASL 2.0. The old version (2.0.x) was licensed under LGP

Java Json Tools 1.4k Mar 14, 2021
Java JsonPath implementation

Jayway JsonPath A Java DSL for reading JSON documents. Jayway JsonPath is a Java port of Stefan Goessner JsonPath implementation. News 10 Dec 2020 - R

null 5.7k Mar 13, 2021
Java JsonPath implementation

Jayway JsonPath A Java DSL for reading JSON documents. Jayway JsonPath is a Java port of Stefan Goessner JsonPath implementation. News 10 Dec 2020 - R

null 5.7k Mar 15, 2021
A streaming JsonPath processor in Java

JsonSurfer - Let's surf on Json! Why JsonSurfer Streaming No need to deserialize entire json into memory. JsonPath Selectively extract json data by th

null 201 Feb 4, 2021
JSON query and transformation language

JSLT JSLT is a complete query and transformation language for JSON. The language design is inspired by jq, XPath, and XQuery. JSLT can be used as: a q

Schibsted Media Group 377 Apr 14, 2021
Fast JSON parser for java projects

ig-json-parser Fast JSON parser for java projects. Getting started The easiest way to get started is to look at maven-example. For more comprehensive

Instagram 1.3k Mar 7, 2021