Goreplay Versions Save

• Engine CPU consumption reduced by 60%
• Implemented M'mapped AF_PACKET for Linux. To enable set --input-raw-engine af_packet. #807
• Added Kafka TLS configuration flag #800
• Fix wrong param value updated while --http-set-param #791
• The engine can now capture more requests with large payload #797
• Created TCP package #97
• Fixed minor bugs in HTTP proto parser

• Fix usage of multiple --input-raw plugins, e.g. --input-raw :80 --input-raw :81 --input-raw :82 https://github.com/buger/goreplay/pull/700
• Allow to set byte size limits in human friedly format: --output-file-size-limit, --output-file-max-size-limit, --copy-buffer-size and --input-raw-buffer-size can now parse inputs from differents bases and data units like: 10mb, 10kb, 100gb, 18t https://github.com/buger/goreplay/pull/754
• Fix calculation of the output file size when using gzip compression https://github.com/buger/goreplay/pull/777
• Added --output-tcp-sticky so request/response with same ID sent to the same connection.
• Better handling of malformed middleware output https://github.com/buger/goreplay/pull/737
• Fixed race when reading or writing using files https://github.com/buger/goreplay/pull/756 https://github.com/buger/goreplay/pull/764
• Improved errors verbosity https://github.com/buger/goreplay/issues/703
• Updated dependencies, including libpcap which should bring more stability and performance
• Move to Go modules
• PRO code moved to the main repository under separate license

Long awaited major release of GoReplay!

• Added option to specify custom BPF filter (e.g. tcpdump syntax). Can be useful in case of non standard network interfaces like tunneling or SPAN port. Example: --input-raw-bpf-filter 'dst port 80’ https://github.com/buger/goreplay/pull/478
• Support for reading directly from pcap file, using --input-raw-engine pcap_file --input-raw ./recording.pcap_
• Added official Docker image docker pull buger/goreplay. Should be run with --network host argument.
• Added RPM and DEB packages
• Added support for HTTP proxies. Just set HTTP_PROXY env variable.
• Added HTTP basic auth filter: --http-basic-auth-filter "^customer[0-9].*. You specify regexp which match userid:passwd string. [https://github.com/buger/goreplay/pull/475]
• Added option to limit size of output file: when limit is reached, it will exit. Can be used for safety reasons. Example: --output-file-max-size-limit 5gb.
• Added way to configure output HTTO request queue, which used to hold requests, if all workers are busy. Example: --output-http-queue-len 5000. Default is 1000.
• Control frequency of --output-http-stats reports using --output-http-stats-msoption. Example: --output-http-stats-ms 5000 (every 5 seconds).
• Configurable way to set minimum number of HTTP workers, by setting --output-http-workers-min. Can be used in conjunction with --output-http-workers which will act as a max worker count.
• Added way to dynamically profile GoReplay performance using Golang pprof tools. Example: --http-pprof :8181. It starts web server on given address, and expose special /debug/pprof endpoint with list of reports.
• Added —input-raw-buffer-size which controls size of the OS buffer (in bytes) which holds packets until they dispatched. Default value depends by system: in Linux around 2MB. If you see big package drop, increase this value.
• In addition to buffer size, added option to turn pcap immediate mode when packets delivered without buffering. Can help reduce packet drop. Example: --input-raw-immediate-mode.
• Added way to use standard Golang HTTP client, by adding --output-http-compatibility-mode
• Snaplen (max number of bytes being read for each packet) now dynamically set based on interface MTU + max header size. In most situations it should reduce package drop, because each packet will consume less space in the buffer. However in some virtualized environments like OpenShift, packet size can be significantly bigger then MTU, so you can disable optimization by setting --input-raw-override-snaplen.
• Added support for ElasticSearch basic HTTP auth
• Numerous fixes to improve quality of HTTP packets parsing

• Added support for HTTP proxies. Just set HTTP_PROXY env variable.

• Add support for TLS connections between --input-tcp and --output-tcp. --input-tcp-secure --input-tcp-certificate ./cert.pem --input-tcp-certificate-key ./key.pem --output-tcp-secure #457
• Add basic support for old server, like HTTP 0.9 #468 #463 #467
• GoReplay now exit once finished replaying with --input-file #456
• Fix prettifier issue, when there is problems with gzip encoding payload #470

Finally a new big release, a lot of changes on all fronts. New features, usability stability, fixes.

Thank you, everyone, who made it happen!

It would not be possible without this people @smostovoy @exklamationmark @manjeshnilange @kudos @oivoodoo @sattvik @ylegat @nrwiersma @SophisticaSean

CHANGELOG

Major

• [PRO] Added S3 input and output https://github.com/buger/goreplay/wiki/%5BPRO%5D-Using-S3-for-storing-and-replaying-traffic
• Added new NodeJS middleware framework https://github.com/buger/goreplay/tree/master/middleware
• Add Kafka input and output https://github.com/buger/goreplay/wiki/Streaming-from-and-to-Apache-Kafka
• Improve accuracy of replay by using timestamp of when TCP packet was received by network interface (previously used time of package capture by GoReplay)
• Using —prettify-http option, you can automatically decode Gzip encoded responses, and de-construct chunked bodies.
• Now you can enable HTTP response tracking by providing —output-http-track-respose. Previously was available only if middleware is turned on.

Minor

• Add new option --input-raw-expire to configure TCP message expiration
• Filter response if its request was filtered
• Allow space inside filters syntax: `--http-disallow-header 'Host: www.vertaa.fi' now valid syntax
• File name pattern now support request id %r, will log each request to separate file, and %t for payload type (0 - request, 1 - response, 2 - replayed response).
• Add basic SNI support to support HTTP replay to hosts that require SNI, such as Amazon API Gateway. API Gateway)
• Fix --output-file-size-limit option
• Do not add port to Host header #383
• Improve malformed TCP packet handling to avoid panics
• Fix HTTP timeout for 204 (No content) responses
• Fix handling of HTTP error codes like 400 or 304.
• Fixed replay of HTTP PATCH requests

• Add support for basic http auth for --output-http 1674b58
• Fix handling of "Connection: close" for POST requests fc0361d
• Fix inconsistency in replayed response meta when used in middleware a2f5e43

• Gopacket package now vendored same as the rest of 3-rd party dependencies

• [PRO] Added support for working with binary protocols (thrift/protocol buffers)
• [PRO] Recording and replaying keep alive TCP sessions
• Replaying request in proper order when using multiple files #300
• Validate bodies when Content-Length or Transfer-Encoding found #317
• Return of ElasticSearch support! #331 #333
• Added --exit-after option to specify duration after which Gor will exit #336
• Updated response output file format, now third value is timestamp and forth is latency
• Properly read body when no Content-Length but there is Connection: close d34c27c
• Remove --input-http option (too confusing and never really worked) f0acd31
• Force Go DNS resolver (C one caused crashes for multiple people) 07fa6d9
• Fix interception on loopback interface when non-local IP used 1ed8691
• Fix intercepting traffic from virtual interfaces 51860e1
• Fix 100-continue header when it places not in the end #314
• Fix relative file names when using --output-file #304
• Properly cleanup used resources on exit #305
• Fix connection timeout when using --output-http-timeout c4271ff