Move Fields from package libbeat/common to libbeat/mapping. 11198

For "metricbeat style" generated custom beats, the mage target GoTestIntegration has changed to GoIntegTest and GoTestUnit has changed to GoUnitTest. 13341

Build docker and kubernetes features only on supported platforms. 13509

Need to register new processors to be used in the JS processor in their init functions. 13509

The custom beat generator now uses mage instead of python, mage GenerateCustomBeat can be used to create a new beat, and mage vendorUpdate to update the vendored libbeat in a custom beat. 13610

Altered all remaining uses of mapval to use the renamed and enhanced version: go-lookslike instead, which is a separate project. The mapval tree is now gone. 14165

Move light modules to OSS. 14369

Deprecate test flags, generate and update_expected, in favor of data. 15292

Python 3 is required now to run python tests and tools. 14798

The type memqueue.Broker is no longer exported; instead of memqueue.NewBroker, call memqueue.NewQueue (which provides the same public interface). 16667

The disk spool types spool.Spool and spool.Settings have been renamed to the internal types spool.diskSpool and spool.settings. 16693

queue.Eventer has been renamed to queue.ACKListener 16691

Require logger as first parameter for outputs.transport.transport#ProxyDialer and outputs.elasticsearch.client#BulkReadItemStatus. 16761

The libbeat/outputs/transport package has been moved to libbeat/common/transport. 16734

The libbeat/outputs/tls.go file has been removed. All exported symbols in that file (libbeat/outputs.) are now available as libbeat/common/tlscommon.. 16734

The newly generated Beats are using go modules to manage dependencies. 16288

Extract Elasticsearch client logic from outputs/elasticsearch package into new esclientleg package. 16150

Rename queue.BufferConfig.Events to queue.BufferConfig.MaxEvents. 17622

Remove queue.Feature and replace queue.RegisterType with queue.RegisterQueueType. 17666

Introduce APM libbeat instrumentation. Publish method on Client interface now takes a Context as first argument. 17938

The way configuration files are generated has changed to make it easier to customize parts of the config without requiring changes to libbeat config templates. Generation is now fully based on Go text/template and no longer uses file concatenation to generate the config. Your magefile.go will require a change to adapt the devtool API. See the pull request for more details. 18148

The Elasticsearch client settings expect the API key to be raw (not base64-encoded). 18939 18945

management.ConfigManager has been renamed to management.Manager. 19114

UpdateStatus has been added to the management.Manager interface. 19114

Remove common.MapStrPointer parameter from cfgfile.Runnerfactory interface. 19135

Replace ACKCount, ACKEvents, and ACKLastEvent callbacks with ACKHandler and interface in beat.ClientConfig. 19632

Remove global ACK handler support via SetACKHandler from publisher pipeline. 19632

Make implementing Close required for reader.Reader interfaces. 20455

Remove NumCPU as clients should update the CPU count on the fly in case of config changes in a VM. 23154

Remove Metricbeat EventFetcher and EventsFetcher interface. Use the reporter interface instead. 25093

Update Darwin build image to a debian 10 base that increases the MacOS SDK and minimum supported version used in build to 10.14. 24193

Removed the common.Float type. 28279 28280 28376

Removed Beat generators. 28816

libbeat.logp package forces ECS compliant logs. Logs are JSON formatted. Options to enable ECS/JSON have been removed. 15544 28573

Removed deprecated disk spool from Beats. Use disk queue instead. 28869

Wildcard fields no longer have a default ignore_above setting of 1024. 30096 30668

Remove common.MapStr and use mapstr.M from github.com/elastic/elastic-agent-libs instead. 31420

Remove queue.Consumer. Queues can now be read via a Get call directly on the queue object. 31502

The queue.Batch API now provides access to individual events instead of an array. 31699

Rename queue.Batch.ACK() to queue.Batch.Done(). 31903

queue.ACKListener has been removed. Queue configurations now accept an explicit callback function for ACK handling. 35078

Split split httpmon out of x-pack/filebeat/input/internal/httplog. 36385

Fix how Prometheus histograms are calculated when percentiles are provide.https://github.com/elastic/beats/pull/36537[36537]

Stop using mage:import in community beats. This was ignoring the vendorized beats directory for some mage targets, using the code available in GOPATH, this causes inconsistencies and compilation problems if the version of the code in the GOPATH is different to the vendored one. Use of mage:import will continue to be unsupported in custom beats till beats is migrated to go modules, or mage supports vendored dependencies. 13998 14162

Metricbeat module builders call host parser only once when instantiating light modules. 20149

Fix export dashboard command when running against Elastic Cloud hosted Kibana. 22746

Remove event.dataset (ECS) annotion from libbeat.logp. 27404

Errors should be thrown as errors. Metricsets inside Metricbeat will now throw errors as the error log level. 27804

Avoid panicking in add_fields processor when input event.Fields is a nil map. 28219

Drop event batch when get HTTP status 413 from Elasticsearch to avoid infinite loop 14350 29368

Allow to use metricbeat for named mssql instances. 24076 30859

Setting DEV=true when running mage build now correctly generates binaries without optimisations and with debug symbols 31955

The beat.cgroup.memory.mem.usage.bytes metric is now a gauge 31582 32652

Fix the integration testcase docker port mapping for sql and oracle modules 34221

Fix the ingest pipeline for mysql slowlog to parse schema name with dash 34372

Fix the multiple host support for mongodb module 34624

Skip HTTPJSON flakey test. 34929 35138

Fix ingest pipeline for panw module to parse url scheme correctly 35757

Renamed an httpjson input metric to follow naming conventions. httpjson_interval_pages_total was renamed to httpjson_interval_pages because the _total suffix is reserved for counters. 35933 36169

Fixed some race conditions in tests 36185

Fix Stringer implementation of fingerprint processor 35174

Re-enable HTTPJSON fixed flakey test. 34929 36525

Make winlogbeat/sys/wineventlog follow the unsafe.Pointer rules. 36650

Cleaned up documentation errors & fixed a minor bug in Filebeat Azure blob storage input. 36714

Fix copy arguments for strict aligned architectures. 36976

Add new metricset in Kubernetes module, state_namespace. 36406

Add configuration for APM instrumentation and expose the tracer trough the Beat object. 17938

Make the behavior of clientWorker and netClientWorker consistent when error is returned from publisher pipeline

Metricset generator generates beta modules by default now. 10657

The beat.Event accessor methods now support @metadata keys. 10761

Assertion for documented fields in tests fails if any of the fields in the tested event is documented as an alias. 10921

Support for Logger in the Metricset base instance. 11106

Filebeat modules can now use ingest pipelines in YAML format. 11209

Prometheus helper for metricbeat contains now Namespace field for prometheus.MetricsMappings 11424

Update Jinja2 version to 2.10.1. 11817

Reduce idxmgmt.Supporter interface and rework export commands to reuse logic. 11777,https://github.com/elastic/beats/pull/12065[12065],https://github.com/elastic/beats/pull/12067[12067],https://github.com/elastic/beats/pull/12160[12160]

Update urllib3 version to 1.24.2 11930

Add libbeat/common/cleanup package. 12134

New helper to check for leaked goroutines on tests. 12106

Only Load minimal template if no fields are provided. 12103

Add new option IgnoreAllErrors to libbeat.common.schema for skipping fields that failed while converting. 12089

Deprecate setup cmds for template and ilm-policy. Add new setup cmd for index-management. 12132

Use the go-lookslike library for testing in heartbeat. Eventually the mapval package will be replaced with it. 12540

New ReporterV2 interfaces that can receive a context on Fetch(ctx, reporter), or Run(ctx, reporter). 11981

Generate configuration from mage for all Beats. 12618

Add ClientFactory to TCP input source to add SplitFunc/NetworkFuncs per client. 8543

Introduce beat.OutputChooses publisher mode. 12996

Ensure that beat.Processor, beat.ProcessorList, and processors.ProcessorList are compatible and can be composed more easily. 12996

Add support to close beat.Client via beat.CloseRef (a subset of context.Context). 13031

Add checks for types and formats used in fields definitions in fields.yml files. 13188

Makefile included in generator copies files from beats repository using git archive instead of cp. 13193

Strip debug symbols from binaries to reduce binary sizes. 12768

Compare event by event in testadata framework to avoid sorting problems 13747

Added a default_field option to fields in fields.yml to offer a way to exclude fields from the default_field list. 14262 14341

supported-versions.yml can be used in metricbeat python system tests to obtain the build args for docker compose builds. 14520

Fix dropped errors in the tests for the metricbeat Azure module. 13773

New mage target for Functionbeat: generate pkg folder to make manager easier. 15880

Add support for MODULE environment variable in mage goIntegTest in metricbeat to run integration tests for a single module. 17147

Add support for a TEST_TAGS environment variable to add tags for tests selection following go build tags semantics, this environment variable is used by mage test targets to add build tags. Python tests can also be tagged with a decorator (@beat.tag('sometag')). 16937 17075

Add fields validation for histogram subfields. 17759

Add IP* fields to fields.yml generator script in Filebeat. 17998 18256

Events intended for the Elasticsearch output can now take an op_type metadata field of type events.OpType or string to indicate the op_type to use for bulk indexing. 12606

Remove vendor folder from repository. 18655

Added SQL helper that can be used from any Metricbeat module 18955

Update Go version to 1.14.4. 19753

Update Go version to 1.14.7. 20508

Add packaging for docker image based on UBI minimal 8. 20576

Make the mage binary used by the build process in the docker container to be statically compiled. 20827

Add Pensando distributed firewall module. 21063

Update ecszap to v0.3.0 for using ECS 1.6.0 in logs 22267

Add support for customized monitoring API. 22605

Update Go version to 1.15.7. 22495

Update Go version to 1.15.8. 23955

Update Go version to 1.15.9. 24442

Update Go version to 1.15.10. 24606

Update Go version to 1.15.12. 25629

Update Go version to 1.16.4. 25346 25671

Add sorting to array fields for generated data files (*-generated.json) 25320

Update to go-concert 0.2.0 27162

Update Go version to 1.16.5. 26182 26186

Introduce libbeat/beat.Beat.OutputConfigReloader 28048

Update Go version to 1.17.1. 27543

Whitelist GCP_* environment variables in dev tools 28364

Add support for credentials_json in gcp module, all metricsets 29584

Add gcp firestore metricset. 29918

Added TESTING_FILEBEAT_FILEPATTERN option for filebeat module pytests 30103

Improve tests files with shorter statements. 35667

Add gcp dataproc metricset. 30008

Add Github action for linting

Add regex support for drop_fields processor.

Improve compatibility and reduce flakyness of Python tests 31588

Added .python-version file 32323

Use T.TempDir to create temporary test directory 33082

Add an option to disable event normalization when creating a beat.Client. 33657

Add the file path of the instance lock on the error when it’s is already locked 33788

Add DropFields processor to js API 33458

Add support for different folders when testing data 34467

Add logging of metric registration in inputmon. 35647

Add Okta API package for entity analytics. 35478

Add benchmarking to HTTPJSON input testing. 35138

Allow non-AWS endpoints for testing Filebeat awss3 input. 35496 35520

Add AUTH (username) and SSL/TLS support for Redis module 35240

Pin PyYAML version to 5.3.1 to avoid CI errors temporarily 36091

Skip dependabot updates for github.com/elastic/mito. 36158

Add device handling to Okta API package for entity analytics. 35980

Make Filebeat HTTPJSON input process responses sequentially. 36493

Add initial infrastructure for a caching enrichment processor. 36619

Add file-backed cache for cache enrichment processor. 36686 36696

Elide retryable HTTP client construction in Filebeat HTTPJSON and CEL inputs if not needed. 36916

Allow assignment of packetbeat protocols to interfaces. 36564 36852

Add Active Directory entity collector for Filebeat entity analytics. 37854

Deprecated the common.Float type. 28279 28280

Deprecate Beat generators. 28814

Remove garbled PE executable test from auditbeat FIM module testing. 35705 35724