### 代码结构
添加的代码在clients模块内。
属于原项目改动的文件：
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
主要在KafkaProducer的构造函数里启动了自己设计的模块
因此可以仅编译clients模块生成用于client的jar包，替换原有kafka的client的jar包。

### 添加的代码：
clients/src/main/java/org/apache/kafka/upgrade
下有
/dependency 依赖的jar包，三个
/JMX 
JMX.java 通过jmx监控kafka性能，默认不开启，可忽略
/KafkaProducerWatcher 
BestConf.java 描述如何根据预测流量调整包的大小
KafkaProducerWatcher.java 内含三个类
	Measurer 工作的类，负责采集流速样本
	timeSeqSample 表示一个流速样本
	KafkaProducerWatcher 控制Measurer的包装类
		RProcessing.java 内含四个类
RProcessing 维护时间序列样本，控制采样间隔
		RForecast 发送时间样本到Rserve服务，执行预测代码，返回预测结果
		RFormatSample，RForecastFormat 为了对接预测脚本对时间序列格式化的类

### 工作流程
用户的新建KafkaProducer实例时，一个对应的KafkaProducerWatcher被建立，于是Measurer被建立，RProcessing被建立。
当数据通过KafkaProducer.Send()方法进入发送流程时，数据的大小会被KafkaProducer统计到，Measurer会不断累加观测到的数据大小，直到观测窗口结束，对累计的数据大小除以观测窗口的长度就能得到数据流量的速度，这样就能得到一个样本点，样本点在RProcessing中维护。
RProcessing会根据得到的样本点的值来设置下次观测开始的时间以及观测时长。当观测点超过一定个数，RProcessing就将样本序列交给RForecast，RForecast会单独启动一个线程连接远程Rserve服务，得到对未来流量的预测，这之后，预测有BestConf转化为对batchsize的调整。
实现原理参考论文 [流式数据处理的动态自适应缓存策略研究](http://www.jsjkx.com/CN/article/openArticlePDF.jsp?id=19453) ，调整BestConf.java中的参数。

### 远程Rserve
推荐在一台windows服务器上建立Rserve服务
测试过的R版本3.5.3
安装R后安装五个包
install.packages(“Rserve”)
install.packages(“zoo”)
install.packages(“tseries”)
install.packages(“fUnitRoots”)
install.packages(“forecast”)
将Rserve的安装目录下（<R安装路径>\library\Rserve\libs\x64）下三个文件拷到<R安装目录>\bin\x64下
R CMD Rserve --help查看帮助
一般使用R CMD Rserve --RS-enable-remote启动即可
保证kafka服务器可以与Rserve服务器之间ping通

=================  
Apache Kafka
=================
See our [web site](http://kafka.apache.org) for details on the project.

You need to have [Gradle](http://www.gradle.org/installation) and [Java](http://www.oracle.com/technetwork/java/javase/downloads/index.html) installed.

Kafka requires Gradle 4.7 or higher.

Java 8 should be used for building in order to support both Java 8 and Java 10 at runtime.

### First bootstrap and download the wrapper ###
    cd kafka_source_dir
    gradle

Now everything else will work.

### Build a jar and run it ###
    ./gradlew jar

Follow instructions in http://kafka.apache.org/documentation.html#quickstart

### Build source jar ###
    ./gradlew srcJar

### Build aggregated javadoc ###
    ./gradlew aggregatedJavadoc

### Build javadoc and scaladoc ###
    ./gradlew javadoc
    ./gradlew javadocJar # builds a javadoc jar for each module
    ./gradlew scaladoc
    ./gradlew scaladocJar # builds a scaladoc jar for each module
    ./gradlew docsJar # builds both (if applicable) javadoc and scaladoc jars for each module

### Run unit/integration tests ###
    ./gradlew test # runs both unit and integration tests
    ./gradlew unitTest
    ./gradlew integrationTest
    
### Force re-running tests without code change ###
    ./gradlew cleanTest test
    ./gradlew cleanTest unitTest
    ./gradlew cleanTest integrationTest

### Running a particular unit/integration test ###
    ./gradlew -Dtest.single=RequestResponseSerializationTest core:test

### Running a particular test method within a unit/integration test ###
    ./gradlew core:test --tests kafka.api.ProducerFailureHandlingTest.testCannotSendToInternalTopic
    ./gradlew clients:test --tests org.apache.kafka.clients.MetadataTest.testMetadataUpdateWaitTime

### Running a particular unit/integration test with log4j output ###
Change the log4j setting in either `clients/src/test/resources/log4j.properties` or `core/src/test/resources/log4j.properties`

./gradlew -i -Dtest.single=RequestResponseSerializationTest core:test

### Generating test coverage reports ###
Generate coverage reports for the whole project:

./gradlew reportCoverage

Generate coverage for a single module, i.e.:

./gradlew clients:reportCoverage
    
### Building a binary release gzipped tar ball ###
    ./gradlew clean
    ./gradlew releaseTarGz

The above command will fail if you haven't set up the signing key. To bypass signing the artifact, you can run:

./gradlew releaseTarGz -x signArchives

The release file can be found inside `./core/build/distributions/`.

### Cleaning the build ###
    ./gradlew clean

### Running a task on a particular version of Scala (either 2.11.x or 2.12.x) ###
*Note that if building the jars with a version other than 2.11.12, you need to set the `SCALA_VERSION` variable or change it in `bin/kafka-run-class.sh` to run the quick start.*

You can pass either the major version (eg 2.11) or the full version (eg 2.11.12):

./gradlew -PscalaVersion=2.11 jar
    ./gradlew -PscalaVersion=2.11 test
    ./gradlew -PscalaVersion=2.11 releaseTarGz

### Running a task for a specific project ###
This is for `core`, `examples` and `clients`

./gradlew core:jar
    ./gradlew core:test

### Listing all gradle tasks ###
    ./gradlew tasks

### Building IDE project ####
*Note that this is not strictly necessary (IntelliJ IDEA has good built-in support for Gradle projects, for example).*

./gradlew eclipse
    ./gradlew idea

The `eclipse` task has been configured to use `${project_dir}/build_eclipse` as Eclipse's build directory. Eclipse's default
build directory (`${project_dir}/bin`) clashes with Kafka's scripts directory and we don't use Gradle's build directory
to avoid known issues with this configuration.

### Building the jar for all scala versions and for all projects ###
    ./gradlew jarAll

### Running unit/integration tests for all scala versions and for all projects ###
    ./gradlew testAll

### Building a binary release gzipped tar ball for all scala versions ###
    ./gradlew releaseTarGzAll

### Publishing the jar for all version of Scala and for all projects to maven ###
    ./gradlew uploadArchivesAll

Please note for this to work you should create/update `${GRADLE_USER_HOME}/gradle.properties` (typically, `~/.gradle/gradle.properties`) and assign the following variables

mavenUrl=
    mavenUsername=
    mavenPassword=
    signing.keyId=
    signing.password=
    signing.secretKeyRingFile=

### Publishing the streams quickstart archetype artifact to maven ###
For the Streams archetype project, one cannot use gradle to upload to maven; instead the `mvn deploy` command needs to be called at the quickstart folder:

cd streams/quickstart
    mvn deploy

Please note for this to work you should create/update user maven settings (typically, `${USER_HOME}/.m2/settings.xml`) to assign the following variables

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                           https://maven.apache.org/xsd/settings-1.0.0.xsd">
    ...                           
    <servers>
       ...
       <server>
          <id>apache.snapshots.https</id>
          <username>${maven_username}</username>
          <password>${maven_password}</password>
       </server>
       <server>
          <id>apache.releases.https</id>
          <username>${maven_username}</username>
          <password>${maven_password}</password>
        </server>
        ...
     </servers>
     ...

### Installing the jars to the local Maven repository ###
    ./gradlew installAll

### Building the test jar ###
    ./gradlew testJar

### Determining how transitive dependencies are added ###
    ./gradlew core:dependencies --configuration runtime

### Determining if any dependencies could be updated ###
    ./gradlew dependencyUpdates

### Running code quality checks ###
There are two code quality analysis tools that we regularly run, spotbugs and checkstyle.

#### Checkstyle ####
Checkstyle enforces a consistent coding style in Kafka.
You can run checkstyle using:

./gradlew checkstyleMain checkstyleTest

The checkstyle warnings will be found in `reports/checkstyle/reports/main.html` and `reports/checkstyle/reports/test.html` files in the
subproject build directories. They are also are printed to the console. The build will fail if Checkstyle fails.

#### Spotbugs ####
Spotbugs uses static analysis to look for bugs in the code.
You can run spotbugs using:

./gradlew spotbugsMain spotbugsTest -x test

The spotbugs warnings will be found in `reports/spotbugs/main.html` and `reports/spotbugs/test.html` files in the subproject build
directories.  Use -PxmlSpotBugsReport=true to generate an XML report instead of an HTML one.

### Common build options ###

The following options should be set with a `-P` switch, for example `./gradlew -PmaxParallelForks=1 test`.

* `commitId`: sets the build commit ID as .git/HEAD might not be correct if there are local commits added for build purposes.
* `mavenUrl`: sets the URL of the maven deployment repository (`file://path/to/repo` can be used to point to a local repository).
* `maxParallelForks`: limits the maximum number of processes for each task.
* `showStandardStreams`: shows standard out and standard error of the test JVM(s) on the console.
* `skipSigning`: skips signing of artifacts.
* `testLoggingEvents`: unit test events to be logged, separated by comma. For example `./gradlew -PtestLoggingEvents=started,passed,skipped,failed test`.
* `xmlSpotBugsReport`: enable XML reports for spotBugs. This also disables HTML reports as only one can be enabled at a time.

### Running in Vagrant ###

See [vagrant/README.md](vagrant/README.md).

### Contribution ###

Apache Kafka is interested in building the community; we would welcome any thoughts or [patches](https://issues.apache.org/jira/browse/KAFKA). You can reach us [on the Apache mailing lists](http://kafka.apache.org/contact.html).

To contribute follow the instructions here:
 * http://kafka.apache.org/contributing.html

面向异构体系结构的高性能分布式数据处理技术与系统/Producer_Kafka

简介

发行版

贡献者

近期动态

面向异构体系结构的高性能分布式数据处理技术与系统/Producer_Kafka .gitee-modal { width: 500px !important; }

简介

发行版

贡献者

近期动态

搜索帮助

面向异构体系结构的高性能分布式数据处理技术与系统/Producer_Kafka