MMLSpark 标签 - Gitee.com

v0.17

- LightGBM evaluation 3-4x faster!
- Spark Serving v2
- LightGBM training supports early stopping and regularization
- LIME on Spark significantly faster

- Both Microbatch and Continuous mode have sub-millisecond latency
- Supports fault tolerance
- Can reply from anywhere in the pipeline
- Fail fast modes for warning callers of bad JSON parsing
- Fully based on DataSource API v2

- 3-4x evaluation performance improvement
- Add early stopping capabilities
- Added L1 and L2 Regularization parameters
- Made network init more robust
- Fixed bug caused by empty partitions

- LIME Parallelization significantly faster for large datasets
- Tabular Lime now supported

- Added UnicodeNormalizer for working with complex text
- Recognize Text exposes parameters for its polling handlers

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of MMLSpark.

- Ilya Matiach, Markus Cozowicz, Scott Graham, Daniel Ciborowski, Jeremy Reynolds, Miguel Fierro, Robert Alexander, Tao Wu, Sudarshan Raghunathan, Anand Raman,Casey Hong,  Karthik Rajendran, Dalitso Banda, Manon Knoertzer, Lars Ahlfors, The Microsoft AI Development Acceleration Program, Cognitive Search Team, Azure Search Team

bba5c10

2019-04-23 15:06

下载

v0.16

- Added the `AzureSearchWriter` for integrating Spark with [Azure Search](https://azure.microsoft.com/en-us/services/search/)
- Added the [Smart Adaptive Recommender (SAR)](https://github.com/Azure/mmlspark/blob/master/docs/SAR.md) for better recommendations in SparkML
- Added [Named Entity Recognition Cognitive Service](https://azure.microsoft.com/en-us/services/cognitive-services/text-analytics/) on Spark
- Several new [LightGBM features](#LightGBM-on-Spark) (Multiclass Classification, Windows Support, Class Balancing, Custom Boosting, etc.)
- Added Ranking Train Validation Splitter for easy ranking experiments
- All Computer Vision Services can now send binary data or URLs to Cognitive Services

- Learn how to use the Azure Search writer to create a visual search system for The Metropolitan Museum of Art with: [AzureSearchIndex - Met Artworks.ipynb](https://github.com/Azure/mmlspark/blob/master/notebooks/samples/AzureSearchIndex%20-%20Met%20Artworks.ipynb)

- MMLSpark Image Schema now unified with Spark Core
    - Now supports Query pushdown and [Deep Learning Pipelines](https://github.com/databricks/spark-deep-learning)
- Bugfixes for Text Analytics services
- `PageSplitter` now propagates nulls
- HTTP on Spark now supports socket and read timeouts
- `HyperparamBuilder` python wrappers now return idiomatic python objects

- Added multiclass classification
- Added multiple types of boosting (Gradient Boosting Decision Tree, Random Forest, Dropout meet Multiple Additive Regression Trees, Gradient-based One-Side Sampling)
- Added windows OS support/bugfix
- LightGBM version bumped to `2.2.200`
- Added native support for categorical columns, either through Spark's StringIndexer, MMLSpark's ValueIndexer or list of indexes/slot names parameter
- `isUnbalance` parameter for unbalanced datasets
- Added boost from average parameter

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of MMLSpark.

- Ilya Matiach, Casey Hong, Daniel Ciborowski, Karthik Rajendran, Dalitso Banda, Manon Knoertzer, Sudarshan Raghunathan, Anand Raman,Markus Cozowicz, The Microsoft AI Development Acceleration Program, Cognitive Search Team, Azure Search Team

1d29394

2019-03-06 11:43

下载

v0.15

- Add the `TagImage` and `DescribeImage` services
- Add Ranking Cross Validator and Evaluator

- Learn hopw to use HTTP on Spark to work with arbitrary web apis in [HttpOnSpark - Working with Arbitrary Web APIs.ipynb](https://github.com/Azure/mmlspark/blob/master/notebooks/samples/HttpOnSpark%20-%20Working%20with%20Arbitrary%20Web%20APIs.ipynb)

- Fix issue with `raw2probabilityInPlace`
- Add weight column
- Add `getModel` API to `TrainClassifier` and `TrainRegressor`
- Improve robustness of getting executor cores

- Improve robustness of Gateway creation and management
- Imrpove Gateway documentation

- Updated to Spark 2.4.0
- LightGBM version update to 2.1.250

- Fix Flaky Tests
- Remove autogeneration of scalastyle
- Increase training dataset size in snow leopard example

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of MMLSpark.

- Ilya Matiach, Casey Hong, Karthik Rajendran, Daniel Ciborowski, Sebastien Thomas, Eli Barzilay,  Sudarshan Raghunathan, @flybywind, @wentongxin, @haal

fd1f662

2018-11-30 04:26

下载

v0.14

- The Cognitive Services on Spark: A simple and scalable integration between the Microsoft Cognitive Services and SparkML
	- Bing Image Search
	- Computer Vision: OCR, Recognize Text, Recognize Domain Specific Content,
		 Analyze Image, Generate Thumbnails
	- Text Analytics: Language Detector, Entity Detector, Key Phrase Extractor,
		Sentiment Detector, Named Entity Recognition
    - Face: Detect, Find Similar, Identify, Group, Verify
- Added distributed model interpretability with LIME on Spark
- **100x** lower latencies (\<1ms) with Spark Serving
- Expanded Spark Serving to cover the full HTTP protocol
- Added the `SuperpixelTransformer` for segmenting images
- Added a Fluent API, `mlTransform` and `mlFit`,  for composing pipelines more elegantly

- Chain together cognitive services to understand the feelings of your favorite celebrities with `CognitiveServices - Celebrity Quote Analysis.ipynb`
- Explore how you can use Bing Image Search and Distributed Model Interpretability to get an Object Detection system without labeling any data in `ModelInterpretation - Snow Leopard Detection.ipynb`
- See how to deploy *any* spark computation as a Web service on *any* Spark platform with the `SparkServing - Deploying a Classifier.ipynb` notebook

- More APIs for loading LightGBM Native Models
- LightGBM training checkpointing and continuation
- Added tweedie variance power to LightGBM
- Added early stopping to lightGBM
- Added feature importances to LightGBM
- Added a PMML exporter for LightGBM on Spark

- Added the `VectorizableParam` for creating column parameterizable inputs
- Added `handler` parameter added to HTTP services
- HTTP on Spark now propagates nulls robustly

- Updated to Spark 2.3.1
- LightGBM version update to 2.1.250

- Added Vagrantfile for easy windows developer setup
- Improved Image Reader fault tolerance
- Reorganized Examples into Topics
- Generalized Image Featurizer and other Image based code to handle Binary Files as well as Spark Images
- Added `ModelDownloader` R wrapper
- Added `getBestModel` and `getBestModelInfo` to `TuneHyperparameters`
- Expanded Binary File Reading APIs
- Added `Explode` and `Lambda` transformers
- Added `SparkBindings` trait for automating spark binding creation
- Added retries and timeouts to `ModelDownloader`
- Added `ResizeImageTransformer` to remove `ImageFeaturizer` dependence on OpenCV

We would like to acknowledge the developers and contributors, both internal and external who helped create this version of MMLSpark. (In alphabetical order)

- Abhiram Eswaran, Anand Raman, Ari Green, Arvind Krishnaa Jagannathan, Ben Brodsky, Casey Hong, Courtney Cochrane, Henrik Frystyk Nielsen, Ilya Matiach, Janhavi Suresh Mahajan, Jaya Susan Mathew, Karthik Rajendran, Mario Inchiosa, Minsoo Thigpen, Soundar Srinivasan, Sudarshan Raghunathan,  @terrytangyuan

7eed833

2018-09-23 13:50

下载

v0.13

New Functionality:

* Export trained LightGBM models for evaluation outside of Spark.

* LightGBM on Spark supports multiple cores per executor

* `CNTKModel` works with multi-input multi-output models of any CNTK datatype

* Added Minibatching and Flattening transformers for adding flexible batching logic to pipelines, deep networks, and web clients.

* Added `Benchmark` test API for tracking model performance across versions

* Added `PartitionConsolidator` function for aggregating streaming data onto one partition per executor (for use with connection/rate-limited HTTP services)

Updates and Improvements:

* Updated to Spark 2.3.0

* Added Databricks notebook tests to build system

* `CNTKModel` uses significantly less memory

* Simplified example notebooks

* Simplified APIs for MMLSpark Serving

* Simplified APIs for CNTK on Spark

* LightGBM stability improvements

* `ComputeModelStatistics` stability improvements

Acknowledgements:

We would like to acknowledge the external contributors who helped create this version of MMLSpark (in order of commit history)

* 严伟,  @terrytangyuan, @ywskycn, @dvanasseldonk, Jilong Liao, @chappers, @ekaterina-sereda-rf

ad20556

2018-06-29 02:25

下载

v0.12

New functionality:

* MMLSpark Serving: a RESTful computation engine built on Spark
  streaming.  See `docs/mmlspark-serving.md` for details.

* New LightGBM Binary Classification and Regression learners and
  infrastructure with a Python notebook for examples.

* MMLSpark Clients: a general-purpose, distributed, and fault tolerant
  HTTP Library usable from Spark, Pyspark, and SparklyR.  See
  `docs/http.md`.

* Add `MinibatchTransformer` and `FlattenBatch` to enable efficient,
  buffered, minibatch processing in Spark.

* Added Python wrappers and a notebook example for the
  `TuneHyperparameters` module, demonstrating parallel distributed
  hyperparameter tuning through randomized grid search.

* Add a `MultiNGram` transformer for efficiently computing variable
  length n-grams.

* Added DataType parameter for building models that are parameterized by
  Spark data types.

Updates:

* Update per-instance statistics module so it works for any Spark ML
  estimators.

* Update CNTK to version 2.4.

* Updated Spark to version v2.2.1 (the following release is likely to be
  based on Spark 2.3).

* Also updated SBT and JVM.

* Refactored readers directory into `io` directory

Improvements:

* Fix the Conda installation in our Docker image, resolving issues with
  importing `numpy`.

* Fix a regression in R wrappers with the latest SparklyR version.

* Additional bugfixes, stability, and notebook improvements.

0a28077

2018-04-17 10:42

下载

v0.11

43d2a8d

2018-01-31 03:47

下载

v0.10.9

212580e

2018-01-26 05:01

下载

v0.10

New functionality:

* We now provide initial support for training on a GPU VM, and an ARM
  template to deploy an HDI Cluster with an associated GPU machine.  See
  `docs/gpu-setup.md` for instructions on setting this up.

* New auto-generated R wrappers for estimators and transformers.  To
  import them into R, you can use devtools to import from the uploaded
  zip file.  Tests and sample notebooks to come.

* A new `RenameColumn` transformer for renaming columns within a
  pipeline.

New notebooks:

* Notebook 104: An experiment to demonstrate regression models to
  predict automobile prices.  This notebook demonstrates the use of
  `Pipeline` stages, `CleanMissingData`, and
  `ComputePerInstanceStatistics`.

* Notebook 105: Demonstrates `DataConversion` to make some columns Categorical.

* There us a 401 notebook in `notebooks/gpu` which demonstrates CNTK
  training when using a GPU VM.  (It is not shown with the rest of the
  notebooks yet.)

Updates:

* Updated to use CNTK 2.2.  Note that this version of CNTK depends on
  libpng12 and libjasper1 -- which are included in our docker images.
  (This should get resolved in the upcoming CNTK 2.3 release.)

Improvements:

* Local builds will always use a "0.0" version instead of a version
  based on the git repository.  This should simplify the build process
  for developers and avoid hard-to-resolve update issues.

* The `TextPreprocessor` transformer can be used to find and replace all
  key value pairs in an input map.

* Fixed a regression in the image reader where zip files with images no
  longer displayed the full path to the image inside a zip file.

* Additional minor bug and stability fixes.

36740ea

2017-11-15 13:59

下载

v0.9.9

a1b88ee

2017-08-31 02:22

下载

v0.9

c1a08f1

2017-10-13 06:19

下载

v0.8.9

87c6a33

2017-08-31 02:22

下载

v0.8

New functionality:

* We are now uploading MMLSpark as a "Azure/mmlspark" spark package.
  Use `--packages Azure:mmlspark:0.8` with the Spark command-line tools.

* Add a bi-directional LSTM medical entity extractor to the
  `ModelDownloader`, and new jupyter notebook for medical entity
  extraction using NLTK, PubMed Word embeddings, and the Bi-LSTM.

* Add `ImageSetAugmenter` for easy dataset augmentation within image
  processing pipelines.

Improvements:

* Optimize the performance of `CNTKModel`.  It now broadcasts a loaded
  model to workers and shares model weights between partitions on the
  same worker.  Minibatch padding (an internal workaround of a CNTK bug)
  is now no longer used, eliminating excess computations when there is a
  mismatch between the partition size and minibatch size.

* Bugfix: CNTKModel can work with models with unnamed outputs.

Docker image improvements:

* Environment variables are now part of the docker image (in addition to
  being set in bash).

* New docker images:
  - `microsoft/mmlspark:latest`: plain image, as always,
  - `microsoft/mmlspark:gpu`: GPU variant based on an `nvidia/cuda` image.
  - `microsoft/mmlspark:plus` and `microsoft/mmlspark:plus-gpu`: these
    images contain additional packages for internal use; they will
    probably be based on an older Conda version too in future releases.

Updates:

* The Conda environment now includes NLTK.

* Updated Java and SBT versions.

b61bf51

2017-09-02 10:30

下载

v0.7.91

48d65f9

2017-08-31 02:22

下载

v0.7.9

8b3f6fe

2017-08-31 02:22

下载

v0.7.1

06dae08

2017-08-31 05:46

下载

v0.7

5ea6488

2017-08-17 15:24

下载

v0.6

New functionality:

* Similar to Spark's `StringIndexer`, we have a `ValueIndexer` that can
  be used for indexing any type of values instead of only strings.  Not
  only can it index these values, we also provide a reverse mapping via
  `IndexToValue`, similar to Spark's `IndexToString` transform.

* A new "clean missing" data estimator, example:

val cmd = new CleanMissingData()
        .setInputCols(Array("some-column"))
        .setOutputCols(Array("some-column"))
        .setCleaningMode(CleanMissingData.customOpt)
        .setCustomValue(someCustomValue)
      val cmdModel = cmd.fit(dataset)
      val result = cmdModel.transform(dataset)

* New default featurization for date and timestamp spark types and our
  internal image type.  For featurization of date columns, convert
  column to double features: year, day of week, month, day of month.
  For featurization of timestamp columns, same as date and in addition:
  hour of day, minute of hour, second of minute.  For featurization of
  image columns, use image data converted to double with width and
  height info.

* Starting the docker image without an `ACCEPT_EULA` variable setting
  would throw an error.  Instead, we now start a tiny web server that
  shows the EULA and replaces itself with the Jupyter interface when you
  click the `AGREE` button.

Breaking changes:

* Renamed `ImageTransform` to `ImageTransformer`.

Notable bug fixes and other changes:

* Improved sample notebooks, and a new one: "303 - Transfer Learning by
  DNN Featurization - Airplane or Automobile".

* Fix serialization bugs in generated python `PipelineStage`s.

Acknowledgments

Thanks to Ali Zaidi for some notebook beautifications.

bb6a495

2017-07-11 01:46

下载

v0.5

70be8dd

2017-06-02 23:57

下载

MarineJ/MMLSpark .gitee-modal { width: 500px !important; }

搜索帮助

MarineJ/MMLSpark