snorkel travis-ci

A lightweight platform for developing information extraction systems using data programming

v0.4.0

Documentation Build Status

Acknowledgements

Sponsored in part by DARPA as part of the SIMPLEX program under contract number N66001-15-C-4043.

Getting Started

Motivation

Snorkel is intended to be a lightweight but powerful framework for developing structured information extraction applications for domains in which large labeled training sets are not available or easy to obtain, using the data programming paradigm.

In the data programming approach to developing a machine learning system, the developer focuses on writing a set of labeling functions, which create a large but noisy training set. Snorkel then learns a generative model of this noise—learning, essentially, which labeling functions are more accurate than others—and uses this to train a discriminative classifier.

At a high level, the idea is that developers can focus on writing labeling functions—which are just (Python) functions that provide a label for some subset of data points—and not think about algorithms or features!

Snorkel is very much a work in progress, but some people have already begun developing applications with it, and initial feedback has been positive… let us know what you think, and how we can improve it, in the Issues section!

References

Installation / dependencies

Snorkel uses Python 2.7 and requires a few python packages which can be installed using pip:

pip install --requirement python-package-requirement.txt

Note that sudo can be prepended to install dependencies system wide if this is an option and the above does not work.

Finally, enable ipywidgets:

jupyter nbextension enable --py widgetsnbextension --sys-prefix

Note: Currently the Viewer is supported on the following versions: * jupyter: 4.1 * jupyter notebook: 4.2

By default (e.g. in the tutorials, etc.) we also use Stanford CoreNLP for pre-processing text; you will be prompted to install this when you run run.sh.

Alternatively, virtualenv can be used by starting with:

virtualenv -p python2.7 .virtualenv
source .virtualenv/bin/activate

Running

After installing (see below), just run:

./run.sh

Learning how to use Snorkel

There are currently two tutorials for Snorkel, an introductory tutorial and a more advanced disease tagging tutorial. The tutorials are available in the following directories:

tutorials/intro
tutorials/disease_tagger

Issues

We like issues as a place to put bugs, questions, feature requests, etc- don’t be shy! If submitting an issue about a bug, however, please provide a pointer to a notebook (and relevant data) to reproduce it.

Note: if you have an issue with the matplotlib install related to the module freetype, see this post; if you have an issue installing ipython, try upgrading setuptools

Jupyter Notebook Best Practices

Snorkel is built specifically with usage in Jupyter/IPython notebooks in mind; an incomplete set of best practices for the notebooks:

It’s usually most convenient to write most code in an external .py file, and load as a module that’s automatically reloaded; use:

%load_ext autoreload
%autoreload 2

A more convenient option is to add these lines to your IPython config file, in ~/.ipython/profile_default/ipython_config.py:

c.InteractiveShellApp.extensions = ['autoreload']     
c.InteractiveShellApp.exec_lines = ['%autoreload 2']

More badges…

Coverage Status

Code Climate

Test Coverage

Issue Count

Related Repositories

snorkel

snorkel

A lightweight platform for developing information extraction systems using data programming ...

snorkel

snorkel

Snorkel - Bootstrap your Data Science ...

snorkel

snorkel

Automated search for Ruby errors. ...

snorkel

snorkel

MHacks 8: Making Deep Learning Accessible ...

kettle-snorkel-13

kettle-snorkel-13

Kettle River Rainbow Trout Snorkel Count Analysis 2013 ...


Top Contributors

ajratner henryre stephenbach netj bhancock8 SenWu xiaoling pmlandwehr thodrek bryan-he alekdimi alldefector jason-fries lukehsiao mooz kuleshov aliskin

Releases

-   v0.4.alpha zip tar