Random search
Draws a random learning rate and batch size for every trial.
A neural network library in 7,728 lines of ANSI C, compiled to WebAssembly. Every network on this page is trained and run by that C code in your browser. Nothing is sent anywhere.
Pick a shape and watch a network carve up the plane. Tap the plot to add points. The library keeps training on the new data, one mini-batch at a time, with ann_train_step.
Cross-entropy loss per mini-batch (smoothed), log scale.
Draw with a finger or mouse. A 64-64-10 network classifies the 8×8 version of your drawing on every stroke. If it gets one wrong, tell it the right digit and it learns from your example.
Five copies of the same 2-16-16-2 network, with the same starting weights and the same mini-batches, trained by the library's five optimizers. Each uses the learning rate that worked best in a quick sweep on this data. Results change with the seed, so try a few.
Smoothed cross-entropy loss, log scale. Lower is better.
Tuning is the search for the settings that train a network well. Each trial trains a fresh 2-16-16-2 network on two spirals and scores it on 180 held-out points. Grid search tries every learning rate and batch size once. Random, Bayesian and TPE search get 50 trials each. All four run at the same time in background threads.
One trained network per cell, scored on held-out points. Darker means more accurate. The best settings form a diagonal: bigger batches want bigger learning rates.
Hover or tap a cell for details.
| Search | Best | First 95%+ | Trials 95%+ | Time |
|---|
All three searches find a 95% setting within a few trials, because roughly a fifth of the grid already scores that well. The difference is what they do next. Measured over 10 different seeds when this page was built: after the 10 random start-up trials, random search put 21% of its remaining trials at 95% or better, Bayesian search 67% and TPE 80%. A single run varies, so run it again to see the spread.
The library's Bayesian search tunes learning rate and batch size, which are exactly the two settings varied here.
Draws a random learning rate and batch size for every trial.
Fits a Gaussian process to the results so far and tries where improvement looks likeliest. The first 10 trials are random.
Splits past trials into good and bad, then samples where good ones are denser. The first 10 trials are random.
Bayesian search keeps a model of the landscape. After each trial it fits a Gaussian process to every score so far, then trains next wherever that model says improvement is likeliest. The four maps below are that model, rebuilt after the number of trials you choose. Drag the slider, or press Replay to watch it learn.
Grid search's result, for comparison. The model never sees this.
Predicted accuracy. Blue where trials were good, fading away from the trials it has seen.
Standard deviation of the prediction. Shaded areas are the ones it knows least about.
Expected improvement: high where a trial could beat the best score so far, by being either promising or unexplored.
The diamonds are the trials so far. The ringed one is the trial the search actually ran next. The library picks it by scoring 100 random candidates, so it is the best of those 100, not the brightest spot on the map. The model is rebuilt here from the logged learning rate and batch size, with the batch size placed at its list position, so it is the same model up to a small rounding of the batch axis.
The pre-trained model that ships in the repo (mnist-fashion.nnb, 784-128-10), loaded through ann_load_network_binary. It scores 86.98% on the full 10,000-image test set, the figure in the README, and this build reproduces it exactly.
Fill in a solid silhouette: a sneaker, a bag, trousers, a dress. The model learned from photographs, so crude sketches often fool it.
The build uses plain clang --target=wasm32 with no Emscripten. A C shim of 1108 lines supplies malloc, printf and an in-memory file system, so ann_load_network_binary reads its model with no changes.
libann is written by Mark Seminatore and released under the MIT license: github.com/mseminatore/ann. The digit model was trained with the library itself, on scikit-learn's 8×8 digits with added shifts, rotations and stroke-width changes. It scores about 98% on 360 held-out digits.