A Billion Words: Because today’s language modeling standard should be higher

Posted by Dave Orr, Product Manager, and Ciprian Chelba, Research Scientist

Language is chock full of ambiguity, and it can turn up in surprising places. Many words are hard to tell apart without context: most Americans pronounce “ladder” and “latter” identically, for instance. Keyboard inputs on mobile devices have a similar problem, especially for IME keyboards. For example, the input patterns for “Yankees” and “takes” look very similar:

Photo credit: Kurt Partridge

But in this context — the previous two words, “New York” — “Yankees” is much more likely.

One key way computers use context is with language models. These are used for predictive keyboards, but also speech recognition, machine translation, spelling correction, query suggestions, and so on. Often those are specialized: word order for queries versus web pages can be very different. Either way, having an accurate language model with wide coverage drives the quality of all these applications.

Due to interactions between components, one thing that can be tricky when evaluating the quality of such complex systems is error attribution. Good engineering practice is to evaluate the quality of each module separately, including the language model. We believe that the field could benefit from a large, standard set with benchmarks for easy comparison and experiments with new modeling techniques.

To that end, we are releasing scripts that convert a set of public data into a language model consisting of over a billion words, with standardized training and test splits, described in an arXiv paper. Along with the scripts, we’re releasing the processed data in one convenient location, along with the training and test data. This will make it much easier for the research community to quickly reproduce results, and we hope will speed up progress on these tasks.

The benchmark scripts and data are freely available, and can be found here: http://www.statmt.org/lm-benchmark/

The field needs a new and better standard benchmark. Currently, researchers report from a set of their choice, and results are very hard to reproduce because of a lack of a standard in preprocessing. We hope that this will solve both those problems, and become the standard benchmark for language modeling experiments. As more researchers use the new benchmark, comparisons will be easier and more accurate, and progress will be faster.

For all the researchers out there, try out this model, run your experiments, and let us know how it goes — or publish, and we’ll enjoy finding your results at conferences and in journals.


Continua a leggere

Pubblicato in Senza categoria

Review III: ACI Toys "Total Rome!" 1/6 scale Roman Legionary Optio 12-inch action figure

continued from previous post

More on the ACI Toys “Total Rome!” 1/6 scale Roman Legionary / Optio 12-inch action figure.

Unlike the centurion, the uniform was not the distinguishing part of the optio’s uniform. The identifying part would be his helmet; this would have had plumes of horse hair or feathers on either side of his helmet that could be accompanied by a helmet crest.

Having shown pictures of ACI Toys “Total Rome!” 1/6 scale Roman Legionary figure with his full body shield (Scutum) and spear (hasta) previously, now it’s time to look at the Roman Option wielding his sword (gladius) for the close quarter battles.

Scroll down to see the rest of the pictures.
Click on them for bigger and better views.

As mentioned before (in my toy blog post HERE), this ACI Roman figure is too delicate and fragile, and came apart too easily – broken catch / bracket on the right and damaged hook on the left, sword handle / hilt came off but the blade remained in the scabbard, one of the ear flaps of the helmet fell off – resulting in me having a “limited edition battle damaged” 1/6 scale Roman Legionary Optio 12-inch action figure.
Close-up pictures of the Roman helmet with crest and red plumes, befitting of a Roman Optio
The crest was slid on the helmet via the Gallic style “slide-on” crest holder opening found on the top of the helmet

Continua a leggere

Pubblicato in Senza categoria

Remembering Oliver

Workshop member Jim Martin spoke at the funeral for Oliver Clubine on the weekend. For those who were unable to attend, here’s what Jim told a capacity crowd about our friend Oliver…



“When Sandra asked me to say a few words about Oliver I jumped at the chance, but not without trepidation. To eulogize a friend… well, that’s rather like qualifying for a car race. You get only once chance to nail it.

“Nevertheless, I’m honoured to honour my good friend of 30 years, but please bear with me if a few words become several.

“I’d like to start by offering my condolences to Sandra, David, Laura, Connor, Abby, Betty and Bill and to the rest of Oliver’s family and to his many friends. Family and friends are life’s one true currency. Oliver died a wealthy man.

“My happiest drives in southern Ontario have been the ones from my house to Pleasant Ridge Road, where in Oliver’s company I would drink in the dual elixir of trains and cars. That’s the funny thing. A lot of model railroaders like me are also car guys.

“When I first met Oliver 30 years ago at a model railroad-related social function, my first question was, “Are you the guy who races vintage cars?” It turned out I’d been watching him for years. More about the cars and trains later.

“First let’s go back to April 1937 here in Brantford and the birth of Oliver Bryan Clubine, the youngest son to proud parents Henry and Myrtle. Oliver also had an older brother Donald and a sister June, both of whom have also sadly passed on.


“Oliver attended Cainsville Public School – and later, Brantford Collegiate Institute. The Clubine name was already well known in the community. Oliver’s father had a family lumber business called Henry M. Clubine and Sons. That’s where Oliver went to work after graduating and also presumably when Henry added the letter “S” to his sign. Working in the family firm gave Oliver a head for business and a preference for self-employment.


“In 1962 Oliver made the best decision of his life. He married a young woman by the name of Sandra Telfer in Paris. A candle light Christmas service. She has been with him through the good and bad ever since. I never quite know what women see in our foibles but they enrich our lives and Sandra certainly enriched his. After the wedding they moved into a newly built home on Pleasant Ridge Road. It has remained their address for 51 years.

“In 1965 Oliver started to chart his own path in the business world, purchasing a beekeeping supply business from the family. He constructed a building in a new industrial subdivision in north Brantford and manufactured everything a commercial or hobby beekeeper would need… beeswax products, all wooden equipment and the many other addition items required to keep the bees and their keepers happy. At its peak the company had 13 employees and was the second largest company of its kind in Canada, shipping products from coast to coast. Each April and May he would import live bees from Alabama.

“David was born in August 1973 and very early on started receiving his homegrown business education, working beside Sandra and Oliver in their bee supply company.

“In 1976 Oliver got another bee in his bonnet, sold the business to retire and enjoy his hobbies, auto racing and model railroading. Well, we all know how that turned out! Some retirement!

“In 1977 he bought a small carburettor rebuilding business from Poughkeesie, NY, built a small garage beside the house and enjoyed, or expected to, a bit of country living. He called it Britain West Automotive Specialties. This grew to building race engines and rental of race cars and race car preparation.

“When David joined the business in 1994, the entire family worked tirelessly to build what is now known as Britain West Motorsport. Oliver was also a charter member of VARAC, the Vintage Automobile Racing Association of Canada and a past president of the Grand Valley Car Club. He was also the founder and president of the Mono Posto Register… that’s a fancy European word for single seater…. or Mondo Pasta as I used to call it when I was teasing Oliver about his physique.

“Oliver was head over heels about his racing, except for that one time in Trois Rivieres when he was heels over head. A racer never wants to look up at a hay bale, or down at the sky.

“That’s the car part. Now the trains. Sandra wanted me to tell you what model railroading meant to Oliver.


“Those of you unfamiliar with the hobby might ask what a guy like Oliver, immersed in the exciting life of automobile racing, would find in the arcane task of using miniature tools on miniature lumber, building miniature copies of real trains? Ask other racers…his son David, or Chris Creighton or Sam Posey who even wrote a book about it. Or anyone else whose dual interests include alloy wheels and flanged wheels.


“An escape perhaps? That’s what hobbies are about. A chance to concentrate so fully on something that it wipes your mind clear of the competitive pressures of life, health problems, the day’s disappointments and any number of other afflictions.

“But why model railroading? For many of us it’s a deeply rooted, visceral thing. We look at railroad tracks vanishing into the horizon, the way a poet might regard a tree, or an artist a sunset.

“For the hobby’s neophytes model railroading is just fine as a craft activity, but for the truly accomplished, it becomes an art form – and Oliver was an artist.

“Model railroading is a complex and richly layered hobby. Practised at its highest level it’s a 3D post card of times past… our own personal time machine. Its pursuit can teach a curious mind a huge set of skills – from the basic carpentry and wiring, to model building, airbrush painting, landscape painting, sculpting, computer skills, historical research, drafting, machining, photography, advanced electronics, writing, public speaking, teaching and mentoring, lighting, set dressing, and some others I’ve probably forgotten. The great thing though is you don’t have to learn all those skills… just the ones you want. That’s what makes it so richly personal and rewarding.


“Away from the technicalities of the race shop, Oliver found his pleasure in old school craftsmanship. With little more than an X-acto knife, sandpaper and glue, came award winning models that won him international recognition.

“But here too he couldn’t resist the business side. As a manufacturer Oliver produced craftsman kits under the Ridgehill Scale Models name. First came his quality line of wooden structure kits, followed by his exquisite models of Canadian style boxcars and cabooses, rendered in cast resin and photo etched metal. These are now prized and sought-after items.


“Oliver also owned and operated an exclusively S scale hobby shop out of his home. The uninitiated among you don’t have to know about all the different sizes of model trains… only that S scale is what we call a minority scale: Very few people do it. Because of that, it fosters strong friendships among its adherents. Oliver’s shop was good news for regional hobbyists… a chance to socialize as well as to see the goods close up.


“Twenty years ago Oliver was a founding member of the S Scale Workshop, a group of friends who remain together and who have built a widely exhibited, award winning portable model railroad. He most recently joined us at a local show in Copetown two months ago. It was like old times watching him become reinvigorated as he mingled and chatted among his friends. He even found the energy to join us for supper that evening… a special memory that will remain with us.


“Other memories? Perhaps not so special. Without getting into the “hows and the whys” there was an occurrence some years ago in which a beautifully crafted brass replica of a farm windmill found its way from the top of Oliver’s layout to the bottom of my left shoe… where it was flattened. Oliver was quite gracious at the time but took delight in never letting me forget. That monkey is now off my back, but unfortunately David also was a witness and will no doubt continue the family tradition of busting my hump.


“Oliver packed a lot into his life, his work, and his hobbies. He loved his family… he loved the family time spent in the Muskokas… with Sandra he raised a fine and talented son David who, like his dad, excels behind the wheel of a race car and in the art of model railroading. He has grandchildren Connor an Abby who he doted upon. Connor is showing the same interests and Abby the same tolerance.

“Oliver was a great ambassador for the hobby of model railroading and a dear friend. I’m happy that his mind remained fully engaged in his hobby, even during the days he tired so easily.


“Oliver was a gentleman who would never have insisted that you follow his path… only that you also do something meaningful to leave behind. He will be mightily missed, but the evidence of his talent and creativity remains with us – and that brings us comfort.


“Rest easy our very good and fine friend.”

- Jim Continua a leggere

Pubblicato in Senza categoria

Elbinator Atto Finale


Sabato 26 aprile si è conclusa la ormai celebre ELBINATOR, l’ultima tappa si è rivelata come da premesse la più dura della serie. 

Alla partenza l’atmosfera è carica di aspettative e i più scalpitano per andare, il tempo non è il massimo ma più che accettabile, prima tappa: Nivera – Madonna del Monte.


Partiamo già in quota ma c’è da salire ancora, la salita è dolce e poco impegnativa, passiamo attraverso i boschi di castagni per sbucare direttamente sul versante brullo della montagna e nonostante le nuvole basse gli scorci che si godono da questa altezza difficilmente si trovano da altre parti. 

Ancora freschi e carichi si arriva alla prima discesa che ci porterà diretti sulla celebre via Crucis per Madonna del monte. 

Il single track è divertente e tipico di queste parti, fondo secco e con sassi di granito. 
Non abbiamo il tempo di riprenderci che già si ricomincia a salire, direzione il Santuario di Madonna del Monte.


Arrivati al santuario breve sosta alla fonte per rifornimento acqua e via diretti al Serrone, un trail in quota da dove inizia una breve discesa a gradoni di granito che ci offre un assaggio di quella che sarà la parte finale del giro. 

La discesa porta fino all’imbocco dell’ultima estenuante salita. Qui le cose si fanno serie: questo tratto è il più duro, le pendenze sono considerevoli e pedalare diventa impossibile, non rimane che spingere, e anche così la fatica è tanta. Gradoni e sassi di granito  rallentano la marcia ma alla fine, stremati,  conquistiamo la vetta. Ora è tutta discesa senza sosta fino al mare, il tracciato è un vero e proprio tritabici, il fondo non offre un attimo di riposo per mente e corpo, serie di tornanti si susseguono a rettilinei su pietraie che mettono a dura prova mezzi e raider, ma vale la pena prendersi un minuto di pausa per ammirare lo spettacolo della valle che dalla montagna scende al mare. 

Il tempo è migliorato e scendiamo col sole che ci batte in fronte e dopo quella che ci sembra un eternità arriviamo in fondo al paese di Pomonte. 


I più sono distrutti ma soddisfatti, la parte dura del giro si è conclusa e dopo esserci riposati carichiamo le bici sui furgoni e ci trasferiamo a Marciana per il meritato pranzo. 


Giunti in paese, ci aspetta l’ottimo pranzo preparato dalla Paninoteca Monilli, mangiato e bevuto (anche troppo), ci aspetta l’ultima discesa ed è tutto divertimento!


 Con i ragazzi di Marciana a farci da guide affrontiamo una Townhill cittadina. 

Terminata la townhill marcianese raggiungiamo l’imbocco del sentiero che ci porterà sino alla Marina: il tracciato è quanto di meglio l’Elba possa offrire per il freeride, curato in ogni particolare la discesa scorre veloce fino a Marciana Marina.



L’alcool comincia a mietere vitteme, per fortuna in fondo al paese ci aspettano i mezzi per tornare a casa. 



Ormai sono le 18, si caricano le bici e si ripensa alla grande giornata passata in sella, alla fatica, tanta, ma anche la soddisfazione di aver concluso uno dei giri più duri dell’Elba.
Vi aspettiamo tutti quanti al prox evento elbano.

Continua a leggere

Pubblicato in Senza categoria

This is ACI Toys "Total Rome!" 1/6 scale Roman Legionary 12" figure (battle damaged) REVIEW

continued from previous post

The Roman legionary was a professional soldier of the Roman army after the Marian reforms. Legionaries had to be Roman citizens under the age of 45. On the march in unfriendly terrain, the legionary would be loaded down with armour commonly lorica hamata, lorica squamata, or 1st-3rd century lorica segmentata, shield (scutum), helmet (galea), two javelins (one heavy pilum and one light verutum), a short sword (gladius), a dagger (pugio), a pair of heavy sandals (caligae), a sarcina (marching pack), about fourteen days’ worth of food, a waterskin (bladder for posca), cooking equipment, two stakes (sudes murale) for the construction of palisades, and a shovel or wicker basket.

The Roman soldier underwent especially rigorous training; discipline was the base of the army’s success, and the soldiers were relentlessly and constantly trained with weapons and especially with drill — forced marches with full load and in tight formation were frequent. Discipline was important and infractions were heavily punished by the centurions. However, honours, rewards and promotions were frequently awarded to legionaries who distinguished themselves in battle or through exemplary service.

This is the recently released (badly produced) ACI 1/6 scale Roman legionary 12-inch figure. Everything was WRONG with this figure the moment it came out of the box – the catch and the bracket broke from the lorica segmentata armor, one of the ear flaps of the helmet fell off, the sword blade remained in the scabbard / sheath when the handle / hilt was pulled. The pilum (heavy javelin used by the Roman legionnaires) does not connect to the handle at all – it comes as two separate pieces – and keep falling off (would not stay fastened). I’ve had my share and bad experiences with 12-inch figures but this latest figure was really a PAIN to put together. The lorica segmentata armor just keep coming loose and refused to stay in place every time I posed the figure for shots / photography.
So I ended up with a “battle-damaged” ACI Toys “Total Rome!” 1/6 scale Roman Legionary Optio 12-inch action figure.  In this instance, I chose to display the Roman legionary wearing the galea (helmet) without the crest and holding the full body shield (scutum) in one hand with the Hasta (spear) in the other hand. Scroll down to see the pictures. Click on them for bigger and better views.

The legions of the Roman Republic and Empire wore fairly standardized dress and armor. The Galea was the soldier’s helmet. The focale (scarf) was worn by the Roman legionary to protect the neck from chafing caused by constant contact with the soldier’s armor. A tunic was worn under the armor (in this instance, it’s the Lorica segmentata or segmented armour). There was also the Pteruges: skirt of leather or fabric strips that is worn around the waist to protect the upper legs. Pteruges could be fitted with small metal studs and plates to provide additional protection.
The full body shield and spear, used by the Roman military with extreme efficiency and effectiveness.
The baldric was a belt worn over one shoulder that is typically used to carry a weapon (usually a sword) or other implement such as a bugle or drum. Braccae (trousers), was popular among Roman legionaries stationed in cooler climates to the north of southern Italy. Caligae, heavy-soled military shoes or sandals were worn by Roman legionary soldiers and auxiliaries throughout the history of the Roman Republic and Empire.
Scroll down to see close-up views / pictures of the ACI 1/6 scale Roman Gallic helmet (galea)
The Gauls were the peoples who most impacted the design of the Roman helmet hence the popular “Imperial Gallic” type helmets. In addition to this, it is commonly thought that the Gauls also introduced chainmail to the Romans.

Some of the helmets used by legionaries had a crest holder. The crests were usually made of plumes or horse hair. There is some evidence (Vegetius writings and some sculptures) that legionaries had their crests mounted longitudinally and centurions had them mounted transversely.

NEXT post: Check out this ACI Toys “Total Rome!” 1/6 scale Roman Legionary (battle damaged) 12-inch figure with the red crest and brandishing his sword (gladius).

Continua a leggere

Pubblicato in Senza categoria

Communities that Abide—The XIII Commandments

The manuscript for Communities that Abide is coming right along. The co-authors and I are working away, and it looks like I’ll be able to start editing sometime within the next two weeks. I am shooting to have the book out in time for the 3rd annual Age of Limits Conference, which will be held over the Memorial Day weekend at the Four Quarters Interfaith Sanctuary in Artemas, Pennsylvania. I’ll be running one workshop at the conference (on community formation) and moderating the discussion at another.

Read more »
Continua a leggere

Pubblicato in Senza categoria