Man vs. machine 1.0

Welcome to the half-time show of our Man-vs-Machine demo for our article Summarizer! We’ve really enjoyed the community interaction with this project, and we’re excited to continue to show off what this tool can do. We love combining research and machine learning to create unique and powerful products — and the Summarizer is no exception! 💪

‍

The story so far

If you haven’t been following along closely, let us catch you up: we wanted to show off the capabilities of our AI-powered Summarizer, so we matched it up against professional freelance copywriter and ESL instructor Alex Alderman. The contest is simple — over four rounds, both contestants will take a long-form article from a premium news site, and condense it down into a short summary that you can read in under a minute. For the first two rounds, the summaries were between 35 and 45 words long. Follow these links to see the project intro, Round 1, and Round 2 of the demo.

‍

Results of the first half

In the first round, it seems that people unanimously found the summary of the human more useful. Alex was able to include a little bit more context that gave him the edge over the algorithm.

In the second round, however, both contestants received votes, and you could see that the length restrictions of the summary (45 words) required both contestants to decide what information to include, and what to leave out. We think it’s interesting to examine these decisions:

Summarizer:
“Russian opposition leader Alexei Navalny detained on his return to Moscow. Police officers at Moscow’s Sheremetyevo airport surrounded and detained him. Navalny was on a wanted list for allegedly violating the terms of his suspended sentence from a 2014 embezzlement conviction.”

Alex Alderman:
“Russian opposition leader Alexei Navalny flew back to Russia after barely surviving his recent poisoning attempt, which was thought to be carried out by the Kremlin. Navalny was immediately arrested by Moscow police upon landing for allegedly violating the terms of his suspended sentence in 2014.”

You can find the original article here.

‍

You can see that the Summarizer went with just the straight facts: Here’s what happened — here’s where it happened — here’s the official reason given for why it happened. Whereas a human felt that “just the facts” wasn’t enough to tell the whole story, i.e. that more context was needed to understand what actually happened.

One user felt that this approach by the human participant was an advantage. We can clearly understand that reasoning! However, if we could play devil’s advocate for just a moment… 😈

What if what the human summarizer believes is important context is actually subjective and possibly exposes us to human bias? Another way to ask this is: when you’re reading a summary of the news, who do you want to analyze what it actually means — you or the preparer of the summary? 🤔 We’re not sure that there is a 100% clear cut answer, but it’s definitely an interesting question! What do you believe?

‍

‍

Robotic advantages

While the pros and cons of the human vs algorithmic approach can be debated, one thing is pretty clear: humans are slow! Our Summarizer produced its summary in under a second, and it took the professional about 10 minutes per article. That’s a efficiency increase of over 50,000 % 🤯 — and remember, our flesh-and-blood competitor works with texts professionally every single day.

Besides the obvious speed advantage (to be fair, the human didn’t stand a chance here []), the Summarizer also has: a lack of internal biases. Like we mentioned above, we can understand that there are obviously some advantages to the “human element,” especially in news stories that touch on moral issues. However, it’s important to remember that even the best news sources already include bias from the author of the article (even though premier news sites do their best to limit this). Using the Summarizer helps avoid a secondary layer of bias from a human creating a summary of events.

‍

If you only have a limited sized output of text, your bias as to what’s important may significantly impact what to include or not include. The algorithm removes this unwanted interference and gives you as neutral of a summary as possible.

How could the Summarizer be used?

We can imagine several different uses for this product, depending on feedback from the market (that includes you, so make sure to let us know what you think!). One use that comes to mind is a daily news aggregating service. You enter the sites you want to follow (e.g. the New York Times, or the Economist), and the Summarizer sends you a quick summary of all the top articles from the previous day. We can imagine a robust filtering system so that you’re getting exactly the information that is important (or just interesting) to you as an individual or as a company. We can’t think of a more efficient way to stay informedand knowledgeable about the topics or markets that you care about!

‍

Stay tuned 📺

The demo will continue next week as we move onto longer summaries.  We will have to wait and see if this increased length tips the advantage to one side or the other as both sides now will be able to include much more details of each article! We’re excited to see what new information this shows about the strengths and weaknesses of both contestants.

We also think it would be interesting to see if you can clearly spot which summaries come from a human and which come from our Summarizer.

More details to come, so make sure to follow us on Linked In if you want to participate!

‍