Netflix Prize Contest Ends, Down To the Wire 100
suraj.sun updates us on the Netflix Prize now that the competition has officially closed. We discussed the new leader with one day to go in the contest: The Ensemble, taking the lead from long-time leader BellKor's Pragmatic Chaos, the first contestant to submit an entry that broke the 10% barrier. In the contest's final day, BellKor re-took the lead with 20 minutes to go, then The Ensemble apparently pulled a Michael Phelps with 4 minutes to go, squeaking ahead by 0.01%. At least so the leaderboard claims — but those numbers are posted by the competing teams. The NY Times reports that an official winner will not be named until September — Netflix needs that much time to pore through the complex entries and read the code. Netflix contacted BellKor on Sunday to tell them the team remained in first place; The Ensemble has had no such notification.
I'll tell you how they did it (Score:4, Funny)
They realized that all movies starring Matthew McConaughey and Kate Hudson were actually the same movie. The compression on that alone was enough.
Re: (Score:2)
Re: (Score:2)
Hey now! Punch-Drunk Love is an amazing non-Adam Sandler Adam Sandler movie. But of course, it's the only one.
Re: (Score:2)
There's also "Reign Over Me"
Re: (Score:2)
Re: (Score:2, Funny)
Don't forget Adam Sandler movies.
Damn you, I was trying to.
I am still waiting... (Score:3, Insightful)
Re: (Score:3, Informative)
Re: (Score:2)
Except some of us also like movies that may not have had huge releases or made before we had internets...
If it wasn't for the internet I wouldn't have heard of Equilibrium (which was pulled back because of 9/11).
Right now I go through nzbmatrix and open the IMDB links and see what I really like and give it a shot.
Go, The Man who Knew Too Little, Equilibrium, Dark City, etc are all movies I've seen that I hadn't heard of because I was either in Middle School or they had limited releases.
Re: (Score:2)
And if it weren't for you, I wouldn't have heard of Equilibrium. Thanks!
Re: (Score:1)
NB: I don't claim to have a source for *my* assertion, only that the movie is clearly NOT low-budget, but only ran for a short time in my local theaters and has one of the most cheaply-made DVD releases I think I've ever seen. (My buddy's brother made liner notes for it with a chapter listing because there was NOTHING in the box with the disc.)
Also I
Re: (Score:2)
Re: (Score:1)
Considering it's such a terrible, terrible movie, is missing out on Equilibrium such a bad thing?
Re: (Score:1)
Upgrading to Silverlight 3 improved the on-demand image quality and fixed the annoying screen lock issue (at least for me on 64-bit Vista). Now if they could just get some more good movies on there...
Re: (Score:1)
Am I the only one...? (Score:2)
Am I the only one on slashdot who refuses to ever install silverlight?
Yes, I'm an anti MS fanboy, but after the things that happened during the OOXML ISO approval I just cannot support that company anymore. In any way whatsoever.
If you too (speaking to the general audience here) feel this way about MS, I hope that you do not support them by installing silverlight or in any other way really.
Re: (Score:1, Interesting)
No joke.
I was watching a show last night (Leverage, which Netflix suggested for me at a 4.6 stars and they were right, I love it.). 3-4 times per hour episode it would reset and tell me that it was adjusting things. Often, it never came back until I hit Refresh. I noticed that it would only buffer a small amount ahead (30 seconds) and then try to keep it there, calling an incredibly laggy site called something like controls.netflix.com or something like that. Once that happened, it had about a 25% chanc
Re:I am still waiting... (Score:4, Interesting)
I, for one, think the Silverlight player is phenomenal.
I have limited Internet options-- even though I'm living urban I am not close enough to a CO to get decent DSL speeds (the max Qwest offers is 1.5Mbps). Cable is not an option because my complex has a contract with the television provider who wired the buildings at construction, which is good for those who watch any TV since you get 50+ channels of cable television for free, but bad for Internet options.
Long story short, my Internet connection has a very high bit error rate percentage because I am getting my DSL over Qwest's line but from an ISP (AT&T via Covad) willing to boost the artificial limit of 1.5Mbps Qwest imposes to 3Mbps, at the expense of a quality signal. This results in being able to truly realize the faster speeds, but also in having a very burst-y connection.
I find the new Silverlight player to be far superior with its buffering saving the day, allowing me to watch Netflix streaming at maximum quality. The fact that the Silverlight player adjusts quality on the fly is outstanding as well-- when I first start streaming content it may look like shit at first but after a short time it is crystal clear, it realizing my connection can support the data load with a little buffering.
By contrast, with the old player, even before I had this error-ridden Internet connection, I would find myself initiating an instant streaming session only to find the stupid player would decide my connection was slow and give me piss poor video quality. I would have to click the "Back to Browsing" button and reinitiate the streaming several times sometimes in order for it to give it to me in high quality.
The new player also provides a great new feature when seeking through the content, where it will scroll past freezeframes of the content as you scroll forward or backward, which is perfect for skipping the intros for TV shows, for example.
I only wish it would "back buffer" a little because currently when I rewind a little bit, rather than replaying it from memory it rebuffers altogether, as if I hadn't just watched those few seconds prior.
Re:I am still waiting... (Score:4, Interesting)
I, for one, think that the Silverlight player is crap.
I have a dual-head machine with a very nice 1600x1200 IPS-panel NEC LCD as my primary monitor, and a nice (but far lesser) 24" 1920x1080 TN-panel Asus LCD as the secondary.
I want to pop up a Netflix show on the secondary monitor, full-screen, and continue to do stuff like read Slashdot on the other display. Silverlight has no problem putting good-quality video up, full-screen, on the second display -- but as soon as I click outside that window (ie, to browse Slashdot), it shrinks back down to windowed mode. My dual-head computer is therefore retarded into being effectively a single-head machine for the duration of the film, unless I either want to watch it in a little window or soak up a couple of cores worth of CPU power zooming in with Ctrl-+.
Allegedly, if I had Media Center on my computer, this could be worked around. But with Netflix + Silverlight, it cannot be accomplished. Of course, this situation works fine if I'm playing a DVD on my own computer -- it just doesn't work with Netflix's streaming service.
It is therefore retarded (in a very literal sense of the word).
I'd like also to note that Flash seems to have the same difficulty, and that its behavior is similarly inexcusable and retarded.
The best I can do, if I want to watch a film in my office and occasionally fuck around on the Web, is fire up my 4-year-old laptop and use that to browse with instead of my badass dualhead desktop rig. Which, also (and obviously) is retarded.
I've complained to Microsoft Silverlight developers directly about this, and the best they ever say is something like "You're right. It is retarded. Maybe we'll fix it some day. *harumph*" while months/years pass by and it's still an issue.
Re: (Score:2)
Re: (Score:3, Informative)
Re: (Score:2)
Re: (Score:2)
You read poorly.
I specified that it works fine with a DVD on my own computer. And that it works fine with Media Center, which is just another player (though rather more far-reaching than most).
If it's possible with some applications but not others, then it is plainly an application problem and not a Windows problem.
Re: (Score:1)
For all this bickering about whether Silverlight is any good, in my experience offering multiple viewers is the better way to go: each viewer will work better on different machines. If they have a choice of say 3 viewers, then one is bound to work.
Re: (Score:2)
Of course it is great, when all your know is Flash. Flash is one of the worst things that ever happened to internet video.
I hope browser-video-integration (like in Firefox 3.5, but with all formats, like H.264, XviD, MP3, AC3, Matroska containers, etc.) will change this for the better, and kill off Silverlight too.
Look at this (with Firefox 3.5), if you don't think it can kill Silverlight: http://people.mozilla.com/~prouget/demos/ [mozilla.com]
Re: (Score:3, Interesting)
Re: (Score:2)
Speak for yourself. I use the silverlight service at home at it works great for me. I output from my computer to my LCD TV and it looks good to me. As a bonus since it's not flashed based I can full screen it on my TD and still do stuff on my monitor.
these guys are all improving each other's code (Score:3, Interesting)
team a makes algorithm improvement b
team c takes algorithm improvement b and makes algorithm improvement b(+d)
team e takes algorithm improvement b(+d) and makes algorithm improvement b(+d)->f
the guy who squeaked out the extra 0.01% did that on top of someone else's code that eked out 0.05%, etc., ad nauseum
so how do you ascertain who won? all the teams won
they should take the final prize money and try to fractionate each incremental improvement in the algorithm and proportionally dole out the money that aways. anything else is unfair
Re:these guys are all improving each other's code (Score:5, Insightful)
There, fixed that for you.
Re:these guys are all improving each other's code (Score:5, Insightful)
According to the contest rules the winning algorithm will not be exclusive to Netflix but will be published to the public, so we all win.
Re:these guys are all improving each other's code (Score:5, Insightful)
because we all need to provide movie suggestions to our millions of users?
Because there are many, many organizations who can effectively use a high quality, freely-available automatic preference identification system. Some aspects of the winner are probably very specific to movies, but I'm sure most of the core, and the underlying ideas, are relevant to any sort of preference identification and clustering.
Re:these guys are all improving each other's code (Score:5, Insightful)
so how do you ascertain who won? all the teams won
No, they didn't, at all.
Any bozo can get 5% improvement. It's the last 5% that's tough. And, of that last 5%, the first 2.5% is cake, compared to the last 2.5%.
they should take the final prize money and try to fractionate each incremental improvement in the algorithm and proportionally dole out the money that aways. anything else is unfair
As someone who participated, but did not win, the first place team deserves the entire million (if not more). This was a race, and second place is the first loser.
Re:these guys are all improving each other's code (Score:4, Interesting)
Re: (Score:1)
Uh, that's a pretty disgustingly American viewpoint of the issue. Can't we all agree that if you didn't come in first, then you can still be a winner? This has been taught in schools for a long time now, it still hasn't been internalized?
No, this was a contest to see who could improve the algorithm the most. There can be only a single winner. If the contest was to improve the algorithm by x%, then there might be multiple winners. If you start calling everyone a "winner", you just cheapen the experience for the true winner.
Re: (Score:2)
Uh, that's a pretty disgustingly American viewpoint of the issue. Can't we all agree that if you didn't come in first, then you can still be a winner?
No, we can't; not unless you want to redefine the word "winner" in such a way that it loses all meaning.
That's not to say that the people who didn't win, don't get anything out of the experience. I learned a hell of a lot about recommender systems. But, I still did not "win". I'm okay with that.
This has been taught in schools for a long time now, it still hasn't been internalized?
And I hope it never will be.
Re: (Score:2)
It's this childish view that makes people think they should take up tasks they are not suited for, boldly proclaim "facts" they don't know anything about and in general behave unresponsibly.
Kids/People need to know that life isn't always fair, not everyone is a winner and not people just aren't equal (in an individual sense).
Telling them this stupid crap just makes them have unrealistic expectations and gives them a sense of entitlement that won't do them any good and quite probably will cause problems duri
Re: (Score:2)
Yeah, that particular part was probably a bit harsh. But the GGP was just trying to make a point (there can only be one winner), not trying to belittle the efforts of the others. It was a rhetorical hyperbole - not an actual "insult".
The GP may have tried to do the same, but I find his message far less clear. If I have misinterpreted his post, I apologise.
Re: (Score:2)
Yeah, that particular part was probably a bit harsh.
Harsh? I'm like the 300th loser. I'd love to be the first loser.
Re: (Score:2)
it's all relative man... To anyone not involved a statement like yours seems a bit harsh.
At least if they don't understand in what context you meant it. Which someone apparently did not.
Re: (Score:1)
Now, I agree that whether I win or lose (note there's only two categories there) is irrelevant when it comes to the experience. I can still enjoy a competition thoroughly and lose miserably.
There have to be winners and there have to b
Re: (Score:2)
Re: (Score:1)
If you ain't first, yer last!
Re: (Score:2)
The code is not shared unless they wanted it to be. You can't take the current leader's code and make an improvement unless the team gives you their code. This was not how the contest was run.
Re: (Score:2)
Exactly.
I do find it interesting that one of the winning teams (BellKor's Pragmatic Chaos) is a merger of three other top ten teams, and I wouldn't doubt it if you told me The Ensemble is a merged team as well (especially given its name). I certainly can't fault them for merging. Why risk winning nothing, when you can split a cool million with two other teams and have a much better chance of winning?
What would be funny is if the last two teams merged at the last minute ^_^ (Not feasible, I'm sure, but it
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Unfortunately, this is not the case. Each year, the winner of the Progress Prize WERE obliged to publish details of their algorithms to date ... and it was very clear that all the other competitors jumped on these details.
The contest was very clearly defined by these periods (sometimes of 11 months) where no real progress was made except by 1 or 2 top teams, and then 1 month right after publication of the latest ideas and algos, which caused everyone to catch up significantly by clearly copying and tweaking
Re: (Score:2)
> Unfortunately, this is not the case. Each year, the winner of the Progress Prize WERE obliged to ... and it was very clear that all the other competitors
> publish details of their algorithms to date
> jumped on these details.
Unfortunately? :) What's wrong with it? They win money, and NetFlix get them to disclose the details.
This is just great for humanity as a whole.
Re: (Score:3, Informative)
Pulled a Michael Phelps? (Score:2)
So The Ensemble ripped bong with 4 minutes to go which gave them the creativity to squeak ahead by %0.01?
Re:Pulled a Michael Phelps? (Score:5, Funny)
No, they lost to a German wearing a polyurethane suit and then declared they wouldn't race any more until the suits are banned.
Re: (Score:3, Informative)
See Wikipedia [wikipedia.org] and Netflix's own site [netflixprize.com] for details.
Re: (Score:2)
Yes, and I remember reading about it, but it took me quite some time to remember what particular contest they were talking about. It wouldn't have hurt to either give a link to the description page or put in half a sentence about the contest goals.
Re:What contest? (Score:5, Informative)
[2009-07-26]New Leader In Netflix Prize Race With One Day To Go [slashdot.org]
[2009-07-26]Netflix Prize May Have Been Achieved [slashdot.org]
[2007-11-27]Anonymity of Netflix Prize Dataset Broken [slashdot.org]
[2007-11-14]Close but no Cigar for Netflix Recommender System [slashdot.org]
[2006-10-02]Build a Better Netflix, Win a Million Dollars? [slashdot.org]
[2008-11-22]Interest Still High In the Netflix Algorithm Competition [slashdot.org]
[2009-10-09]Netflix Prize Competitor Already Beats Netflix [slashdot.org]
etc..
Re: (Score:2)
Well, even though I rad slashdot more or less every day, all those stories have eloped me for some reason or other (vacation? boring?). I remember only the first announcement of the contest back then.
Re: (Score:2)
Yeah I had no idea what the hell this was about either.
Re: (Score:2)
I see you can get Slashdot articles from October 2009. Could you tell us who your ISP is? I could use a connection that fast.
Re: (Score:2, Insightful)
!those numbers are posted by the competing teams (Score:4, Informative)
Go Bellkor!!! (Score:2)
Published scores irrelevant (Score:5, Informative)
The reason BellKor is still first is that the published scores are irrelevant. The scores that matter for the prize are based on an unpublished data set known only to Netflix (to prevent people submitting answers that are optimized for the challenge data and work poorly on everything else). On this secret data set, BellKor's algorithm apparently performs better than The Ensemble's.
Re: (Score:2)
Which leads me to wonder, what is (Score:2)
I don't use netflix, i'm a blockbuster guy cause we happen to have one close to our house. But What is 10% of zero? in all seriousness, what is their accuracy now? How is it determined?
Re: (Score:2)
# of suggested matches people actually say they like / Total # of suggested matches.
Was it really that hard to guess?
Re: (Score:2)
I don't guess when it comes to statistics. Because there are usually a ton of variables and long algorythms. It's not usually just x/y. Often, it's more like X(a+b-sqrt(z)(2))/Y(.8(z^2)).
That can only work for people who vote. It's a selection bias or something if I remember right from stats. For instance, I don't want to fill out an effing survey everytime I buy a product. I just want the thing in my hands as soon as possible so I can use it. When it's use is up, rentals get returned, disposables ge
Re: (Score:2)
I don't guess when it comes to statistics.
Fair enough.
Because there are usually a ton of variables and long algorythms. It's not usually just x/y. Often, it's more like X(a+b-sqrt(z)(2))/Y(.8(z^2)).
That can only work for people who vote. It's a selection bias or something if I remember right from stats.
It isn't necessarily a selection bias. It can create one, but it doesn't have to. If "people who vote" are representative of netflix customers in general, there is no selection bias. But you are right that selection bias is a concern.
For instance, I don't want to fill out an effing survey everytime I buy a product. I just want the thing in my hands as soon as possible so I can use it. When it's use is up, rentals get returned, disposables get recycled, and Blackbutte porter gets turned into Bud "Lite".
So 1. that means that people have to rent the selection suggested, two actually have watched it when they return it, 3rd vote on how well they liked it. Is it a 1-10 scale? or a yes no?
Come on this is /. I expected at least a little more detail.
You must be new here.
Re:Which leads me to wonder, what is (Score:4, Informative)
They used root square mean [wikipedia.org] for the competition.
Basically, the difference between the guess and the real answer for each vote is squared giving a value between 0 and 16 (as the biggest error is 4 when you guess 5 on a vote that is 1 or vice versa). This is summed up for each vote in the test and then divided by the number of votes in the test. Finally you take the root of that.
The winner score in the competition is around 0.855. Which is smaller than 0.9514*0.9 score. Where 0.9514 was the result scored by the netflix algorithm.
I hope that explains everything.
On Squaring (Score:2)
I always wondered why squaring is always used for distance measurements. It seems it would over-magnify the influence of non-matches and outliers compared to what the "real world" would want. I realize it makes the math easier, but having easier math and having the best answer may not be the same thing. I've asked math experts about this, but it seems it's an under-studied question.
Re: (Score:2)
Well. You pretty much answered the reason why, just by saying "measuring distance". Root Mean Square is the formula for measuring distance in an n-dimensional euclidean space.
But you are right to question if it is the best solution to some real world problems. Why should you represent a person's opions about movies, as a location in an n-dimensional euclidean space?
Re: (Score:1)
Perhaps I misunderstood the original message. I'm mainly thinking about regression formulas.
I'm just an aerospace engineer but.... (Score:2, Interesting)
Re: (Score:2)
Each year that the competition was running NetFlix awarded a progress prize to the best ranked team at the end of the year. Part of the requirements of winning this prize is publication of scientific papers describing key elements of their algorithms. BellKor's Yehuda Koren presented a paper at SIGKIDD in July describing the improvements they made to their algorithm to take advantage of predictable temporal dynamics of ratings. Check out the paper here [yahoo.com]
From an article about the paper:
Re: (Score:2)
Ideally it wouldn't be make any difference.
The idea behind the solution is that it can be applied to any set of data where people make a preference regardless of 'type'. That could be cars, politicians or slashdot comments.
"Based on the way you previously have moderated slashdot comments and the way other moderators have previously ranked similarly to you we think these comments will be +5 insightful to you."
And the winner is BellKor, in a quantum finish (Score:1)
Re: (Score:2)
Secret formula (Score:2)
If ((MovieReleaseDate > 2000) OR (MovieProducer = "Bay, Michael")) AND (CustomerAge > 20) Then .5
Score = Score *
If (MovieFABRating = "Nudity") AND (MovieLeadActor != "Cohen, Sasha Baron")
Score = Score * 10
Here's the Netflix thread where Koren leaks he won (Score:2)
"Thanks. In fact, this is a very happy day for us - our team is top contender for winning the Grand Prize, as we have a better Test score than The Ensemble. (Probably this is the first post revealing this in the forum smile)"
Also, Yehuda Koren is at Yahoo now, not AT&T.
All Incremental Voted Systems (Score:2)
---
Information Retrieval [feeddistiller.com] Feed @ Feed Distiller [feeddistiller.com]