Recsperts - Recommender Systems Experts

Note: This transcript has been generated automatically using OpenAI's whisper and may contain inaccuracies or errors. We recommend listening to the audio for a better understanding of the content. Please feel free to reach out if you spot any corrections that need to be made. Thank you for your understanding.

The reason that these early recommender systems were good was partly because they were wrong a lot.
We are a machine learning or a computation problem in part, but we're bigger than that.
What we're really about is this highly constrained, highly contextualized problem that applies historic data to creating appropriate, valuable user experiences around situations where you're dealing with product choice.
Sometimes the best way to make things useful is not the recommendation itself, but the information put around it.
Whether a recommender system worked or didn't work might depend on what you set the user up for as much as what you did in the recommender system.
You need people with a vision that gets below that quarter and that starts saying, OK, this is just the right thing to be known for.
This is going to be something where we need to measure a year later.
It has always been a conference at the intersection between research and practice.
It has always had substantial industry participation, some of them industry researchers and some of them advanced industry practitioners or even people who are coming just to be brought up to speed in the field.
I think if the field is going to remain relevant, it's going to remain relevant by remaining multi-approach and multidisciplinary.
Hello and welcome to this new episode of Recsperts - Recommender Systems Experts.
For today's episode, I have invited a guest that I've been looking a long time forward meeting and featuring in this podcast, a person that I guess is well known to everybody in the community, a long time luminary of the field, and a person probably most well known for his work on collaborative filtering algorithms and also one of our hosts for this year's RecSys in 2026.
By this time, you might have already an idea about whom we are talking about.
And for those of you who are into reading lots of papers and also do know the fundamentals of the field pretty well, I will just start with the paper that is the most cited paper this person has been co-authoring, which is item-based collaborative filtering recommendation algorithms.
And by this time, you might already know whom I have invited for today's episode.
It's a very great pleasure welcoming to the show Professor Joseph Konstan.
Hey, welcome to the show.
It's a pleasure to be here.
Yeah, thanks for joining me.
There is a bit of also personal history that I will come back to later because you, along with Michael Ekstrand, who has been in one of the previous episodes, were really my first touch points or my first touch point with recommender systems almost a decade ago.
And I'm still very, very grateful for that.
Also for our listeners, like the usual thing we start with in this episode is a short, quick introduction of my guest.
And even making this short and quick is kind of a challenge because you have really worked on a lot of things and achieved a lot of things and are well known for good research in this community and contributions to the community.
Nevertheless, I guess we should do you the honor and introduce you properly before handing over to you and continuing with that.
So Professor Joseph A. Konstan is a Distinguished University Teaching Professor at the University of Minnesota.
He is also Associate Dean for Research for the College of Science and Engineering.
In 2009, he has also been named a Distinguished McKnight University Professor and joined as a professor at the University of Minnesota in 1993.
Apart from that, Professor Joseph Konstan has also been receiving the ACM and IEEE fellowship.
He has numerous publications at conferences such as, of course, RecSys SIGIR, UMAP, CHI, and many others.
He is probably best known for his work on collaborative filtering recommenders for the GroupLens project and also today for the POPROX project that he has been doing along with other fellow scientists.
Joe Konstan received his PhD in Computer Science in 1993 from the University of California at Berkeley.
And he has been the first General Chair for the RecSys Conference, which took place for the first time in 2007.
And he is the General Co-Chair for this year's conference, the 20th RecSys that's gonna be taking place in Minneapolis.
So basically back to the roots.
So who would be the best to talk a bit more about his experience, his passion in recommender systems than yourself?
So Joe, please enlighten us.
What makes you so fascinated about recommender systems and what did I possibly miss about introducing you or what is an important point in your career so far?
I think an important point in all of this is that I came into recommender systems as a very fortunate accident.
My training was in the technology of human-computer interaction.
So I was one of those people who built user interface toolkits and tried to advance the ability to program interfaces.
And when I came to the University of Minnesota, that's what I started working on.
And before very long, a colleague of mine, the late John Riedel, came to me having returned from the CSCW Conference in 1994 with great excitement.
And he came back with great excitement because the paper on group lens that he and Paul Resnick then at MIT published then, which was what we all in the field think of as one of the three original papers on automated collaborative filtering, along with the work that was done on the Bellcore video recommender and the Ringo Homer later Firefly system out of MIT.
He came back with great excitement about, hey, people really think this is important.
It could go somewhere.
And he came to me because he was a systems person and I was a human centered computing person.
And he thought this would be a good possibility to work together.
And he was right.
We spent the next summer, the summer of 1995, hacking together with Paul and a bunch of students, a more ambitious group lens system that we were able to launch and get about 200 different humans using as participants over Usenet news.
The experiment showed it worked, that we could actually on the one hand, do a pretty good job predicting which articles you would like in a personalized way.
And on the other hand, that the people who used our interface changed their behavior.
This is the piece that I think always comes back to where my passion is.
I don't care about prediction at all.
I care about changing people's behavior.
And if we don't add value, it's not useful.
And with that, that led to all sorts of things.
It led to our forming the company Net Perceptions, which was not the first, but one of the early companies that had a very successful run in this field with customers you may have heard of, places like Amazon, which seems to have done pretty well for itself.
It also was what put us in the position where we had the software infrastructure to take on creating movie lens when the folks at Digital Equipment Corporation's research center in Palo Alto were shutting that system that they had called Each Movie Down.
And they said, gee, we hate to have nothing like this.
Does anybody wanna run one of these?
And after much negotiation, we got to use their data, but had to re-invite people to join on their own.
And we've been running movie lens ever since.
And it's a system that will soon hit 30 years of running people through recommender system studies.
And it continues today with the work that we do.
But if there's sort of one unifying theme in all of this, and this is probably how I view technology as a whole, not just recommender systems, it's that I'm always looking at, well, what can it do for you?
This was not my example.
It was somebody else who first presented it.
And I no longer remember who it was who said, I can create a perfect or nearly perfect recommender system in an American supermarket by printing cards that say, buy bananas and bread and putting them on every shopping cart, and then counting the number of people that buy bananas and bread on the way out.
And you discover that, holy mackerel, my recommendations are really accurate.
They're also completely useless because those people were buying bananas and bread without the recommendations.
And so all of the work that I've been doing for nearly three decades has really focused very closely on the question, can we deliver recommendations that tell you something you didn't know on your own already, or change your behavior, or give you options you might not have had, and in some cases simply reduce the effort that it takes for you to get to the conclusions that you wanted to come to.
And that's why you find this field so exciting, because there really are a lot of cases where this continual growth of information and products and options that are out there, where we can add real value by helping people navigate through that and find the things that are valuable to them with less effort and hopefully higher success.
That sounds great.
And a common theme that I keep hearing when I look at the talks that you gave or also at different venues alongside RecSys, for example, workshops or tutorials, I guess it was a specific workshop a couple of years ago when you put that slide up there and saying that this conference, this actual conference was annoying you a bit, because it seemed like everything is about squeezing out the last relative percentage from some gains in NDCG or MRR.
And that somewhat resonated with me when in preparation for this interview, I was also watching a talk that you gave in 2019, where you mentioned, and I quote here, I sort of hate and love machine learning.
Why is that the case?
So the reason I made that statement was because I really am torn.
There are times in which I think all of this work we've done with machine learning and recommender systems hasn't actually made things better.
And by certain measurements, if you go back and you look at, was the early recommender system technology, the user-user collaborative filtering, have we gotten a whole lot better now that we use so much more data and so much more structure.
And by certain measures, we've gotten a lot better.
By the measure of, are we introducing people to things that are surprising to them and delightful that they wouldn't have explored, often the answer is no.
That what we've done is we've squeezed so much data together that we've done a very good job finding what's obvious.
And the reason that these early recommender systems were good was partly because they were wrong a lot.
It was the fact that a recommender system that took 10 people like you and found things that some of them liked, and if two of them liked it and thought you might like it, often was wrong, but when it was right, it was also often surprising.
And we've moved to systems that are wrong a lot less, but also useless a lot more.
And so the challenge I have with machine learning is when you use it right for the right problem, it's amazingly powerful and it does amazing things, but it's very easy to ask it the wrong question and then it optimizes for the wrong thing.
In the domain I've been working in recently in news, if you train a machine learning system to optimize for click-through on news articles, you are very likely to have a system that's very good at recommending clickbait because clickbait gets people to click through it.
Oh, you've told me nothing about the story, but you've given me an attractive heading like, you wouldn't believe what this famous model thinks about some sports star that you haven't heard of.
And I click on it and 12 pages scrolling down later, I find out that the model had never heard of that person and I just wasted my time.
Well, I wanna be able to say, don't show me that stuff, but if I'm optimizing for click-through, I'm gonna keep getting it.
And what I want is to optimize for value.
And machine learning doesn't do that by default.
You need to ask the right questions.
Now, can it do it?
Yeah, we've done some studies where we've shown that if you use the right reinforcement learning techniques and you measure your rewards far enough down the road so that you're really finding out about long-term engagement, long-term retention of customers or users, you can get some valuable stuff out of the system.
But that's why I'm torn, because I think you can look at it both ways.
And some of the time people do some really amazing things, but too often in the rush to publish papers, it's not that I'm against getting that tiny increment more, I'm against getting that tiny increment more when it's basically fake, when it doesn't change the user experience at all.
I think our field has moved by pulling together hundreds of tiny increments.
And that's a good thing in research, but you wanna make sure that your tiny increment isn't just you measured against a baseline that didn't do something and so you added that something back in, but real systems were already just as good as what you could produce.
I think this has been a debate and especially conducted among top researchers in the field for quite some time.
Like how do we incentivize especially researchers properly so that they care more about how they do research in the field of RecSys really influences or changes user experience to the positive apart from like changing tiny metrics.
This might be some incentive problem, but I guess taking this more broadly and you have elaborated on this, that you have had a lot of industry exposure that also in industry, there might be incentives to jump into solutions that are not sustainable, but both at the same time, we see that there's industry research that for example, leverages reinforcement learning based algorithms to take care of more longer term effects that are positive, both for the user and for the business.
So if we see the effect, but similar or different patterns in industry and academia that lead to this effect, what would it be that we would need to do differently in academia mainly, which you can I guess talk about, but maybe also in industry to change that?
I don't think you have a podcast that's long enough to solve this problem, but let me take on two parts of this.
I think the academic piece in some sense is the question that we have been asking across academic research generally of how do we get people to do important impactful work rather than easy to complete and easy to publish work.
And there are a lot of efforts often in the computing field, sometimes beyond computing to try to do this.
And so far they've almost all failed.
One of the simple ones and it sounds simple, it's not really easy to implement would be to say, can we put a cap on the length of people's resumes?
Right?
You can have on your resume one paper for every two years of your career.
Well, then you don't wanna publish 16 papers this year, you wanna publish one paper that's better than the papers that are currently on your resume.
And if you could just do one big important thing that would work.
Do I think that's an easy solution?
No, not at all.
Partly because you can't get people to agree, but partly our model is tied up with students and trainees and what kind of student wants to come in and work with you and find out, yeah, you're only gonna be here five years, we're probably not gonna do a paper together cause I don't think it's gonna be important enough and it's not gonna show up on my resume.
We don't have the answers to how to do this, but getting people to reduce the quantity, we thought at one point that maybe if you had to pay to publish stuff, it would do that.
And that hasn't seemed to work, but getting people to reduce quantity will naturally get them to want to do work that is more impactful and stands out as being better and more rigorous.
But I don't have the answer there.
In industry, it's a different problem.
There are very few companies where people at the level that are deploying recommender systems to solve problems have the authority to think long-term.
And in many companies, people will tell you nobody has that authority, that the chief executive officer is responsible for this quarter's results.
And they may not be here in five years and we don't see a whole lot of incentives structured around, well, it's okay if you have a bad year or two as long as we have a good decade.
This is not a new problem, by the way.
The first story I remember hearing about effectively the misuse of recommender systems, I think it was one of the music sites.
It might've been CD Now.
It may have been one of their competitors in the early to mid 90s that had gone through and not with us, but with external consultants had done a personalized set of recommendations that they would send out by email.
And when they sent it out, sales went up and they thought this was great.
And so they were sending it out.
I don't remember how often, maybe it was quarterly, maybe it was monthly.
And then they start getting towards the end of the quarter and they need extra money because at that time every internet business needed higher sales.
So they sent it out an extra time and sales went up a little bit.
And then the next quarter they started sending it more and more often and then a whole bunch of people unsubscribed.
Because if your short-term view is, hey, wait, what can I squeeze out of my customer this quarter?
And you're not looking at, did I just poison the relationship I have with that person and make them feel used?
You're missing the point.
And I have this discussion with people with companies all the time.
My frustration with Amazon's recommendations is that they almost always recommend stuff that I've either already bought, already shopped for, or is an obvious relationship to that.
And I believe the long-term benefit to them and to me would be introducing me to entirely new sets of products that I don't even know they carry, but that they discover I might like.
They have some of the best people in the business in this technology.
What's stopping them is not the technological ability to do recommendations that balance novelty, that balance unfamiliarness and other things.
It's business leadership making the decision that the recommendation is not really about introducing people to more stuff.
It's about getting people back to the store.
How do I find something you're gonna click on, even if it's, oh yeah, I need soap again?
Because once you're in the store and you're shopping, you'll buy stuff.
And that's the important measure that they care about.
And this has been true, by the way, before we even had Amazon.
The coupons that are printed for many grocery stores and convenience store chains and things like that, the key feature on the coupon was not the product.
It was the expiration date.
When do we have to get you back in the store so that you feel you'll save money?
When in fact, by saving money, you leave, having spent $100 that you had no idea were things that you needed.
But once you're in the store, you see more stuff.
As long as the customer is the product and not the partner, I don't know that there is a simple solution to this problem.
I think that we're gonna find that solution in different cultures in some cases, different business cultures, private companies that are not publicly owned where the people in charge of them really do believe in that we're building this for my grandchildren, not for the stockholders next quarter.
I think there's different places where this can come from, but it's not an easy solution and it's not primarily a technological one.
So when you say that this is more a matter of myopic prioritization by business leader, less than one of technical capability, you mentioned this example of Amazon and I would fully agree there.
There are very capable researchers and engineers working for this company, but nevertheless, we as an end user keep seeing the same problems and it feels sometimes somewhat get caught by people.
So it's almost every second time I have to explain to someone who is not involved with the field what I do, then like this example is being brought up.
Like last week I bought this washing machine or whatever from Amazon and now Amazon keeps recommending me more washing machines.
I don't need them.
So why do you do that?
And I'm sometimes as clueless as that person himself.
So isn't there at least a bit of judgment or responsibility also attributable to those researchers and engineers because they are the ones who actually need to better explain to business leaders the long-term effects of like, let's say having more diversified recommendations or having more longer term rewards.
And I mean, we are not talking about years here, but maybe like take into account the actions or the rewards that offers a down the stream within month to explain these advantages better because they are reference examples.
So I'm always thinking about this example, I guess it was from YouTube and the longer term effects of diversified recommendations.
So there are these examples from other industries.
It doesn't always mean that YouTube streaming translates directly into something at Amazon, but like at least if there's a reference, we could take that maybe as an example.
So is it really like this myopic sense or prioritization of business leadership or also a call for better explaining the dynamics or what's possible from the like practitioner side?
I think you raise a really important question.
And I think the truth is we don't know what's going on within those company internal discussions.
I expect that everyone who's an expert in this field knows how to make that argument.
And many of them do make that argument.
I think at least some of them probably go a step further and they're given the opportunity to run a small experiment to try to demonstrate that that argument is successful.
Whether they're given the length of time it takes to do that is an interesting question, right?
I've had the same experience that I was shopping for something on Amazon, but I wasn't going to buy it there because my university was going to buy it and they were gonna purchase it through their account.
Are you talking about that treadmill?
Right, I did that with the treadmill.
That was exactly one of them.
I've actually done this with five or six different significant products.
I'm sure there was a way I could have figured out how to tell Amazon, I bought this.
I even bought it at Amazon, but not through this account, so please stop.
But there was no data in the system that would have revealed that I'd finished this.
And more to the point, even if there were, I don't know if the short-term results would have made a difference.
There are customers out there that do a lot of post-shopping confirmation or checking for regret.
I have colleagues who, you know, they'll buy a car and they're still looking at the car ads for a period of time afterwards to make sure that they really did get a good deal.
I try not to do that.
I'm sure there are times when I slip into that behavior too, but it might be that this is a case where what we know is good for the long-term, when you try to test it in a shorter-term scenario, doesn't actually produce more dollars yet.
And so some of this might be, you need people with a vision that gets below that quarter and that starts saying, okay, this is just the right thing to be known for.
This is gonna be something where we need to measure a year later, how many of our different product categories did this person buy in and did we improve that number?
There may be a lot of things there.
It's also the case that some of the things that we think are good might not make money, right?
To me, it's an obvious good that if I know Amazon has a line of products I've never bought from them and I therefore might buy something from, that they should be better off because now I can use them for more stuff.
But it might be wrong.
Just because it's obvious doesn't mean it's correct.
It might be that if I start thinking of them as too diffuse, maybe they stop being the first place I think for the things that I'm going to.
And maybe when I start realizing, oh, they've got tools, they've got this, they've got that, they feel like Walmart, I stopped going to them for books because I now think of them as having a Walmart-like collection of books rather than a Library of Alexandria-like collection of books.
And so it's also possible that some of our intuitions about this won't bear out when we test them.
That's why I think testing on people is so important because not just testing on people, but testing on people over the long run, longitudinal testing, is sometimes the only way we can understand not just our algorithms, but human behavior.
Spot on on that.
And this will also bring us to talking about Pop Rocks later in this episode.
Before we dive into that, we have just already performed quite of a deep dive on what is not working that well, but I guess also a lot of the things are working quite well.
A lot of this centers around what you coin as usefulness.
How would you describe what really renders recommendations as useful apart from everything that we mentioned that might add to them becoming useless?
So I think you have to come back to understand why a person is interacting with the system that the recommender is embedded in in the first place.
And when you understand what they're there for, you can understand things like, is the most useful thing you can do to return the obvious thing, but you're making it faster for me to find it?
Or is it the most useful thing to show me a set of choices that are very close alternatives because I'm very clear what I want.
I just don't know which one, or do I want a diversity of content because I'm trying to make a broader decision?
Google is effectively a recommender system bundled into a search engine.
If I go to Google and I type RECSPERTSs, I want it to return the pointer to your webpage.
I don't want it to say, oh, you've been to that page multiple times already.
Let me see what else might be interesting and unknown because I went to that site with a very specific purpose.
Similarly, if I go to an Amazon or a drug store chain and I come in and I say, oh, what I'm looking for is Advil.
Maybe I want, well, I pretty definitely want Advil products and I might be interested in different sizes.
Maybe I'd be interested in some of the generic and other brands of the same drug.
Maybe I'd even be interested in other drugs that relieve pain or reduce swelling.
But this probably isn't the right time to say, did you know that we also carry heavy metal albums in case you're looking for something to listen to through your headache?
This is not the right context.
I'm searching pretty narrowly, so recommend pretty narrowly.
That goes very different as you get further out and more broad.
As you operationalize this as a researcher, the point I try to remind people is that the historic information retrieval driven approach of let's leave one out or leave K out and see if we retrieve them.
That's only useful when there's an assumption that the user did the right thing in the first place.
And that's what they want.
They're not looking for anything new.
They're just trying to retrieve something they already knew but had forgotten, which might work when you're doing search it's completely useless as a metric if what you're trying to do is enable discovery.
And so if you're in a context where what the user is looking to do is discover products they didn't know about and you come back and say, look, we do a really good job with leave one out.
Then what you're telling me is, so what your system does is add exactly zero value to what I had figured out how to do before your system existed.
And that's where I don't think things become particularly useful.
And so I don't think you can just measure usefulness on a particular metric without taking it in the context of the problem and the system.
And sometimes the best way to make things useful is not the recommendation itself but the information put around it.
Are we displaying stars?
Are we displaying reviews?
Are we putting up a comparison table among different products that tells you the differences in features or whatever?
But it starts with a deep understanding of the problem you're trying to solve.
I should say, this is one of the reasons I love the fact that this community has been rallied around challenges where every year at Rexxis, we have a challenge.
This is what happened with Netflix and all sorts of different things where people come in and are told, hey, solve this problem.
Now, they don't all have the right metric.
They don't all turn out to be very user centered but it's a much better way of moving towards being user centered and problem centered when a stakeholder who really cares about the output is engaged to try to figure out, well, what's the measure that would really matter for us?
Yeah.
Yeah, I guess this is a good way of putting it.
And especially, I like that point about one solution does not necessarily fit them all, especially if you think about grocery recommendations and you brought up that example about the bananas and the bread, but sometimes users want to explore something.
So really finding out maybe based on the contextual signals you're possibly getting, like is the user more in exploration discovery mode or are they like in a recurring mode?
There was this paper also by Dieser Research, I guess two or three years ago on really finding out this repetitive versus new consumption behaviors and trading them off against each other or really finding out like which is the situation the user is in, what is their current intent?
Yeah, and different users have a different balance between those, but all users seem to have some sort of a balance in different contexts.
I mean, there are some people who were thrilled when you found grocery stores that would let you set an order and it would get delivered the same order every week because that's what they want.
There are other people who would think, I don't know why I would do shopping at all if I can't walk through the aisles and see what looks good and what smells good.
And you want to be able to address all of them and sometimes people can tell you if you ask, sometimes they don't even know and you just have to look for the cues in what they're doing.
And circling back to what you said about performing user studies and distilling what users might perceive as useful and whatnot, your name has been associated, I guess, the most with the Grouplands research group and especially with Movielands and the Movielands data sets being, I guess, some of the most widely used data sets in RecSys research.
If you look on this long history of the Grouplands group, which has been doing more than just running Movielands, what do you think has been going well and what has been going not so well or which objectives have you been able to reach with this project and yeah, how does this also tie possibly into the new project that is called Pop Rocks?
That's a big question, but let me try to offer three answers.
I think one of the things that we have done remarkably well in the group from its beginning to today has been to pivot our research to what's needed as the field evolves.
If you look at the early Grouplands papers, they were mostly algorithmic.
And when you introduced me, you said, oh, he's known for algorithmic work.
If you ask how many algorithms I've done any significant changes in in the last 15 years, it's probably none.
I've certainly used machine learning techniques and anyone who doesn't is gonna become obsolete fairly quickly, but I'm no expert in them.
And in the early years, we needed those algorithms.
User user as an algorithm, I hate to say it, but it was basically obvious.
I define obvious as if that many different groups came up with basically the same algorithm at the same time, it was a very simple extrapolation of our understanding of the math of that kind of pattern-based computation at the time.
It was also inherently impractical.
The reason we don't use it is that as it was written, it scales really poorly.
It also has some interesting problems in different parts of startup and everything else.
The item item algorithm, we did a lot of the early work on it.
I think you have to give a lot of credit to Amazon, which was internally, had patented work around this and was doing work in that space that wasn't public, but that influenced the way the field was moving.
But it was entirely an algorithmic response that said, hey, we need something more efficient, something that can work and deal with the recognition of the asymmetry between the number of customers we have and the number of products we have.
That seemed reasonable.
The work we did on latent factor approaches and matrix factorization approximations was another piece of that same puzzle that said, well, wait a minute, now that we understand, there were certain downsides to the item item algorithms that we recognize, including a user experience downside when we started using it in our systems.
Maybe this would overcome that, and it has this benefit of being mathematically elegant and having a latent structure and a taste space behind it.
And it had some advantages, and it's still being used in a lot of ways.
It also had disadvantages, many of them similar.
And when we got to that point, by then much of our research had focused on user experience.
We were looking at the interface and what do you display?
How many stars do you need?
And do we show people how much data is behind or how much variation there is in the data behind a particular recommendation?
We started going into understanding the different kinds of tasks and circumstances people were going into.
And as we've pushed through, this takes me into the second piece.
We didn't just change, but we made a commitment very early that we were gonna make this experimental.
Part of that was making the original investment into running a study over the internet on Usenet News, which was completely unsustainable, but it was a great way to get data.
The reason we built MovieLens was we needed a platform where we could control the user interface without having to release a new news reader to people and get them to install it on a regular basis.
The original GroupLens experiment was just local stuff.
The version we expanded to a couple hundred people, we wrote all these special instrumented news readers and people had to download them and install them to participate.
That's a big burden.
If you tried to do that today, people's systems would say, you don't have the authority to install that.
It would be a real problem.
And so going to the web and going to movies made that much easier, allowed us to control the interaction.
And since then, we've gotten to experiment on more than 300,000 people who didn't come because we recruited them to an experiment.
They came because they like movies and they were willing to be part of that experiment with us.
Let me take you to PopRox and then I'll say the third thing I'm pretty happy with.
PopRox is a news recommender platform.
People can go see what we're doing at poprox, P-O-P-R-O-X.ai, but it is a platform that was built from the ground up to support other people in research and not just us.
It includes the Group Lens Group, but it also includes collaborators at four other universities that helped build it.
We have grown a team.
We were very grateful to get funding from the US National Science Foundation to build this up.
And its intent is that people who subscribe will get a daily newsletter with a bunch of news articles that match their interests in some way reflecting a profile they edit and a profile that we build from what they click through on, but that they're also research participants in studies, some of which we're conducting and some of which other researchers will come in and say, can I have 200 people divide them into two groups and some of them get this and some get that and we'll see who clicks more or who thinks better things in a survey at the end.
It's our attempt to take that next step in transforming the field, in being out there and making it easier for people to run experiments.
And I think it's an important step.
And that takes me to the third thing I'm really pleased about is one place where we've really succeeded is in growing the community.
Partly all of our work has been centered on graduate students who do and go out there into industry or into academia.
And many of them have continued to work in places that influence the field.
But we've also done a lot of work from the online course that Michael and I have done that has brought people into the field, the tutorials, the doctoral symposia, and now, you know, and the tools, the recommender toolkits.
You know, Michael's work has been immense in that space, but in general, we've tried to be as open as we can with everything we have, because the goal is to grow a community that not only cares about recommender systems, but that we hope thinks about recommender systems considering the same factors that we consider.
Yeah, yeah.
This PopRocks platform that, yeah, one of their newer things, like when did it start, like two or three years ago?
Yeah, we got funding for it about two and a half, maybe three years ago.
And we started building it pretty right away.
We've been operating for more than a year with public users.
We're growing.
At this point, we have probably five or 600 people who have signed up, not all of them still active, but most of them still actively receiving the newsletter every morning and receiving a survey about once a week.
And we've run at this point, a handful of studies that try different things that rewrite the article headlines to see if that changes people's willingness to read them, that change which articles are picked.
We've tried to make the newsletter better in everything we do, but we also will run studies where we just don't know what's better.
What is currently the biggest bottleneck there?
I believe it's possibly the number of recipients of the newsletter, so of users actually using the platform, or is it really, is there a lack of research proposals coming from you or coming from the external side?
So I've already checked this, so you can go to poprocks.ai and submit such a proposal, get in contact with a team like you.
How would you further grow this to facilitate more research being conducted there?
It's not that we're gonna have a lack of proposals, although that doesn't mean we wouldn't like to have more of them.
We would always like to have more proposals.
Part of it is having enough active users.
We have lots of people who get the newsletter, but we don't know what they're doing, but they're not clicking on articles.
And if they're not clicking, if they're not answering, we don't get any data.
But I think a big part of it, and it's a part we knew when we got into this, and we hope we will at some point get the funding to go to stage two of this, is our feedback is highly limited by the newsletter model.
When you send something through email, you can't track very much more than what links people click on.
And so what we would love would be to be able to understand where do they dwell, are they reading the headline, and then not clicking because they learned what they needed to from the headline, and it's positive, but it's not necessary to read the article, which happens all the time, right?
There's a sports article going on, and you hear, oh, Argentina upsets Brazil three to one.
Oh, I don't need more than that.
That's enough.
If you're a real fan, you might wanna know who scored the goals and all the rest, but if what you want is the news, that might be all you care about.
And we don't have a way for you to say to the system, that was a good one.
Keep sending me more of this, because today's mail browsers don't mostly let you provide feedback that gets sent back in the form of a message.
They don't let you track different types of images or other things in them.
And so not having enough data, we're gonna address partly by, yes, we'd love to recruit more people who are interested.
We would love to get people who are more active, which is why we keep trying to improve the newsletter.
Eventually, if this is gonna be as successful as we want, we're also gonna have to move to having a mobile app and some other ways of reading that are viable for getting more detailed feedback.
So really rendering this from a newsletter that I keep getting via email into a full-fledged news app, where I can see those news that are actually, I see it's the Associated Press that you use for these news that are- That's what we have right now.
Our intent has been to try to get other sources as well.
It took a long time just to get one contract in place for content.
And so we've got to see where things go, but people out there who have good contacts for good quality sources that they think would be willing to be included, we're always interested in talking to folks.
So hear this out.
And like as always, we will add all the references to the research, but also to get in contact with Joe in the show notes.
So that if there is somebody who can help further grow this project in whichever direction, especially in the one that you mentioned, getting more news providers there can help and reach out.
With this intent of making this a platform for online experimentation, which is actually part of this acronym, have you already seen a shift in what people want to research, what people want to produce?
Because like one could also not argue, like there are a lot of commercial news agencies and providers, and they do a lot of recommendations and personalization already there.
But we maybe just get to know one side of this, like all the successful project that are then presented at news recommendation workshops, or as part of papers in the main track or wherever else.
So we see research being conducted in the fields of news recommendations, but maybe it's just this research that, and this goes back to how business sets priorities and their research that is conducted through this platform is kind of free from the surrounding priorities.
And with that, do you feel that there's a shift in what is being proposed and what people really want to assess and research without being like tied to this business priorities within that project?
You know, honestly, I haven't seen that.
One would expect that I would.
I think there are two things that have stopped that.
One is that right now there's very little in the way of news personalization being carried out by the people who produce news.
They're all interested in it, they're all a little scared of it.
That doesn't mean that there aren't portals.
You know, Google News has been around for a long time and, you know, derives from one of the earliest personalization systems from the 90s, but they're not the news producers.
And what we've seen talking with people who are news producers is they have a very strong journalistic ethic.
You know, you don't go into news to get rich because you won't.
You don't even go into news being sure you'll be able to spend your entire career there, but you do go into news because you feel it's important and you feel it's important for people to know the stuff that's important.
And for good and for bad, our system actually does a pretty poor job of that specific thing.
We don't know what's important.
We're very much positioning ourselves as a supplemental source of news because wherever you go first in the day, we'll tell you that there's a war going on or the price of oil just doubled, but there's not as much focus on telling you from what among the next thousand interesting stories are the ones that you'd actually find interesting to hear about.
I think that balance, and this will be a challenge for us like it is for everybody else, that balance of what would it be irresponsible if you didn't know, and what will give you the delight to continue reading is a really interesting challenge for news as an industry.
And I think there's a lot of fun work to be done in this field.
The stuff that has started is pretty human centered and pretty human based, right?
The news letters that are out there to the most extent have human editors who have picked what are the important stories.
The newspapers have human editors.
The websites have human editors.
Even Google News, assuming it still works the way it has in the past, searches around the web to figure out, well, what did all the human editors think were important?
That's probably what we should think is important too, which is a pretty reasonable strategy.
It becomes very different when you come back and say, okay, now what can we do if we don't start and that is the starting point.
Something I would like to talk with you more about is the ACM Conference on Recommender Systems, short RecSys, which has been taking place for the very first time in 2007.
And in 2007, it was taking place in Minnesota and you were the general chair for this conference.
And it's nice that it's kind of a homecoming this year for RecSys coming back for the 20th RecSys.
So it's been a tremendous history for that conference, for this premier venue for RecSys practitioners and researchers.
How have you perceived this grow of the conference and where did things start actually before the first RecSys?
Sure, so the community that we think of today as the recommender systems community, the first time it got together, as far as I know, and I don't think there was much of a community before that, was in the spring of 1996 at what is sometimes called the Berkeley Collaborative Filtering Workshop.
It was organized by Hal Varian, who was the Dean of the School of Information Management and Systems at Berkeley and Paul Resnick.
And it was supported by a company Infonautics, which I honestly don't know if they still exist, but at the time they were known for products geared at school students.
They had a system called the Homework Helper and the Electric Library that were early internet systems.
And their founder and CEO had a lot of interest in this personalization stuff and was willing to underwrite the workshop.
Most of the people doing work in the field at that time, at least in the US that we knew of, got invited and almost all of them came.
There was one exception, which was a Patty Ma's group at MIT that had already started commercializing what they did at Agents Inc that became Firefly Networks and later I think was acquired by Microsoft, but they were in a period of not talking about what they were doing for obvious reasons.
And that group got together and it was amazingly useful.
Just seeing what everybody else was doing and the different techniques people were using and the different problems they were solving was really valuable for the field.
From that point over the next 10 years, there were a whole bunch of different workshops.
Most of them were workshops that were being held at conferences and people would go to SIGIR or CSCW or CHI or some other conference.
And at some point we realized sometimes there's two or three of these in the same year.
A couple of them were standalone workshops.
They were mostly in Europe.
There was a workshop from the digital libraries community that was held as part of the Delos series in Europe.
There was a summer school on recommender systems that happened in 2006 that brought a bunch of people together to Bilbao.
And around that time, I had started the discussions with John Riedel and then eventually with others in the field saying, you know, we're all running into the problem that we can't go to all the different workshops.
We're missing what's going on.
Maybe we're ready to have our own conference.
And ACM was very happy to serve as the sponsor and provide the support we needed.
But honestly, at that point, we didn't need very much support.
We were a small community.
We held it in a room at the university campus.
We had lunch outside that room or I think on the weekend today we were there, we ate at the campus club.
We didn't do anything very fancy, but we had over 120 people come together.
And at that very first rexis, a substantial number of those people were from industry.
There was a delegation from Amazon.
There were people from digital.
There were different places that were all there.
We did one smart thing.
I remember having a bunch of the leaders in the field over for dinner at the end of the conference.
And I told people, we will feed you and let you go.
Once we figure out who's gonna host the second conference.
And we had a very quick volunteer and we moved to Lausanne and then New York and continued to grow as we went around.
The interesting two things about this conference are it has always been heavily international.
We hold this conference as often or more often in Europe as we do in North America, because while a lot of the rapid commercial growth was in North America, the large number of companies and university groups has been in Europe.
And we've since then extended into Asia as well.
And that's been really important for the conference.
And the other side of it is, it has always been a conference at the intersection between research and practice.
It has always had substantial industry participation.
Some of them industry researchers and some of them advanced industry practitioners, or even people who are coming just to be brought up to speed in the field.
There were a lot of people who come to exactly one RecSys conference, because they can come, they can take these tutorials, they can understand what's going on and go back into their job, a little better prepared for what's going on.
I was very excited when we knew we were coming up on the 20th conference, which is weird, because it's not really the 20th anniversary, but it is 20 conferences, which is a pretty nice milestone.
And so we were very excited about bringing it home.
I was able to have two excellent co-chairs who were part of this community for much of its history, George Karapas, who was a key figure in the item item work that you've talked about earlier, and Gedis Adamovicius, who was involved, I think since at least the third, maybe even earlier than that in the RecSys series, he has been involved from the business school side of this, which has always been a piece of the community as well.
It's not been all computer scientists.
We've had psychologists and business school people and others who see RecSys at this really interesting coming together point.
And so I think it's gonna be a fantastic conference.
We're looking forward to having people here.
We'll show them a very good time and we'll give them a chance to interact and engage with people in the field.
Yeah, for me, when I joined this field in 2016, so the first thing that I did is watch out on online tutorials or online courses, and then I found the course that you and Michael created on Coursera, and I guess it is still there, maybe in the same state.
And this really got me jump-started.
And yeah, about nine months later, I finished up the thesis, and then I sat along with my mentor, okay, now let's visit the venue where this whole community meets.
And this was then actually RecSys 2017 in Como in Italy, which was great.
And since then I've been attending RecSys and it's always like a great experience.
You come back and your head is full of ideas and you also feel a bit drained.
It's an intense week, of course, not just from the angle of seeing all this research, talking to a lot of people, but also because there's, I would say, a very good connection beyond the pure conference within this community.
So this actually brings me to one specific question you might be able to answer or not, but was there already Karaoke at the first RecSys or not?
Not that I remember, let me rephrase that.
Not that I was invited to.
If it happened, it happened somewhere that I didn't know about.
I don't think there was a readily available Karaoke venue, but some interesting things have grown as traditions as the conference has matured.
Looking back at these 20 years of RecSys, what was kind of the things that surprised you the most in that journey?
Like something that you would have never expected or something that surprised you positively or negatively.
I mean, we already talked about some disappointment about papers getting like too much centered or focused on just some tiny algorithmic improvements that don't matter in reality.
But are there other things among this that looking back to this 20 year or at least 20 years of history in RecSys surprised you?
Well, I think the biggest surprise and there's positive and negative aspects of it, but I'll focus on the positive, is that we're still here and we're still thriving and maybe even growing.
If you had asked me 10 years ago, how many more RecSys conferences will we have?
I would have guessed that at some point we were gonna wind down, that it wasn't clear to me that the community recognized what would make us different, that we needed a separate conference anymore.
And some of that is negative.
Some of that is this feeling that if what we are is really just a machine learning application and we're becoming that to some extent, do you need a conference for that?
Do you need a thousand people getting together to talk about some application of machine learning?
And I think what I and many in the field who've been here for a long time were trying to do was to say, don't go down that path.
Don't make that mistake of we're just another application because what we saw was that the flood of submissions, of research papers and in some cases workshops and other things, looked like they were going in that direction.
I think we were pretty deliberate about trying to set a vision of why are we here?
And the question was, would the community buy that vision?
And my version of that vision, it's not written down, other people will have their own versions of this, is that we are a machine learning or a computation problem in part, but we're bigger than that.
What we're really about is this highly constrained, highly contextualized problem that applies historic data to creating appropriate valuable user experiences around situations where you're dealing with product choice, product information, et cetera.
And that's not just an algorithms problem, it's an interfaces problem, it's a data problem of what data do you look at and use.
It's a business problem.
And that part of what makes these problems so interesting is as people recognize, oh, it's not even just optimize some measure, it's fundamentally multi-measure, it's often multi-stakeholder.
Now we've got people studying recommendation for marketplaces where if the producers starve, then there won't be anything for the consumers in the future so we have to actually care about balancing both sides.
We have people who are looking at recommendation in contexts where overuse doesn't work.
We're mostly used to products that work because any quantity is okay, right?
So if you're streaming movies, any number of people can stream them at once.
Why was Netflix so interested at the beginning?
Because they didn't wanna have more copies of the DVD to mail out, and they needed a system that would help people find the DVDs they had extra copies of that they wanted to watch.
And so all of these things were your quantity constrained.
That could be tourism, right?
Not everybody can be sent to Barcelona unless they wanna be bombarded with water guns and who knows whatever else.
But it can also be dating, it can be matches between suppliers and customers, it can be matches between news information that if nobody sees certain information, those producers go out of business.
To me, that's part of the delight is that we have managed to keep alive a really interesting problem in part by understanding it, honestly, much better than any of us did when we built the early solutions.
From that experience, what would be your advice for starters in this field?
On the research side or on the practitioner side, maybe in their answers, this doesn't make too big of a difference, but how given all that experience and giving also this like shift of attention away from the algorithmic problem to the user experience, how would you recommend a person nowadays getting started in this field?
I know we are, or this is a podcast that positions itself to be for advanced or intermediate people, but there are still like people joining this field and getting started.
Like what should they maybe do different?
What would be your advice for them?
My main advice, and I get to give this advice often because I still have graduate students and I still mentor them, is immerse yourself in an application and find the opportunities, either the problems or the opportunities to do more, do better that are relevant in that application.
And as you move forward, you might be able to find multiple applications and find commonalities.
The thing that always becomes challenging for me is when somebody comes in enamored by a specific tool and they just want to hit every nail with their hammer without noticing that it's a nail that we're trying to pull out, not hit in.
If you come in and say, oh, I just found this great large language model, I think it can do everything.
It's not that you're wrong, maybe it can, but if you don't stop first and say, well, let me understand the domain, let me understand the users and what they're trying to get out of this domain, let me understand the operator of the recommender system or the producers or the others and what their interests are.
Okay, what is it that I believe my wonderful large language model could improve for them?
What's their biggest pain point or at least what's a pain point for them that I think this tool lines up with?
I'm excited about technology too, but you've got to anchor it to something.
Maybe circling back to the course, which was, as I said, my starting point on my RecSys journey, is there some intention to revamp this, bring it onto a new level, incorporating insights from the past decade?
I'm not sure what has happened since then when I took it in 2016, late 2016.
What is your take on that?
So we've been through a couple of editions, but that might be that you hit the latest of those editions or maybe the one before the latest edition.
The course was actually something that John Riedel and I had planned before he passed away untimely.
And I was really grateful that Michael wanted to say, no, let's go through and still do it, which is a tribute to John, his advisor, but especially to Michael as a graduate student, this is not what you normally sign up to do.
And I hope people recognize the immense amount of effort that he put in.
I put in work too, but I was used to teaching at a pace and I was ready to do this.
And he was thrown into it with a relatively little prior preparation.
And I thought he did a fine job.
We did a first version that was a single course.
We did some updates to that, primarily around the assignments and other pieces.
And then we got some pressure or some incentive to turn it into a multi-course series with a capstone project and a certificate that people could earn.
That might be the version that you took where it was already broken into four different courses.
Yeah, I guess so.
And we have periodically gotten together and said, hey, it's out of date.
Two or three times we started outlining what's all the stuff that's out of date and are we gonna do another one?
And quite honestly, we've never found ourselves with the time and energy to take away from what we're doing.
To really do that other one.
And I feel bad about that because I think it would be valuable for the field.
I also feel that there are other resources that have come out.
There are now textbooks in the field.
There are handbooks.
There are other good things out there that if we were to do it again, I think we would take a very different perspective on it.
And what would likely end up happening is we'd come back and say, what is a meaningful sort of mid-level university course?
And can we deliver the material to support that course online and then offer it to people who don't have a local instructor to self-study, but really gear it towards people who might have that local instructor who's using it to supplement doing exercises and learning all the stuff in person?
I think we made a mistake trying to do everything in one course, including levels of software support that were very hard to maintain as things changed.
And if we think of it more as a library of components, I think we would have more success.
But I can't give you a time or date.
Honestly, if somebody else stepped up and said, hey, we wanna do it, I would feel probably very relieved rather than anxious or jealous and would offer them all the support I could.
Yeah, yeah.
Okay, then this might also be the right place to advertise for this.
So if somebody wants to relieve you of that burden, then you heard Joe Constan at this point.
So maybe give it a chance.
Maybe if we step back from the actual course and the learning material, but take a look back at the RecSys conference.
We have been talking a lot about the past, also the near future and the present of the RecSys conference.
But we should definitely also cover the future and where things might be heading or what would be your desire of them heading to without like you maybe wanting to influence people too much or wanting that, I'm not sure.
Where do you see the future of recommender systems?
It's a very broad question, especially these days where we keep getting haunted by the next best agentic AI solution and whatnot.
So will we be all using some form of agentic AI recommender system, whatever that could be meaning in the future?
What would the field look like maybe from today on in 10 years?
I do think that whatever AI techniques are out there, the field is going to find ways to incorporate.
I think if the field is going to remain relevant, it's going to remain relevant by remaining multi approach and multidisciplinary.
It is going to have people who think about business and people who think about human decision making and consumer psychology.
It's gonna have people who think about technological solutions and it's gonna have something that I've been thinking a lot more about lately.
One of the great things that happened with PopRox is we got the chance to hire a full-time research engineer.
And we brought in Carl Higley, who has been in this community for a while and has done work at a couple of companies that were interested in doing recommender systems work.
He's worked at Spotify, he's worked at Nvidia and a big part of what he has been showing me is, hey, the research community always focuses on a small part of the problem.
They think it's the algorithm, it's the model.
It's bigger, it's the whole system.
Your model doesn't do any good if you haven't figured out which data to get and how to get it and how to get it into the shape that you want.
Your algorithm is useless if it's not embedded in a system that can deliver its results in a useful way.
You know, to some extent, people who build recommender systems have always realized there are other components to put together, but I don't think they've always thought about all the design choices that those components expose.
And so I think there's some really exciting engineering, both commercial engineering and research engineering, that still remains here.
And I think if we really do rally around a vision of the problem, there's a pretty good chance that we can continue to do some significant advances and keep educating people and keep pushing the field forward and all the other things that a good conference should be about.
We don't do a lot explicitly, but I think we've all known from day one, one of the objectives of a conference like this is to put companies that are hiring in contact with students who might be looking for jobs.
And a lot of people have gotten hired.
That's a great thing.
I think there's other things like that, that just bringing a community together can be very powerful.
Yeah, I guess spot on on what you are saying about this distinction between a model and a system.
I guess we had that very early on in one of the first episodes of RecSys.
And just as you were saying this, I needed to look it up.
It was the episode with Even Oldridge who also works for NVIDIA mentioned to make that distinction.
Sometimes when we say recommender system, what we actually mean is the model, but really thinking about the system goes well beyond just the model, which is just a tiny piece in that overall and that systematic perspective.
So would this be what you are referring to, thinking this problem more broadly involving other areas of expertise in the community?
Absolutely.
I think those areas of expertise are in the community.
We may need to grow some of them.
But if you're Walmart and you think about a security system, you don't think about, well, what's the algorithm that detects whether somebody has scanned the right purchase and say, that's what do we do?
You start with the greeter and the design of your shopping carts that you can see everything through them and you don't have opaque areas to hide something all the way through the entire system.
And I think that's gonna be the case in a recommender system as well.
We don't often do this today, but more of our recommender systems as we have greater familiarity and interaction through chatbots might be more proactive in engaging with their user or customer to answer the things they don't know about.
Somebody comes in and says, hey, can you suggest a nice pen for under $30?
Rather than just showing a bunch of pens, maybe you come back and say, sure, but are you thinking about one that you're gonna wanna use for yourself for work or home or for a gift for somebody or whatever?
We could do a lot of different things.
Or maybe that person comes in and you look and say, they bought a bunch of pens in the past they never told us what they thought about them.
Maybe this is a time to have a little discussion like any good salesperson would if they had the data saying, I know you got this cross pen and I know you got this Mont Blanc pen.
What did you like or dislike about them?
Or how's what you're looking for today the same or different from what you got those times?
We can start to do that when we realize that, oh, conversational interfaces are now common.
And obviously people tried to do that 25 years ago but the conversational interfaces there didn't have the level of natural language support we have today.
That's an interesting perspective.
And what also resonates with me as you're saying this is maybe better bridging the virtual and physical world of things in that sense.
If we take a step back this podcast is one of the efforts in this and giving people a voice like eliciting these different perspectives on recommender systems.
As we were talking about Michael, like we talked about the role of fairness in recommender systems.
More recently, we put some emphasis on certain aspects such as serendipity, diversity.
With Elizabeth Lacks, I had the chance to talk about psychology aware recommender systems.
So there again, we see this different perspective interdisciplinary views on RecSys and how we can like merge things together.
If you look forward at this which people would you recommend among those people that come to your mind or which specific areas you feel are under covered and should get hurt in this podcast?
Wow.
I don't think I'm gonna give you a concrete answer.
I feel that the ones that I've noticed I don't know are the ones that I would recommend if I had seen all the ones that I didn't notice.
I think broadly speaking, I love the work that follows three themes.
I love the work that looks at human decision-making and psychology in the context of choices.
And I think there's been some excellent work from many people about thinking about how our preferences created in the first place.
How do they change?
How are they instantiated in things?
And recognizing that part of what that means is whether a recommender system worked or didn't work might depend on what you set the user up for as much as what you did in the recommender system.
I really like the people who come at this from a perspective that's closer tied to the business world.
Not all of the people who come from business schools are business type people.
Some of them are computer scientists in a business school who do the same thing that I would do.
But some of them come in really thinking about these with a context that brings in marketing and pricing and all of the things that are the practical concerns that any of us who want our system to work in the real world have to think about.
And lately, I've been really excited by the work that's going on in multi-stakeholder, multi-sided recommendation problems in marketplaces.
That this interesting space at the very simplest where you have two different groups of stakeholders, you know, one of them may be producers, one may be consumers, or they may both be co-consumers being matched with each other, plus a third stakeholder that so often is invisible of the market maker.
Because if the market maker doesn't succeed, then the market goes away.
And how do you maybe not optimize, but at least engineer for sufficient outcomes for each of them is really exciting.
I think there are other areas where it's gonna be important to read.
I think we're still trying to figure out the ethics of these systems.
And that's gonna be an interesting issue because there's sort of an academic ethics point of view here.
But there's also a very practical business ethics perspective and business ethics are often a lens through which that long-term view we talked about earlier comes to bear.
The point of business ethics is often about building reputation and trust.
And it might be a way to capture the long-term value that we don't know how to capture with our other metrics.
And so I think there's a lot of exciting stuff out there.
None of this is to say you shouldn't read the latest large language model work.
I mean, I read an AI brief every morning.
Does this see what's going on?
Because the capabilities are remarkable.
But I think what makes it uniquely RecSys is when you take those other perspectives and bring them in.
Let us know what is the AI brief that Joe Constant is reading every morning?
Right now I'm reading the TLDR one.
All right.
I've looked at some others.
I continue to, when I see something good, I will give it a try for a week or two and try to keep my number of subscriptions down to seven or eight total and do it.
Sounds good.
All right.
I guess this has been plenty of great insights, recommendations as well from a long-time luminary of the field.
I feel this definitely is a great honor because as I said, almost a decade ago, I started with it.
And today I have the chance to interview you for my own podcast on this topic.
And I hope that many people will enjoy listening to this very session and also to many of the others.
For the moment, I want to say thank you very much for taking the time for this.
It was great to have you on the show.
Well, it's been a pleasure and I enjoy what you're doing and I think it's valuable work.
Please keep it up.
Thank you.
We'll do so.
Joe, have a wonderful rest of the day.
And I guess this time I don't need to ask you whether we'll see each other at RecSys because I guess for the general chair, RecSys is quite very obvious.
I'm pretty sure I'll be there.
Sounds good.
Take care.
Thank you.
Bye.
Thank you so much for listening to this episode of Rexxperts, recommender systems experts, the podcast that brings you the experts in recommender systems.
If you enjoy this podcast, please subscribe to it on your favorite podcast player and please share it with anybody you think might benefit from it.
If you have questions or recommendation for an interesting expert you want to have in my show or any other suggestions, drop me a message on Twitter or send me an email.
Thank you again for listening and sharing and make sure not to miss the next episode because people who listen to this also listen to the next episode.
Goodbye.
Bye.