Hey /r/personalfinance,
I am a developer by trade and love commenting and lurking on this sub. I see a lot of similar questions being asked on this sub (not that it's a bad thing) and began wondering what the most common questions were. Being a developer and wanting to try my hand at some (very basic) natural language processing, I decided to make a script. Here was what I did:
- Grabbed the post title from the top 1000 posts from each month of the last 6 months (6,000 total)
- Filtered out to only have questions (ended with a question mark)
- Looked for the most common words, phrases, etc.
- Tried to distill these phrases into questions
Most common topics
To get topics I essentially filtered out any pronouns (I, he, she, etc.) & common words (and, the, etc.) that wouldn't give me a topic. I then counted which words appeared most often. Here are the top 10:
| Word | Number of times seen |
|---|---|
| credit | 252 |
| pay | 159 |
| money | 114 |
| card | 109 |
| car | 90 |
| student | 88 |
| 401k | 86 |
| debt | 77 |
| job | 76 |
| account | 69 |
Pretty interesting, I expected to see more about retirement. It looks like a lot of people concerned with debt & credit, which makes sense.
Most common phrases
To get common phrases I had to use a natural language processing package using Python. Essentially, it did the same thing as the getting topics, but required a package as finding similar 2-3 phrases is more difficult than just counting words.
Most common 2 letter phrases
- credit card
- to pay
- my credit
- pay off
- the best
- credit score
- student loans
- to buy
- a house
- a car
Most common 3 letter phrases
- a credit card
- my credit card
- my credit score
- credit card debt
- buy a house
- a roth ira
- can i afford
- how much should
- off student loans
- student loan debt
This is where it gets interesting. There were still a ton of questions about purchasing, loans, and debt. I think this is the level where you can start to see what the common questions might be. Using these phrases I then went and looked at the actual post titles (by hand) .
Most common questions
- Should I get a credit card? / Will I qualify for a credit card?
- How to best pay off my credit card?
- How can I improve my credit score?
- Why did my credit score drop?
- When can I buy a house? / Should I buy a house?
- How much should I be saving?
- How much can I afford for a house/car/etc.?
- Should I be contributing to a Roth IRA?
- Should I pay off X debt vs Y debt first?
- Should I pay off debt faster?
- Should I max out X account?
Nothing too earth shattering, but still interesting nonetheless. I suspect that this list might change as the economy does well/poorly. It would be interesting to do another one in 6 months from now.
I will be open sourcing the code if anyone is interested in the near future. I suspect that this type of analysis would be interesting to do on other subs. If you are interested in how I did the analysis for this sub, I have a post here on how I did it. I'd love any feedback on this as well and how to improve!
TL;DR
I analyzed a bunch of questions from post titles in this sub. Many people seem to be asking about debt and being able to afford items. Not as many questions focused on retirement as I expected.
Submitted January 23, 2019 at 12:04PM by DoomxPatrol http://bit.ly/2FZtvyP
