Forum data fetching

[Dolda2000]

Wouldn't it be nice to implement a machine-readable output for the forum? That way, instead of polling the forum manually every now and then, one could have a little process running, polling it for new posts, notifying the user whenever a new posts comes up.

Sure, this is possible right now by parsing the HTML, but machine-parsing HTML into something more semantic is just... yucky, and also, who knows when the HTML output of the forum changes? A real machine-readable format would also probably be saving a lot of bandwidth and so on.

And by machine-readable, I am not referring to XML, but rather some nice, UNIX-parsable ASCII format. Something that could easily be parsed by Perl, for example.

I realize that it would be quite a bit of job, but won't you at least think about it, Jason?

[jasonlambert]

have a look at feeds.linuxforums.org - you can choose to receive the feeds in a plain text format which you should be able to parse easily with perl.

J.

[Dolda2000]

Indeed, I just noticed that there was such a feed as "My Topics", which was what I had primarily been looking for.

However, would it be possible for you to add some more info to the feeds? I'm looking for "Date of last post" and/or "Last poster", so that the feeds can be compared to each other to see if anything has actually happened to the individual posts since the last poll.

Btw., how does the "My Topics" feed work? The URL ends with "myposts-10-AReallyLongId". What's the "10", and what does the really long ID refer to, and does it require cookies and the "remember me" feature?

I'm also noticing that the "My Topics" feed isn't synchronized with the ego search page. Is it just caching somewhere, or might I have lost some "remember me" status in a cookie or something?

[lakerdonald]

Indeed, I just noticed that there was such a feed as "My Topics", which was what I had primarily been looking for.

However, would it be possible for you to add some more info to the feeds? I'm looking for "Date of last post" and/or "Last poster", so that the feeds can be compared to each other to see if anything has actually happened to the individual posts since the last poll.

Btw., how does the "My Topics" feed work? The URL ends with "myposts-10-AReallyLongId". What's the "10", and what does the really long ID refer to, and does it require cookies and the "remember me" feature?

I'm also noticing that the "My Topics" feed isn't synchronized with the ego search page. Is it just caching somewhere, or might I have lost some "remember me" status in a cookie or something?

The 10 is the number of links in the feed. I believe the max is 50

[Dolda2000]

Actually, I did discover myself that the "10" is my LFDO user id. It would have been strange otherwise, since the number of links in the feed is 15. The question of what the ID means still remains, though.

By the way, the feed of my posts still hasn't updated. Is this a bug or a feature?

[lakerdonald]

Actually, I did discover myself that the "10" is my LFDO user id. It would have been strange otherwise, since the number of links in the feed is 15.

By the way, the feed of my posts still hasn't updated. Is this a bug or a feature?

Hmmm...I was almost sure that the number was the number of links. I guess that's somewhere else in the url.

[Dolda2000]

Yup:

http-//feeds.linuxforums.org/rss/15/myposts-10-1f351f5d0be2024bdd5dc7cd7c5c467e.html

(Colon replaced with dash to avoid auto-URLification)

[lakerdonald]

ah...I don't use the feeds much. I basically just check here in bursts anyway, answering like 20 threads, then leave my computer/do something else and come back in an hour, and repeat the process... I just read down the list of Unanswered topics, so the e-mail topic reply notifications suffice.

[jasonlambert]

However, would it be possible for you to add some more info to the feeds? I'm looking for "Date of last post" and/or "Last poster", so that the feeds can be compared to each other to see if anything has actually happened to the individual posts since the last poll.

I could make a verbose text feed with the extra data, leave it with me.

Btw., how does the "My Topics" feed work? The URL ends with "myposts-10-AReallyLongId". What's the "10", and what does the really long ID refer to, and does it require cookies and the "remember me" feature?

The "10" is your user ID, and the 32 character long ID is a private secondary MD5 hash that is associated with your account, the MD5 hash and the matching user ID need to be presented at the same time to retrieve an individual's "myposts".

I'm also noticing that the "My Topics" feed isn't synchronized with the ego search page. Is it just caching somewhere, or might I have lost some "remember me" status in a cookie or something?

Not 100% sure, off the top of my head I think the RSS feed is ordered by topic start time and the ego search on the forum is ordered by post times.

[Dolda2000]

I could make a verbose text feed with the extra data, leave it with me.

That would be great. Thanks!

The "10" is your user ID, and the 32 character long ID is a private secondary MD5 hash that is associated with your account, the MD5 hash and the matching user ID need to be presented at the same time to retrieve an individual's "myposts".

Hmm... how is it "associated" with my account? Is it so intricately associated that I should edit it out of my previous post and use HTTPS with it?

Not 100% sure, off the top of my head I think the RSS feed is ordered by topic start time and the ego search on the forum is ordered by post times.

Indeed, you seem to be right. I just posted in a couple of threads I hadn't posted in before, and they did indeed pop up, but not in the order I posted (but rather by their creation time). Would it be possible to change this?

Of course, when I think about it, I could probably just perform the ego search with wget. If anything, that would break if anyone were to change the HTML code backing the search page, but I guess that's not an imminent threat.