Data visualization of trends in Pseudopod forum participation.

[rendall]

I created a quick data visualization of forum participation for each episode of Pseudopod which you can see at this link: https://vigilant-hugle-1fb35f.netlify.com

Each episode is represented in chronological order along the bottom, x-axis, from left to right. The left y axis represents the number of comments (replies) in its respective Episode Comments thread. The right y-axis represents the corresponding number of views of each episode's thread.

You can see the values for individual episodes by mousing over the graph. In the screen capture below, for example, you can see that the forum thread for episode 186 Ankor Sabat received 27,733 views and 69 comments.

[Bdoomed]

Hey this is really neat! What did you use to gather the data?

[rendall]

I remember I just scraped it from the forum webpage itself using javascript in the browser console

[rendall]

What did you use to gather the data?

To be more helpful, I'll write out briefly what I did, but I will probably write a more comprehensive tutorial in the future.

I opened the browser console window and scraped the data from there. I do not exactly remember the process, but we can more-or-less reproduce it this way:

• In Chrome, CTRL+SHIFT-J opens the javascript console.
• With the Episode Comments page open, cut and paste this code into the console and press ENTER

Code:

[...[...document.querySelectorAll('table')][7].querySelectorAll('tr')].filter((e,i) => i != 0).map( e => ({id:parseInt(e.children[2].innerText.match(/(\d*):/)[1]),name:e.children[2].innerText.trim(), views:parseInt(e.children[5].innerText)}))

• Never insert code into the console just because some rando on the internet asks you to. If you do not already understand the code, it is possible to get hacked. I promise that code is okay, but then, a hacker would say that, right?
• Right! Exercise caution in your life!
• Briefly, that code scrapes the webpage and outputs specific pieces of information into the console, so I can cut and paste it into a text editor.

• This will show something that looks like this

Code:

(20) [{…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}]

• Click the little arrow to the left of that to expand it
• The full deal should look like this:

Code:

0: {id: 628, name: "PseudoPod 628: A Spider Trapped in Wax", views: 123}
1: {id: 627, name: "PseudoPod 627: The Grave by the Handpost", views: 86}
2: {id: 626, name: "PseudoPod 626: Blue John", views: 64}
3: {id: 625, name: "PseudoPod 625: The Golgotha Dancers and These Doth The Lord Hate", views: 239}
4: {id: 624, name: "PseudoPod 624: Flash On The Borderlands XLV: Personal Narratives", views: 117}
5: {id: 610, name: "PseudoPod 610: Beneath Their Hooves", views: 726}
6: {id: 623, name: "PseudoPod 623: Greener Pastures", views: 202}
7: {id: 622, name: "PseudoPod 622: En Plein Air", views: 379}
8: {id: 621, name: "PseudoPod 621: Voices", views: 293}
9: {id: 620, name: "PseudoPod 620: Farewell Concert at the World’s End", views: 173}
10: {id: 618, name: "PseudoPod 618: Goblins & Little Orphant Annie", views: 336}
11: {id: 617, name: "PseudoPod 617: The Culvert", views: 655}
12: {id: 619, name: "PseudoPod 619: The Ghost Guide’s Tale & The Halloween Parade", views: 428}
13: {id: 613, name: "PseudoPod 613: WEIRD SCIENCE HORROR ISSUE #2: The Challenge from Beyond", views: 540}
14: {id: 611, name: "PseudoPod 611: WEIRD SCIENCE HORROR ISSUE #2: The Vaults of Yoh Vombis", views: 457}
15: {id: 616, name: "PseudoPod 616: Flash On The Borderlands XLIV: Objectification", views: 749}
16: {id: 605, name: "PseudoPod 605: The Town Manager", views: 875}
17: {id: 604, name: "PseudoPod 604: The Gorgon", views: 470}
18: {id: 615, name: "PseudoPod 615: The Shrike", views: 392}
19: {id: 614, name: "PseudoPod 614: WEIRD SCIENCE HORROR ISSUE #2: Figure 8", views: 461}

• From there it's a matter of cleaning it up in a text editor, removing the line numbers, adding commas and such in order to get it into a form that d3 likes.
• Repeat for all pages
• Once all data is collected, d3.js can consume the data and spit out a nice interactive visualization

If you want a more full write-up, let me know.