THIS GUIDE IS OUTDATED. WILL WRITE A NEW ONE SOON

INTRODUCTION

This guide has been in development for a long time. I’ve always thought that RSS readers such as Feedly or Innoreader were bothersome at best and useless at worst, they constrict your news-consuming to their platform and besides their paid subscriptions, the free version doesn’t bring much to the table for me to justify having to install one more app on my devices JUST for this reason. Which is why I always edged the idea of taking every single RSS feed I follow into a singular Telegram group chat. Well, now it’s time.

The worst part about this is that every single FOSS software I’ve tested so far has been underwhelming or abandoned. Since I’m not a coder the best solution that came to mind was using node-red as I remember successfully experimenting with it on a LoRa network with MQTT environmental sensors.

As a plus, you get your Telegram bot with a cool name, a not a lame one.

This guide is based on my installation on a Raspberry Pi 4 (4GB) and Ubuntu 20.04LTS ARM64

WHAT YOU WILL NEED

This is just a small list, I will explain how everything comes together in the guide.

A MENTION ABOUT RASPBERRY PIs

If you’re using an RPi and if you don’t already know you should protect your SD card from wear and before doing anything in this guide I at least recommend installing log2ram from azlux to temporarily mount your /var/log folder to RAM and avoid SD card wear.

Besides the ENORMOUS issue with SD cards, the device is perfect for this guide.

As a side note: always have backups. ALWAYS.

TELEGRAM

I want to start with Telegram because it’s the most painless part of the guide, and I’ll keep it straightforward:

  1. Open a Telegram chat with BotFather
  2. Send the /newbot command and answer its questions
  3. Save the API key somewhere (you can recover this at any time)
  4. Send the /mybots command and pick your newly created one
  5. Go to Edit Bot to change name, username, profile picture and description (Optional)
  6. Create a new Telegram group, if you haven’t already and invite the new bot to it
  7. Go back to BotFather and your new bot’s settings and disable Group invitations

Once you’ve done all of this, you’ll find yourself with a new group and a new bot inside this group. Now we need to find the group chat ID so that the function we will write on node-red will know where to direct the bot when it sends messages.

Now, we could stretch our gigantic IT e-peen and I could write you all cool commands to find out your group chat ID but let’s not, so do as follows:

  1. Go to add members in your group
  2. Add myidbot
  3. Send /getgroupid@myidbot to the chat
  4. Save the output -abunchanumbers (make sure you save the - too)

If you’re paranoid like me, you’re gonna save this chat ID like the one ring if you were Gollum and kick IDBot from your group chat, never to see it again.

If you’re not paranoid maybe you should be.

So at the end of this very easy Telegram step, we should have:

  1. A new bot with its API Token
  2. A new group chat
  3. The chat’s ID

NODE-RED

Here’s the final boss.

installation

I’ll shoot through node-red's installation and assume you all know what the heck you’re doing with your networks and terminals.

Installing node-red is as simple as not reading what bash script you execute from the internet do:

  1. Go here and pick the installation script for your distro
  2. Go to your terminal and execute it (You don’t need sudo, node-red runs as your user)
  3. Enable the systemd service and start it to check if it’s working:
sudo systemctl enable nodered.service
sudo systemctl start nodered.service

If everything went right you should be able to reach node-red from your browser either at localhost:1880 or your-device-ip:1880 if you’re not on localhost.

Oops, I can’t reach it Wise, what do I do!?

  1. Check logs
  2. Check systemctl status nodered.service or journalctl -xe after starting the service
  3. If you’re using Red Hat, Fedora, CentOS —> check SELinux and Firewalld
  4. If you’re on Ubuntu —> check UFW
  5. If you’re fancy pants and want to reverse-proxy the thing, you’re on your own. I have my own NGINX problems to fix.

Once everything is working properly, turn it off:

sudo systemctl stop nodered.service

Configuration

To proceed, we need to enable the local file context store setting within ~/.node-red/settings.js.

  • Using your favourite Vi editor, open: ~/.node-red/settings.js (Just kidding you can use nano if you’re a loser /s):
vim ~/.node-red/settings.js
  • Find the context storage setting and enable by uncommenting everything and adding these options as follows:
contextStorage: {
   default: {
       module:"localfilesystem",
       config: {
           flushInterval: 1200,
       },
   },
},

Please note that 1200 is the value in seconds where data written by node-red to your localfilesystem will be flushed away. (1200 seconds is 20 minutes). The default is 30 seconds.

Now you can start node-red back again:

sudo systemctl start nodered.service

Configuring the bot flow

This is the juice, we will configure our flow to parse any XML RSS feed and deliver it down a chain of different palettes ultimately reaching the telegram palette that will be tasked with sending our modified RSS object to the group chat.

Here’s an example of how my functional flow is structured.

Flow Example

Required palettes

You can install these by going to the upper right hamburger menu inside node-red's web interface and clicking on Manage palette, install the following ones:

  • node-red-contrib-feedparser-x

This one is the RSS feed parser tasked with delivering a JSON payload containing an array of objects composed of all parsed RSS information. We’re using the x version that can keep persistent tabs on any feed you give it so that it won’t spam your chat with EVERY SINGLE ITEM in the feed any time you reload the flow in node-red (As opposed to what the non-x version does node-red-contrib-feedparser)

  • node-red-contrib-moment

This palette is tasked with converting ISO (Or Unix) time format to humanly readable time. This is entirely optional to you as you will see when we configure the entire flow and connect the palettes.

  • node-red-contrib-telegrambot

This is the obvious (external) palette that we need, and it connects to your bot and tasks it to send messages coming from our palette logical chain.

  • node-red-node-feedparser

I mentioned this earlier, we’re still gonna need to install this palette to figure out which objects any feed we want to use gives us so that we can properly configure each palette down the line.

The internal palettes we’re going to use are:

  1. inject
  2. function
  3. change
  4. debug

Creating the flow

Before we begin, some advice:

This is very important, so please read:

I told you to install node-red-node-feedparser, I highly recommend you use this coupled with a debug palette (connecting it at the end of the feedparser) and moving to the debug menu on the right of your node-red interface to see exactly what the feedparser gives you in terms of objects that you can use to compose your function’s content down the line. AS OBJECTS IN THE ARRAY MIGHT CHANGE FROM FEED TO FEED

You do this by connecting the end of feedparser to a debug palette and configure debug to show the complete msg object in its output setting.

Now every time you will update the flow, your debug window will give out every feed item with their JSON object array that you can pick and choose from. Make sure to expand every array and check were stuff you might need are and take note of their object level. For example: msg.article.pubdate contains the date and time of when the article was published.

You should use the debug palette at the end of every palette to figure out if and when something is giving you issues.

If you miss this important step before everything, God bless you and the desk punches you’ll throw to figure everything out.

 

The flow logic follows this connection line

inject –> feedparser –> change –> date/time formatter –> function –> telegram sender.

If you want to place these palettes in your flow already, go ahead as I will explain what each one of them does within this logic so that you can configure them as I go along.

 

inject is the first palette in our logic chain and it is tasked with sending the specific RSS feed URL you want to monitor for changes to feedparser it should be configured as follows:

  • One msg.payload containing the URL of your RSS feed
  • Inject once after 0.1 enabled
  • Repeat interval of n minutes (where n is the number of minutes you want it to inject the payload to feedparser)

Why this? Because feedparser-x needs to be injected with the feed continuously for it to receive the feed payload (as you can see the feedparser palette doesn’t have any configuration) and keep tabs on what’s a new item, and what isn’t and then pass it along in case something new comes up.

Inject Config

 

feedparse x only accepts the palette name. Its behaviour is dictated by inject, as I explained already.

 

change in my case, is used to Set the msg.title object with the name of the feed we will be using to put as the first message of how we will compose the telegram message to be sent. In the case of the US-CERT Vulnerability Bulletin I’ve put the following string inside a msg.title object: New Weekly Vulnerability Bulletin from US-CERT.

 

date/time formatter is an optional palette I added to make use of the msg.article.pubdate object from the feedparser palette, and since it comes out as ISO time I needed to convert it to DD/MM/YYYY HH:mm (or MM/DD/YYYY hh:mm if you’re weird).

To do so, you need to find out if your feed contains an object with the published date information, usually msg.article.pubdate, the time-zone used in that date, usually UTC, your output format, your time-zone and a different msg object to output inside the stream as a converted time object to be picked up later by the function.

Time Format Config

Please make sure to use the same level in your object name, so for example since we’re inside .article with .article.pubdate as our input, then use .article.time as your output. If you create a different level or use a previous one, it will NOT work.

 

function

At this point, we have our feed object, our title object and our properly formatted time object, but we need to put everything together and make a nice msg.payload object to push to the telegram palette (which is what it accepts as an object).

I’ve written this little js function to compose the msg.payload using all of our objects:

msg.payload = {
  chatId: -YOURCHATID,
  type: "message",
  content:
    msg.title +
    "\n\n" +
    msg.article.title +
    "\n\n" +
    msg.article.link +
    "\n\n" +
    "Bulletin created at:" +
    " " +
    msg.article.time +
    " " +
    "from:" +
    " " +
    msg.article.author,
};
return msg;

So the msg.payload is composed of these elements:

  1. chatId contains your chat ID which we recovered earlier
  2. type is the type of message we’re sending to Telegram, obviously it’s text, so message
  3. content is your message content itself, the way I wrote will create a message like this:
Content Example

So

msg.title

msg.article.title

msg.article.link

Bulletin created at: msg.article.time from: msg.article.author

As I explained previously, all of these objects might change from feed to feed and need to be written accordingly to what you find out using the debug palette connected to your regular feedparser palette.

 

telegram sender is the final palette in our logic stream and so it is tasked with communicating with the Telegram API and our bot. To configure your bot open the palette settings and near the bot field click on the pencil to add your bot info.

  • Bot-Name is the name you gave your bot, the one you can use to @ it, not the display name
  • Token is the API token I told you to save earlier, remember?

You don’t need to give it a ChatId parameter in here because we deliver it via the function I wrote.

Press Update when you’re done.

CONCLUSION

If everything goes to plan, the flow should spam your group chat with all the news feeds the first time and then every 15 minutes onward as long as node-red is running (or how often you configured it to do) and it will only give the new ones, even if you completely restart node-red or restart every flow. (Probably not if you restart your entire device.)

Thank you for reading.

Want to support me?

Find all information right here

You can also support me here:

Credits

  • My mom